As a counterpoint, though, unless you are using a select few time coincident loudspeakers (Vandersteen, Dunlavy, Quad, original Thiel, et al.) or applying digital impulse response correction, you are experiencing even more extreme time domain distortions from the crossovers in your loudspeakers.
Very few loudspeakers can preserve waveforms. Nearly all loudspeakers skew in time any non single sinusoidal waveforms. Just look at a typical loudspeaker acoustic output of a square wave. It is anything but square, more like a jigsaw – because not all frequencies are being reproduced in the proper time relationship.
Yet, this is standard for most loudspeakers. So, the audibility of time domain distortion is in question.
AJ