When we hear the sound of instruments or voices, yet the source is blocked from our view, we are still able to tell in most cases whether the sound comes from a loudspeaker or is live. What allows us to recognize the difference? What cues might we work from? What forms of distortion affect the loudspeaker reproduction?

Most likely it is not the on-axis frequency response of the speaker, because if the source is in some other room or building, the frequency spectrum that reaches us may be rolled off at the high end or modified in some unpredictable way. It could be the power response, the radiation in all directions that is often different between a speaker and a live source, that gives us a clue. Most speakers have a power response that drops 10 dB to 20 dB from low to high frequencies.
Most likely it is not the start up transient response of the speaker, because by the time we hear the signal, it has undergone many reflections and the waveform fidelity has been lost. It could well be, though, the slow decay of transients due to energy storage in resonant mechanical and acoustical structures of the speaker which we recognize as typical for a loudspeaker and missing in the corresponding live event.
Most likely it is not simply the dynamic range between loud and soft that gives us a cue, because we can usually tell the nature of even a very faint sound that comes from a far distance. It could be, though, that for loud sounds we recognize the change in sound character that is caused by intermodulation in the loudspeaker and which creates new spectral components that are foreign to the live source.

We normally listen to a speaker relatively close up and the above observations do not apply completely, but I have found that an excellent test of a speaker is to listen to it from the next room or from down the hallway. Consistent with the above observations, I have found a set of priorities for loudspeaker design that must be followed to obtain an outstanding product.

Low non-linear distortion, e.g. drivers that can move sufficient amounts of air linearly in all parts of the frequency range but especially at the low end.
Minimal excitation of room resonances, particularly at low frequencies. This requires low frequency directional speakers such as dipoles.
Low amounts of stored energy in drivers, cabinet, air cavities and filters for fast transient decays.
Smooth, extended frequency response from 20 Hz on up and without exaggerated high frequencies, both on-axis and off-axis. Minimal roll-off in power response.

There are additional requirements, such as an acoustic center for the speaker at ear height, vertical extension of the source, etc, etc, - but I consider the above four as most important in their given order.

Many speaker designs fall short of these priorities and try to use psycho-acoustic effects to overcome their sonic deficiencies. A lack of bass due to drivers and cabinets which are too small can be compensated to some extend by a boost in the 100 Hz to 200 Hz region. Hearing the overtones of the bass notes our mind fills in the missing fundamentals. Or, the particular form of distortion of a given small driver can give the impression of a lot of "bass punch" and "greater output" than is physically possible. Yet, we can recognize this as loudspeaker sound and are not fooled into believing that we listen to the real thing.

Sound reproduction is like trying to create an auditory illusion in our mind, similar to the visual illusion that a magician creates on stage. Just as the magician has to present us with the right cues, so has a loudspeaker. It is most important to avoid giving the wrong cues as in the case of the small, distorting driver. It is likewise very important that the correct cues from the loudspeaker are not masked by the acoustic properties of the listening room.

When all this is taken into consideration it is indeed possible - despite the multitude of examples to the contrary - to build loudspeakers that will do a respectable job of reproducing complex sonic events, provided they have been recorded with a minimum of acoustic and electronic processing.

Loudspeaker evaluation is not like wine tasting, though a majority of audio sales persons and magazine reviewers treat it like that. Unlike in wine tasting, you have an absolute reference in naturally occurring sounds. Familiarize yourself with a wide range of un-amplified sounds and keep them in mind when you try to judge the accuracy of a loudspeaker.