The Moving Look Of Music
    By Mark Changizi | July 28th 2010 09:26 AM | 7 comments | Print | E-mail | Track Comments
    About Mark

    Mark Changizi is Director of Human Cognition at 2AI, and the author of The Vision Revolution (Benbella 2009) and Harnessed: How...

    View Mark's Profile

    I believe that music sounds like people, moving. Yes, the idea may sound a bit crazy, but it’s an old idea, much discussed in the 20th century, and going all the way back to the Greeks. There are lots of things going for the theory, including that it helps us explain (1) why our brains are so good at absorbing music (…because we evolved to possess human-movement-detecting auditory mechanisms), (2) why music emotionally moves us (…because human movement is often expressive of the mover’s mood or state), and (3) why music gets us moving (…because we’re a social species prone to social contagion). And as I describe in detail in my upcoming book – "Harnessed: How Language and Music Mimicked Nature and Transformed Ape To Man" – music has the signature auditory patterns of human movement (something I hint at here ).

    Here I’d like to describe a novel way of thinking about what the meaning of music might be. Rather than dwelling on the sound of music, I’d like to focus on the look of music. In particular, what does our brain think music looks like?

    It is natural to assume that the visual information streaming into our eyes determines the visual perceptions we end up with, and that the auditory information entering our ears determines the events we hear. But the brain is more complicated than this. Visual and auditory information interact in the brain, and the brain utilizes both to guess the single scene to render a perception of. For example, the research of Ladan Shams, Yukiyasu Kamitani and Shinsuke Shimojo at Caltech have shown that we perceive a single flash as a double flash if it is paired with a double beep. And Robert Sekuler and others from Brandeis University have shown that if a sound occurs at the time when two balls pass through each other on screen, the balls are instead perceived to have collided and reversed direction. These and other results of this kind demonstrate the interconnectedness of visual and auditory information in our brain. Visual ambiguity can be reduced with auditory information, and vice versa. And, generally, both are brought to bear in the brain’s attempt to infer the best guess about what’s out there.

    Your brain does not, then, consist of independent visual and auditory systems, with separate troves of visual and auditory “knowledge” about the world. Instead, vision and audition talk to one another, and there are regions of cortex responsible for making vision and audition fit one another. These regions know about the sounds of looks and the looks of sounds. Because of this, when your brain hears something but cannot see it, your brain does not just sit by and refrain from guessing what it might have looked like. When your auditory system makes sense of something, it will have a tendency to activate visual areas, eliciting imagery of its best guess as to the appearance of the stuff making the sound. For example, the sound of your neighbor’s rustling tree may spring to mind an image of its swaying lanky branches. The whine of your cat heard far way may evoke an image of it stuck up high in that tree. And the pumping of your neighbor’s kid’s BB gun can bring forth an image of the gun being pointed at Foofy way up there.

    Your visual system has, then, strong opinions about the proper look of the things it hears. And, bringing ourselves back to music, we can use the visual system’s strong opinions as a means for gauging music’s meaning. In particular, we can ask your visual system what it thinks the appropriate visual is for music. If, for example, the visual system responds to music with images of beating hearts, then it would suggest, to my disbelief, that music mimics the sounds of heartbeats. If, instead, the visual system responds with images of pornography, then it would suggest that music sounds like sex. You get the idea.

    But in order to get the visual system to act like an oracle, we need to get it to speak. How are we to know what the visual system thinks music looks like? One approach is to simply ask which visuals are, in fact, associated with music? For example, when people create imagery of musical notes, what does it look like? One cheap way to look into this is simply to do a Google (or any search engine) image search on the term “musical notes.” You might think such a search would merely return images of simple notes on the page. However, that is not what one finds. To my surprise, actually, most of the images are like the one in the nearby figure, with notes drawn in such a way that they appear to be moving through space. Notes in musical notation never actually look anything like this, and real musical notes have no look at all (because they are sounds). And yet we humans seem to be prone to visually depicting notes as moving all about.

    Could these images of notes in motion be due to a more mundane association? Music is played by people, and people have to move in order to play their instrument. Could this be the source of the movement-music association? I don’t think so, because the movement suggested in these images of notes doesn’t look like an instrument being played. In fact, it is common to show images of an instrument with the notes beginning their movement through space from the instrument: these notes are on their way somewhere, not an indication of the musician’s key-pressing or back-and-forth movements.

    Could it be that the musical notes are depicted as moving through space because sound waves move through space? The difficulty with this hypothesis is that all sound moves through space. All sound would, if this were the case, be visually rendered as moving through space, but that’s not the case. For example, speech is not usually visually rendered as moving through space. Another difficulty is that the musical notes are usually meandering in these images, but sound waves are not meandering – sound waves go straight. A third problem with sound waves underlying the visual metaphor is that we never see sound waves in the first place.

    Another possible counter-hypothesis is that the depiction of visual movement in the images of musical notes is because all auditory stimuli are caused by underlying events with movement of some kind. The first difficulty, as was the case for sound waves, is that it is not the case that all sound is visually rendered in motion. The second difficulty is that, while it is true that sounds typically require movement of some kind, it need not be movement of the entire object through space. Moving parts within the object may make the noise, without the object going anywhere. In fact, the three examples I gave earlier – leaves rustling, Foofy whining, and the BB gun pumping – are noises without any bulk movement of the object (the tree, Foofy, and the BB gun, respectively).  The musical notes in imagery, on the other hand, really do seem to be moving, in bulk, across space.

    Music is like tree-rustling, Foofy, BB guns and human speech in that it is not made via bulk movement through space.  And yet music appears to be unique in this tendency to be visually depicted as moving through space. In addition, not only are musical notes rendered as in motion, musical notes tend to be depected as meandering.

    When visually rendered, music looks alive and in motion (often along the ground), just what one might expect if music’s secret is that it sounds like people moving.

    A Google Image search on “musical notes” is one means by which one may attempt to discern what the visual system thinks music looks like, but another is to simply ask ourselves what is the most common visual display shown during music. That is, if people were to put videos to music, what would the videos tend to look like?

    Lucky for us, people do put videos to music! They’re called music videos, of course. And what do they look like? The answer is so obvious that it hardly seems worth noting: music videos tend to show people moving about, usually in a time-locked fashion to the music, very often dancing.

    As obvious as it is that music videos typically show people moving, we must remember to ask ourselves why music isn’t typically visually associated with something very different. Why aren’t music videos mostly of rivers, avalanches, car races, wind-blown grass, lion hunts, fire, or bouncing balls? It is because, I am suggesting, our brain thinks that humans moving about is what music should look like…because it thinks that humans moving about is what music sounds like.

    Musical notes are rendered as meandering through space. Music videos are built largely from people moving, and in a time-locked manner to the music. That’s beginning to suggest that the visual system is under the impression that music sounds like human movement. But if that’s really what the visual system thinks, then it should have more opinions than simply that music sounds like movement. It should have opinions about what, more exactly, the movement should look like. Do our visual systems have opinions this precise? Are we picky about the mover that’s put to music?

    You bet we are! That’s choreography. It’s not enough to play a video of the Nutcracker ballet during Beatles music, nor will it suffice to play a video of the Nutcracker to the music of Nutcracker, but with a small time lag between them. The video of human movement has to have all the right moves at the right time to be the right fit for the music. 

    These strong opinions about what music looks like make perfect sense if music mimics human movement sounds. In real life, when people carry out complex behaviors, their visual movements are tightly choreographed with the sounds – because the sight and sound are due to the same event. When you hear movement, you expect to see that same movement. Music sounds to your brain like human movement, which is why when your brain hears music, it expects that any visual of it should be consistentwith it. 

    This was adapted from Harnessed: How Language and Music Mimicked Nature and Transformed Ape to Man (Benbella Books,2011).


    Very cool, Mark. I've heard the theory put out their before that musical instruments were made to imitate human voices, but your theory is much more complete, and makes much more sense to me logically. One question I would ask, though, that I don't know if you answered, is WHY this audio-visual link between human movement and music exists. I realize why, if the brain sees music as human movement, an animal like us so dependent on social interaction would enjoy it so much, but I'm not understanding the reason the link exists in the first place. What is it about music that sounds like human movement? I guess there's a sort of rhythym to the way we move, rather than a herky-jerky or stop-and-start, but why would that inherently look to us like music sounds (and vice versa)?

    I'm also pretty sure your hypothesis lends itself quite nicely to "If the Beatles' music sounds like Brooklyn Decker looks, then Amy Winehouse's music sounds like Amy Winehouse looks" jokes.

    I am always a fan of those A is to B as C is to which of the following standardized test jokes, which I think makes me a nerd.

    Mark Changizi
    The link from music to human-movement is that music actually sounds like human movement. And has been culturally selected to sound like this. I argue. In the next book I make an extended case for this, uncovering the signature patterns we make when we move (in pitch, loudness, tempo and rhythm), and provide data showing that these patterns are the fundamental patterns found in music.
    Yes, I understood your argument on the cultural selection of music, maybe instead of asking "why", I meant "how does music sound like human movement looks?" From the 2nd half of your response, it looks like you are expanding that argument in your book. I think that's the missing piece of the argument that can tie everything together.

    Mark Changizi
    Yeah, I'm being vague on that, because it isn't published anywhere yet. I give some more hints here...
    Gerhard Adam
    Why aren’t music videos mostly of rivers, avalanches, car races, wind-blown grass, lion hunts, fire, or bouncing balls?
    Hmmm ... I'm not completely convinced this is complete, after all, music plays a major role in precisely these situations when used as a background to movies and storytelling.  In fact, one can often describe specific emotional/psychological responses that a particular piece (or type) of music evokes (such as the minor key sounding sad).

    There are types of instrumental music that can create visual images of a story being told, just as specific passages can contain emotions (i.e. a solo sounds "angry" or "happy").  Similarly it clearly suggests more than movement when B.B. King describes his phrasing as being reminiscent of someone singing, or Eric Clapton uses the phrase "woman tone" to describe a particular guitar setting.  Even Eddie Van Halen describes his particular tone as the "Brown sound", as well as tones we describe as being "warm".

    I've even seen animals react to particular songs in quite specific ways (i.e. repeatable) which suggests that they also are subject to the influence of music.  While I can't possibly know what they hear, or how they interpret it, nor do I know what type of "feeling" it might elicit in them, it seems clear that there are patterns that appear to be almost species related (i.e. flutes and light strings seem to elicit singing in larger birds like Cockatoos and Greys).  It is this aspect that makes me wonder if many of the sounds produced by music don't trigger a neural reaction to environmental noise (i.e. birds - flutes - other birds - jungle) rather than simply movement.

    An interesting question to consider is whether animals also generate a form of music?  After all, there is little in human music that suggests an independence from storytelling (even loosely with instrumental music), so how would we differentiate an animal's production of "music" versus straight communication (and is there a difference)?
    Mundus vult decipi
    There are also studies that show music effects how people act (e.g. classical music played in the background at department stores tends to correlate with a drop in shoplifting).

    There is also the fact that for many people, music brings up images of color. There's an entire genre called the "blues", and although the name is also based on the color associated with sadness, an emotion, one can easily make the argument that the color, music, and emotion are all interrelated in human sensory perception. The popular images of the psychedelic tye-dyed pinwheel during the era of acid/hippie rock n roll also come to mind as a color association with certain instruments/notes.

    Mark Changizi
    It's true that we can put music to anything. In movies, the visual is usually the thing that exists first, and then they have to find an appropriate audio track. I'd have to speculate here that they try to choose the audio having the kind of human movement and emotion that best fits the visual, even when the visual has no human in it. When the music exists first, and they have to decide what visual to put in, the clear tendency is to have people, moving.

    On music for other animals, some aspects of our human-movement sounds would generalize to other animals, and some would not. On the basis of my approach, one could, in principle, design music for some particular kind of animal. If they're not social, then this wouldn't work, on my view. But, of course, speculating only now.