Our five senses–sight, hearing, touch, taste and smell–seem to operate independently, as five distinct modes of perceiving the world. In reality, however, they collaborate closely to enable the mind to better understand its surroundings. We can become aware of this collaboration under special circumstances.
In some cases, a sense may covertly influence the one we think is dominant. When visual information clashes with that from sound, sensory crosstalk can cause what we see to alter what we hear. When one sense drops out, another can pick up the slack. For instance, people who are blind can train their hearing to play double duty. Those who are both blind and deaf can make touch step in—to say, help them interpret speech. For a few individuals with a condition called synesthesia, the senses collide dramatically to form a kaleidoscope world in which chicken tastes like triangles, a symphony smells of baked bread or words bask in a halo of red, green or purple. (For more on how the senses can cross each other and into unusual territory, see “Edges of Perception,” by Ariel Bleicher, Scientific American Mind, March/April 2012.)
Our senses must also regularly meet and greet in the brain to provide accurate impressions of the world. Our ability to perceive the emotions of others relies on combinations of cues from sounds, sights and even smells (see “I Know How You Feel,” by Janina Seubert and Christina Regenbogen, Scientific American Mind, March/April 2012). Perceptual systems, particularly smell, connect with memory and emotion centers to enable sensory cues to trigger feelings and recollections, and to be incorporated within them (see “Smells Like Old Times” by Maria Konnikova Scientific American Mind, March/April 2012). But the crosswiring of the senses themselves provides some of the most fantastic fodder for illusions, inventions and just plain art. Here are a few of the best examples of the complex interactions – and extraordinary feats – of our cross-wired senses.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
Seeing What You Hear
We can usually differentiate the sights we see and the sounds we hear. But in some cases, the two can be intertwined. During speech perception, our brain integrates information from our ears with that from our eyes. Because this integration happens early in the perceptual process, visual cues influence what we think we are hearing. That is, what we see can actually shape what we "hear." This visual-auditory crosstalk, which happens every time we perceive speech, becomes obvious in this video of a phenomenon called the McGurk Effect. In this case, despite the fact that you are listening to the same sound (the word "bah"), what you hear depends on which face you are looking at. The effect persists even after you learn about it, so reading about the McGurk Effect won't spoil it for you.
Beep Baseball
Blind baseball seems almost an oxymoron. But since 1975, when a few blind Minnesotans invented "beep baseball," those who lack sight have taken part in America's favorite pastime. Thanks to a one-pound beeping oversized softball and some tweaks to the game, players can hit a home run without ever seeing the ball. They use the sound the ball emits to orient themselves, make contact using a bat, and run to base. They might be particularly well-suited to this form of the game, as previous research suggests that blind individuals can more easily localize sounds than sighted people can. You can see how well they play in this video.
Calling What You See
Bats and whales, among other animals, emit sounds into their surroundings—not to communicate with other bats and whales—but to “see” what is around them. They read echoes of the sound waves, which bounce off objects, to identify and locate objects. This sensory system is called echolocation. Although most of us can only imagine the pictures that form from sound, some blind people have managed to master a form of echolocation. By uttering sounds and clicks, these individuals can use their ears to navigate. Some, such as Daniel Kish, have even taught others to use this form of human sonar. Kish once described human echolocation as “something like seeing the world in dim flashes of light.” In this video, an artist show how those flashes might create useful impressions of the outlines of objects.
Let Your Fingers Do The Hearing
People who are both deaf and blind are incredibly good at using other senses such as touch to navigate and understand the world. Some use the Tadoma Speechreading Method to perceive speech by touching the lips of another person as they talk. First taught in the 1920s, lip-reading by touch was a popular form of communication among the deafblind. Helen Keller was one of its early adopters.
If taught early in development, the Tadoma Method can help a deafblind child learn to speak as well as to understand others. Those who lose their sight and hearing later in life can use it to read lips. But because the method is extremely difficult and time consuming to learn, by the 1950s it began to lose ground to American Sign Language as the dominant teaching method. In ASL, the deafblind place their hands over another signer’s hands and follow the motions with their fingers—which is easier because the movements are far less subtle. Today, only about 50 people in the world still use of the Tadoma Method. Watch some of them at work in this clip.
Do You Have Synesthesia? Take This Test
People with synesthesia have a particularly curious cross wiring of the senses, in which activating one sense spontaneously triggers another. They might see colors when they hear noises, associate particular personalities with days of the week, or hear sounds when they see moving dots. Synesthesia is thought to be genetic, and recent research even suggests that it may confer an evolutionary advantage. Most synesthetes don't notice anything strange about the way they perceive their environments until it is brought to their attention. One young woman only found out she was a synesthete in her freshman year of college after attending a talk on the topic. This video is a test for one form of synesthesia. Watch the dots and “see” if you hear anything!
A World In Which Senses Fuse
What might life be like if you had synesthesia? Here is one artist's rendition of the experience of a synaesthete. In this surreal world, music records smell like different colors, foods tastes like specific noises, and sound comes in all varieties of textures and shapes.