“When I see equations, I see the letters in colors – I don’t know why. As I’m talking, I see vague pictures of Bessel functions, with light-tan j’s, slightly violet-bluish n’s, and dark brown x’s flying around. And I wonder what the hell it must look like to the students.” – Richard Feynman
Born to integrate
The human senses do not work in isolation, as we have written previously in posts on the individual senses and also on the role of synaesthesia in creativity and shaping our perceptions of the world to create meaning. Some research suggests that synaesthesia is more common in young children than adults (although there is some controversy over this). It is generally accepted that synaesthesia is related to greater brain connectivity, and one of the most plausible theories for its development is based on differential ‘development’ and ‘pruning’ of these connections in young childhood as the brain develops.
And we all use multiple sensory input every day, even when we don’t realise. For instance, we can interpret speech based on watching (or even touching) someone’s face as well as the more normal method of listening. In fact we use all of these inputs all the time, with our brain interpreting and combining multiple inputs to decide their meaning. Consider the McGurk effect, which is a very well understood outcome of the interaction between visual and auditory perception. The best known example is of listening to someone say ‘ba’ while watching the same person say ‘ga’. Even if you have previously seen the person saying ‘ba’, the effect of the visual of ‘ga’ is to hear ‘da’ – which is the brain’s best guess at the sound based on combining what you can see with what you can hear!
Can you see what I’m saying?
There are many examples of similar effects. Did you realise that holding a pen in your mouth changes your perception of the facial emotions of others? And that the taste of food and drink is a subtle mixture of the basic tastes on the tongue, smells in your nose, the sounds from chewing, the appearance of the food, and the touch and feel of the muscles and bones moving to chew the food. The ‘rubber hand’ trick demonstrates the relationship between sight and touch (you can kid someone you are stroking their hand).
Most impressively, men use a whole range of sensory clues (especially sight and smell) to judge the reproductive potential of women without being aware of this (which is why strippers earn up to 40% more in tips just before ovulation). It’s likely that greater facial symmetry. lower hip waist ratios, lighter skin colour and changes in dress all contribute to the greater attractiveness of women even though they and their admirers are unaware of the changes.
Patterns of behaviour
In terms of the brain, there is no strict specialisation and although generally specific areas of the brain focus on specific tasks, there is considerable evidence that boundaries are very blurred. More importantly, the way that perceptual information is interpreted is not simply top down, but consists of a number of lower and higher level feedback mechanisms driving different mental and body systems.
However, the top down function is very important for some of the ‘effects’ described above, and for explaining many of the anomalies of behaviour in behavioural economics and psychology. Once the brain has decided that inputs represent a particular event in the outside world, then this interpretation will drive behaviour and shape outcomes, often over-riding new evidence. Integration works by trying to link specific perceptions to some previous experience, which is why context is so critical to behavioural outcomes. If the brain recognises one or more particular stimuli as consistent with a specific previous experience then other stimuli will be interpreted in light of that event.
The interpretation of the outside world is all about pattern recognition, and there is some evidence that the different senses are more similar than we realise. If you think about a car moving towards you, then there are changes in your visual perception (the car gets bigger) and in your hearing (the sound of the engine gets louder). In fact, these changes will happen at the same rate, so the rate of change of both perceptions is actually equivalent, and your brain therefore has two reasons to tell you to get out of the way. Behaviour is thus linked to the recognition of a particular pattern (in this case of a change), independent of which sensory modality is the cause.
Of course, when the senses provide the same patterns the signal is strengthened, and there is compelling evidence that providing stimulus across multiple senses has a multiplicative effect on their impact. The lessons for sensory branding are that engaging one of the senses is great, but engaging multiple senses is so much better, as long as you ensure that the messages are consistent across all the senses!
A Natural History of the Senses by Diane Ackerman (1991)
See What I’m Saying: The Extraordinary Powers of our Five Senses by Lawrence D. Rosenblum (2010)