Romanticism prepared the ground for an interest in synesthesia [1] in the arts insofar as after the Enlightenment classical values shifted toward more subjective ones, with reason, measure, moderation, stability, permanence and clarity being superseded by imagination, affect, excess, movement, shadow and ambiguity. In the 19th century, a few scientifically minded individuals became curious about the peculiar types of perceived relationships between color and sound, and between letters and colors, whether reported by others or experienced firsthand. A more explicit interest in what would later be known as synesthesia reached a high point with Symbolism, in large part because of Symbolisms revival of the ancient theme of correspondence. Richard Wagner's concept of the Gesamtkunstwerk also played a role in the growing interest in synesthesia. Correspondences between aural, visual and textual elements were sought out in theory as well as in practice, but less so as autonomy became a defining value of modern art. With photography, cinema, radio and so forth, the dominance of traditional art gave way to newer competing ones. The interest in synesthesia was explicitly revived as artists in avant-garde cinema and intermedia pushed arts disciplinary boundaries, allowing new territories to emerge from the overlapping areas. Much discussion of Wagner's concept has already been taking place in multimedia studies for some years [2]. With the increasing presence and importance of digital art on all continents, it comes as no surprise that some multimedia artists pursue today what they call “digital synesthesia” [3].

With arts as with the senses, the question remains: Are the distinctions between them mainly “cultural” or “ideological,” or at least very much context-dependent? In light of the extraordinary developments in the neurosciences, the idea that the senses are separate from one another, as was long believed, is difficult to maintain. Digital media technologies constantly modify the way our senses are addressed, making synesthesia a theme to be reckoned with, as much so as was the case with cinema under the pen of Gene Youngblood [4].

Sustained research into peripheral vision was galvanized by the development of flight simulation and immersive virtual reality. Interest in peripheral vision in the arts is otherwise scant; the distinction between it and foveal vision is not always made, however significant it may be in neurophysiology and perceptual psychology. Simulator sickness shows that important challenges relating to immersion and peripheral vision need to be addressed, and research could prove especially beneficial for gamers [5]. The aim of this article is to contribute to the discussion on immersion by assessing synesthesia and proprioception through their link with peripheral vision. A few important artistic works have been selected to exemplify this topic—both well-known, recent artworks as well as little-known artworks from years ago.

MODAL AND CROSS-MODAL PERCEPTION

Merce Cunningham's Biped (1999)

With projected animations by Marc Downie, Shelley Eshkar and Paul Kaiser. © Photo Stephanie Berger. Image courtesy of the Merc Cunningham Foundation.

Proprioception is primarily defined as the perception of the body itself as spatial and the awareness of the relative position of limbs and body segments to one another. It includes kinesthesia, which refers to the sensations provided by variations in body and limb positions that are congruent with actual displacement through physical space. Proprioception is also defined as including the postural system, which is responsible for posture control through an ensemble of nervous pathways (efferent, afferent and reflex). According to Berthoz and Petit, posture control challenges the view that action and perception are separate. Posture is preparation for action: It is expressive, reflects intention, is dictated by culture and various other factors and always contains an emotion [6]. Sacks maintains that proprioception may very well constitute the “fundamental, organic mooring of identity—at least of that corporeal identity or ‘body-ego,’ which Freud sees as the basis of self” [7]. Furthermore, Gallagher writes, “Whenever consciousness begins, it will already be informed by embodiment and the processes that involve motor schema and proprioception” [8]. All of this is to say that proprioception is not to be considered one of the sense modalities but their condition of possibility: Without the a priori “sense of self” or the sense of embodiment that proprioception provides, sensory perception has no ground.

The awareness or sense of body is obtained in cooperation with vision and equilibrium (vestibular sense); a deficit in one can, in some cases and to some extent, be compensated for by reliance on the other [9]. Proprioception and kinesthesia convey sensory impressions not reducible to a discrete organ or a type of receptor but, through different configurations of these, provide particular patterns of activation. For example, we can experience kinesthesia either with or without the contribution of vision or of audition, that is, in darkness, in silence or in both. We can even experience it with either visual or auditory stimulations alone.

Sherrington [10] originally defined proprioception as resulting from sensory influx provided by receptors in muscles, tendons and joints and thus distinguished it from exteroception (sight, hearing, smell, taste and touch) and interoception (sensations from organs and pain). Currently, proprioception is understood as including the postural system, itself relying on exteroceptors for sensory information relating to weight (from vestibular receptors of the inner ear and plantar receptors in the foot), making the simplistic distinction between proprioception and exteroception problematic. Feedback loops link exteroception, proprioception and sensorimotor activity, allowing the brain to work through its internal modeling of motor space.

Sensorimotor activity involves the adjusting of motor commands to sensory perceptions through feedback loops in the cerebral cortex; the existence of multiple loops between various cerebral centers is suspected. Feedback is at play in motility, perception and postural control. The latter allows us to position ourselves, move and act in the physical world according to internal models. The brain essentially compares its predictions with reality, projecting its pre-perceptions and hypotheses onto the world [11].

Discussions concerning synesthesia are often framed within the bounds of the traditional concept of distinct modal perceptions: the five canonical senses. Far from accepting the medical portrayal of synesthesia as a neural disorder, Ramachandran and Hubbard speculate no less than a synesthetic theory of the origin of language [12]. There are many types of synesthetic phenomena and causes to which they can be attributed, and thus many possible theoretical understandings of synesthesia. Synesthesia can therefore also be understood as the fundamental rule of sensory perception, not merely as a collection of strange and rare exceptions to it [13].

THE RELATIONSHIP OF VISION TO KINESTHESIA

The body relies greatly on visual cues to locate itself and move in space. Peripheral vision plays a key role in space perception, locomotion, postural control and kinesthesia. Vision and kinesthesia are bound in a way that usually eludes conscious awareness. Individuals who are deficient in peripheral vision are reminded of it constantly, as they are prone to losing their balance, whether engaged in locomotion or not; simply standing upright may even prove to be a challenge.

Moving images, for example in the context of film viewing in a cinema theater, produce kinesthetic sensations in viewers. They already do so in any situation where our movements result in varying sequences of moving images impressed on our retinas. While walking down a busy downtown street, for example, I perceive through the corner of my eye an astounding quantity and variety of moving visual patterns and objects, either directly or by reflection. Visual impressions play an important role in the complex process of integration of spatial and bodily perceptions. The central nervous system (CNS) must reconcile kinesthetic and visual sensory information; the relative movements of objects in the perceived space around us must be coherent with our own body perception, whether in movement or still. The CNS has to attribute causes to perceived movement; it processes retinal input and compares it with other sensory ones, such as somatic proprioceptive information from the neck and the six extraocular muscles that control the eye movements. Motion of the head and of the eyes has to be accounted for in order for the CNS to decide whether the perceived moving images result from our own displacements or are independent of them.

The phenomenon of visual vection shows that the visual and vestibular systems are intricately interconnected. Most of us have experienced vection, perhaps when suddenly realizing that its not the train that we are in that is departing but the one on the next track. In this situation, if we are standing, we will likely shift our weight in the opposite direction of the visually perceived movement in order to counteract its effect, only to quickly realize that our body is no more put in motion than the train we are in. We are made to feel as if we are moving on the basis of the CNS's interpretation of the visual cues. Another instance would be what happens when we are sitting in a car, stopped at a traffic light, and the car in the next lane starts off first. In this case we may feel as if we are moving backward. Such instances of vection arise from the fact that the CNS assumes that the visually perceived movement is caused by something appearing in the visual field, rather than by the subject's own movement, since vision is a dominant sense.

In both the railway car and automobile examples above, the CNS interprets the optic flow as resulting from the subjects motion despite the fact that he or she is not moving; it is an automatic response to contradictory sensory information. When we walk about we do not perceive lampposts and buildings as whirling around; we perceive ourselves as moving about them. Information from the visual and equilibrium system concur—in other words, the optic flow impressed on the retina agrees with signals from vestibular receptors. This follows the rule that information from exteroceptive sense modalities “comes into a complex intermodal relationship with somatic proprioception to form a coordinated and intermodal sensory feedback” [14]. Vision affects the sensory quality of a proprioceptive experience, as does audition [15].

PERIPHERAL VISION AND ARTISTIC CONTEXT

Peripheral vision is part of our automatic surveillance and control system, which evolution has made especially sensitive to any large object looming in suddenly from the rear. In such instances, we are startled; perhaps we even let go of whatever was in our hands. Our motor system is primed to make us flee unimpeded should the threat prove to be real.

We are largely unaware of peripheral vision since our conscious attention is devoted to central vision. According to Jack Heggie, a child learns to shut out peripheral stimuli as he or she learns to read [16]. Central vision is active, conscious and precise, and it is sensitive to color and form. It is what many call “predatory” vision. In the animal kingdom, predators and prey have different visual fields, mainly because of the different locations of the eyes. Predators have forward-facing eyes that allow them to see best in the direction in which they are chasing. Forward-facing eyes also allow stereo vision, which helps estimate distance relative to an object. Prey usually have eyes situated more to the sides, which allows them better peripheral vision, vital for survival since they are likely to be attacked from above, behind or the side.

Fig. 1. View of visitor in Monika Fleischmann and Wolfgang Strauss's installation The Home of the Brain–Stoa of Berlin (1991)

© Monika Fleischmann and Wolfgang Strauss

The quality of the experience of immersion in a visual environment—whether material or virtual, laid out in a gallery or displayed on a stage—depends greatly on the contribution of peripheral visual perceptions in the total visual field of the spectator or user. When given the opportunity to move through an installation, equilibrium, postural adjustments and whole-body movements affect the perception of the space, along with its visual content. The users mobility has to be taken into account since it determines which visual content falls on the fovea and which falls on the periphery—in other words, what will be processed in terms of color and resolution for object recognition and what will affect proprioception by providing spatial references for the body’s equilibrium.

In contexts where we are seated, immobilized and looking straight ahead, sensorimotor activity is pretty much limited to the eyes and neck. When we sit down in front of a computer monitor, less is required of the postural system than when we are moving about. A higher level of activity is demanded with interactivity, that is, when the device is responding to the body’s movements, such as hand movements involved in the manipulation of the computer mouse, keyboard and data glove, and even eye movements when ocular tracking is used. An immersive context involving head motion alone is already a rather complex cybernetic system. Simulator sickness can result from delays between the content of the displayed visual scene and the body movements that triggered them [17].

Fig. 2. Installation room for virtual environments Osmose (1995) and Ephémère (1998), both by Char Davies.

Digital still image captured during immerive performance of the virtual environment Ephémère.© Char Davies

In immersive environments, visual and auditory outputs are combined to provide proprioceptive experiences. Tactile and haptic components are also often at play. HMD or projection screens can be used. Even both can be used, as with Fleischmann and Strauss’s The Home of the Brain—Stoa of Berlin (1991) (Fig. 1) and Davies's Osmose (1994-1995) (Fig. 2), which were designed as immersive environments for a firsthand user with the added possibility for outside observers to witness what the user sees and perhaps vicariously experience virtual immersion. According to Strauss, The Home of the Brain “is a ‘morphological simulation space, in motion,' which can be experienced polysensually and interactively.... In our opinion, the interactive media are supporting the multisensory mechanisms of the body and are thus extending mans space for play and action” [18].

Fig. 3. The position of the user relative to the screen in Jeffrey Shawand Matt Groeneveld's Legible City (1989).

© Jeffrey Shaw.

From the users perspective, the range of body movements the device allows is of fundamental importance to the intensity of the feeling of physical embodiment in virtual immersion. In Shaw and Groeneveld's Legible City (1989) (Fig. 3), the user is in a frontal situation and sitting down, but since she is on a bicycle pedaling and steering her way ahead, so to speak, the body’s engagement is much more physically expansive than it usually is in virtual navigation. Maine de Biran and Charles Bells intuitions about “sense of effort” and “muscle sense” come to mind here.

Osmose promotes contemplation rather than action; the HMD-fitted “immersant” navigates by means of a motion-capture vest equipped with breathing and balance sensors. Davies writes:

I think of immersive visual space as a spatio-temporal arena, wherein mental models or abstract constructs of the world can be given virtual embodiment in three dimensions and then kinaesthetically, synaesthetically explored through full-body immersion and interaction [19].

Fig. 4. Simulation of audience members within the "circle of screens" in Rebecca Allen's installation The Brain Stripped Bare (2002).

© Rebecca Allen. 3D graphics: Laura Hernandez Andrade.

Allen's The Brain Stripped Bare (2002) (Fig. 4) and Paxton's Phantom Exhibition (2009) (Fig. 5) imply deambulation by the user through an installation composed of several screens, both of which propose new ways of representing the body. The postural system of the visitor is explicitly addressed, its role being to maintain body equilibrium through visual, vestibular and other perceptual cues provided by both worlds, virtual and physical.

Allen's opus is described thus:

Surrounded by a circle of screens the audience is free to shift their point of view. Live performers merge with shadows, projected images and sounds, revealing stark human forms that move in startling and perplexing ways. This creates a raw, very physical yet illusory interactive experience that connects an audience to a performance in a way not previously explored [20].

Fig. 5. View of Steve ¨Paxton's screen installation Phantom Exhibition (2009).

© Steve Paxton. Photo: Baptiste Andrien and Florence Corin.

Five large screens surrounding the exhibition space show images of Paxton and other performers moving according to the method [contact improvisation], as well as dance moves simulated with computer graphics, along with poetically rhythmical explanatory narration. Within this overwhelming visual setting, the visitor perceives with all his senses the relationship between the human body and gravity [21].

The fifth screen of the “five large screens” is on the ceiling of the exhibition space. Paxtons installation offers the visitor an unusual perspective on the body.

Fig. 6. Merce Cunningham's Biped (1999)

With projected animations by Marc Downie, Shelley Eshkar and Paul Kaiser. © Photo Stephanie Berger. Image courtesy of the Merce Cunningham Foundation.

For a quarter of a century, thanks to the use of video projectors in theaters, the visual background has become quite animated, compared to earlier times when it was mostly static, burdened by bulk and weight and other inconveniences tied to physical objects. Projected scenography made possible a new breed of dance—multimedia dance—in which the constraints of software and hardware replace those of wood and canvas, as exemplified by Merce Cunningham's Biped (1999) (Fig. 6, Article Frontispiece) and Klaus Obermaier's Apparition (2004) (Figs 7-8). In these stunning multimedia choreographies, the spectators visual field is immersed in movement from two qualitatively different sources: movement of the dancer that addresses foveal vision and attention and movement of the surrounding images falling in the peripheral field. The merging of the dynamic impressions from these two sources corresponds to a merging of the senses brought about by the interrelation of foveal vision, peripheral vision, proprioception and kinesthesia. Cunningham eschewed correspondence, intent, meaning, causality, theatricality, illusionism and expression; being a Fluxus artist, he preferred the arts to exist in interference with one another rather than in concert. Cunningham's attitude still bewilders most audiences. Because Cunningham devised the choreography, music and projections independently from one another, no predetermined or causal relationship exists between the dancers movements and the nature of the projected images in Biped; there is no interactivity. Obermaiers aesthetics in Apparition are comparatively quite Wagnerian. Interactivity in this artwork is at the core of the relationship between the dancer and the projected image, which merge in a visually and dramatically coherent whole. The flowing quality of the movement contributes to the hypnotic effect of the ensemble. In one sequence of Apparition, the projected patterns of moving particles—flowing toward a center that is determined by the position of the dancer in front of the screen—induce vection strongly. This allows the spectator to indulge in the illusory proprioceptive sensation of being taken along, as if beyond the screen. Figure and background are distinctiy set off against one another. While the central vision is focusing on the dancer, the periphery is filled with congruent dynamic content. The scale of the staged display and the perceptual strategy put forward—by which we are led to identify or kinesthetically empathize with the dancers as protagonists in a dynamic visual drama—contribute to a feeling of immersion. I have previously argued that perceptual effects from the interplay of foveal and peripheral visual stimulations when the dancers move against an equally dynamic visual background produce a sensory experience that I call “kinetic synaesthesia” [22].

Fig 7 and 8. Klaus Obermaier, Apparition (2004), Lines (left) and Particles (right).

© Klaus Obermaier.

CONCLUSION

Peripheral vision is part of the intermodal neurosensory system in which posture, balance, proprioception and kinesthesia are integrated. It provides us with a sense of body and of the space in which the body can move. The feeling of immersion in installations rests upon the integration of various sensory perceptions in which neuromuscular activity plays an important part, especially when some form of interaction is provided. Sensory immersion can be understood in the wider context of synesthesia in art, as it provides contexts specifically designed to mesh the sensory perceptions into a perceptually coherent whole.

How we pay attention to visual stimuli and distinguish seeing from looking is a continual matter of choice. I find that being consciously aware of what is happening in my peripheral visual field as I walk down the street has an immediate effect on my proprioception. In this meditative/contemplative state/process, I definitively feel lighter and more relaxed when I open up to the periphery. In contradistinction to this phenomenological note, it is likely, in the context of the ubiquitous use of touchscreen mobile devices by pedestrians all over the world, that the complex cortical integration of central and peripheral vision inputs involved in the tasks of reading or texting while avoiding stepping into manholes or bumping into lampposts is transforming some human perceptual capabilities.

To conclude on a general note: Spinoza wrote that “Mind and body are one and the same individual which is conceived now under the attribute of thought, and now under the attribute of extension” [23]. Just as we have learned to live with two theories of light, corpuscular and electromagnetic, we should be able to accommodate two different perspectives regarding the senses: the canonical view, according to which the senses are separate, and the “romantic” one, according to which they are united.