|UFDC Home||myUFDC Home | Help|
This item has the following downloads:
1 COLOR AND SOUND: SYNAESTHESIA AT THE CROSSROADS OF MUSIC AND SCIENCE By MATTHEW LEONARD MCCABE A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 2010
2 2010 Matthew Leonard McCabe
3 To my family: Mom, Dad, Pop, Tyler, Lucy and Darcy
4 ACKNOWLEDGMENTS The list of people that merit acknowledgment is enormous. Foremost, my parents Peter and Susan deserve immense thanks for letting me pursue my own interests across the entire space of my life. My brother Tyler has been a steadfast friend, even when geograph ically separated us by hundreds of miles. My teachers, Benjamin Broening, Elainie Lillios, Mikel Kuehn, James Paul Sain, Russell Robinson, and Paul Richards deserve a huge amount of credit for helping me tune my own voice and express myself in the most profound way I can imagine. Jamie Reilly, Lise Abrams, and Linda Hermer deserve many thanks for taking a chance on a musician in their scientific world (and waiving prerequisites to take their classes!). Lab projects, discussions, and coursework with them and several other members of the UF psychology community have been infinitely inspiring. Amy Corning, Rick Dietrich, and Cain Norris deserve specific thanks for their assistance with the statistics and data manipulations found in this document. Finally, to my friends from Gainesville, thank you for being with me on this incredible journey: Ben Baldwin, Berkeley and Gar Hoflund, Russell Brown, Heather McReynolds, Stefanie Acevedo, Joo Won Park, Mike Solomon, Rick Dietrich, Kyle Vegter, and the countless others too numerous to list. This document is dedicated to the memory of Dr. Anne Kish, my first music teacher, who showed me how to see and hear everything differently. Not a day goes by in my life as a musician that she does not cross my mind. I miss you, Doc.
5 TABLE OF CONTENTS page ACKNOWLEDGMENTS...............................................................................................................4! LIST OF TABLES...........................................................................................................................7! LIST OF FIGURES.........................................................................................................................8! LIST OF OBJECTS.........................................................................................................................9! ABSTRACT...................................................................................................................................10! CHAPTER 1. INTRODUCTION..................................................................................................................14! Victorian-Era Curiosity into Human Sensory Systems..........................................................14! Contemporary Notions of Cross-Modal Perception...............................................................16! About This Study....................................................................................................................18! Overview of the Chapters.......................................................................................................21! 2. REVIEW OF THE LITERATURE........................................................................................22! The Ancient World.................................................................................................................22! Historical Examples of Synaesthetic Behavior in Western Culture.......................................23! Scientific Angles: from Francis Galton to Behaviorism........................................................27! Synaesthetic Elements in Music and Art: The 20th Century..................................................31! Contemporary Scientific Research: Synaesthesia Reemerges...............................................35! Anatomical Substrates, Brain Plasticity, Psychedelic Drugs, and Neonatal Synaesthesia.................................................................................................................36! The Relevance of Synaesthetic Behavior........................................................................39! The Term Synaesthesia in Modern Research...............................................................39! Potential Problems with Synaesthetic Behavior..............................................................42! Specific Work on Chromaesthesia..................................................................................43! Continuing Engagement with the Cross-Modal: A Hypothesis............................................45! 3. EXPERIMENTAL METHODOLOGY AND PROCEDURES.............................................48! Introduction.............................................................................................................................48! Participants.............................................................................................................................49! Participant Compensation.......................................................................................................49! Procedures...............................................................................................................................50! Stimulus Preparation.......................................................................................................50! Pre-Experimental Procedures..........................................................................................52! Experimental Procedures.................................................................................................53! Experiment I: Color Swatch to Pure Tone Cross-Modal Matching........................53!
6 Experiment II: Pure Tone to Color Swatch Cross-Modal Matching.......................54! Experiment III: Color Swatch to Piano Tone Cross-Modal Matching....................54! Experiment IV: Piano Tone to Color Swatch Cross-Modal Matching...................54! 4. EXPERIMENTAL RESULTS...............................................................................................55! Introduction.............................................................................................................................55! Participant Demographics...............................................................................................55! Data Conversions.............................................................................................................56! Data Analysis..........................................................................................................................57! Analysis of Pitch vs. Lightness.......................................................................................58! Analysis of Pitch vs. Color Difference (!E)...................................................................63! Analysis of Pitch to Cone Space (" LMS)......................................................................65! Summary of Results................................................................................................................68! 5. COMPOSITIONAL METHODOLOGY................................................................................70! Introduction.............................................................................................................................70! A New Composition: Color Shifts .........................................................................................71! Movement I: Red Pipes ..................................................................................................71! Movement II: Green Voices ...........................................................................................72! Movement III: Blue Orchestra .......................................................................................73! Summary.................................................................................................................................74! 6. DISCUSSION AND CONCLUSIONS..................................................................................76! Introduction.............................................................................................................................76! Discussion of Findings...........................................................................................................77! Sound Symbolism in Music.............................................................................................78! Susanne Langers Concept of Secondary Illusion in the Arts.........................................81! Conclusions.............................................................................................................................83! WORKS CITED ............................................................................................................................84! BIOGRAPHICAL SKETCH.........................................................................................................90!
7 LIST OF TABLES Table page 2-1 Cytowics elements of synaesthesia (Cytowic, 2002; Hochel & Miln, 2008)..................40! 3-1 Pitches used in the experimental procedures, in ascending order using Acoustical Society of America pitch convention (Young, 1939) ........................................................51! 4-1 Participant demographics and assessment scores ...............................................................55! 4-2 Statistical correlations of lightness vs. pitch, Experiments 1 and 3....................................58! 4-3 Statistical correlations for pitch vs. lightness, Experiments 2 and 4...................................60! 4-4 Stati stical comparisons between experiments for pitch and lightness................................62! 4-5 Statistical comparisons between experiments for pitch and colorfulness...........................65! 4-6 Statistical comparisons between experiments for pitch vs. LMS....................................68! 5-1 Filter example: granular settings for Red Pipes .................................................................72! 5-2 Filter example: the resonance filter in Blue Orchestra ......................................................73!
8 LIST OF FIGURES Figure page 3-1 Color swatches used in the experimental procedures, with their respective RGB values. Swatches are sorted by hue ranging from 0 to 345............................................51! 4-1 Light and dark bias in pitch selection, Experiment 1.........................................................59! 4-2 Light and dark bias in pitch selection, Experiment 3.........................................................59! 4-3 Linear mapping of pitch to lightness, Experiment 2..........................................................61! 4-4 Linear mapping of pitch to lightness, Experiment 4..........................................................61! 4-5 Pitch to colorfulness trend, Experiment 2..........................................................................64! 4-6 Pitch to colorfulness trend, Experiment 4..........................................................................64! 4-7 Pitch to LMS trend, Experiment 2.................................................................................67! 4-8 Pitch to LMS trend, Experiment 4.................................................................................67! 5-1 Screen shot of Apples Space Designer with blue filter loaded..................................74!
9 LIST OF OBJECTS Object page 5-1 Composition: Color Shifts (AIFF, 127.5MB)....................................................................75!
10 Abstract of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy COLOR AND SOUND: SYNAESTHESIA AT THE CROSSROADS OF MUSIC AND SCIENCE By Matthew Leonard McCabe May 2010 Chair: James Paul Sain Major: Music Human beings, like many other species, possess finely developed sensory systems that give each individual access to the environment, the ability to communicate with others, and the means to acquire knowledge and experience. In humans, sensory systems are traditionally described in five specific domains: the visual, auditory, gustatory, olfactory, and tactile. The human body is constructed to send and receive signals in each of these sensory modes, and the human brain is acutely capable of processing multiple sensory information streams simultaneously. Since these sensory systems provide the interface between the interior and exterior worlds, they embody perhaps the most salient element of existence: we are defined, to a certain degree, by our senses. They act as the pipeline through which we sense our surroundings and thus, place ourselves into the larger world. Scientific research on human perception and the endeavors of creative artists have consistently revealed that the traditional five senses are not strictly discrete and that the intermixing of these senses is far more common than generally acknowledged. Such mixtures appear critical to the shaping of human knowledge, experience, and creativity both on an individual and a group scale. Such observations have established new ideas about sensory interconnectivity and humanitys ability to engage in metaphor and creativity. From this
11 perspective, multisensory integration acts as a critical substrate in the continued evolution of both language and artistic expression. These two traits, found nowhere else among living species, are essential elements of humanity, driving our cultural and interpersonal lives. Proponents of this theory of sensory integration argue that the construction of the human sensory system offers the basic capacity for abstraction, the fundamental building block of artistic expression and creative capacities. The ability to connect two seemingly disparate pieces of knowledge, the researchers argue, allows human beings to, for example, name objects and create artistic works. These capacities are further enriched through theoretical expansions of the five traditional senses, with some researchers including proprioception, memory, and emotion as additional internal senses that are equally as relevant as the external modes. Though arguments about the number of sen ses humans possess remains debatable, expansion of the list only expands the possibility of intermixing. Recent findings on sensory interaction would appear to verify the cross-modal nature of the system as a whole, and have led to theories on how cross-modal perception, sensation, and association contribute to various human faculties. At the same time, trends in cognitive sciences have encouraged a recent surge in music as a topic viable for scientific exploration. Music, art, and other creative capacities have become prime subjects for study, and when paired with the revitalized efforts to study synaesthesia (a neurological condition of cross-wired senses) and cross-modal perception, new ways of examining human experience have achieved relevance and popularity. Though many researchers and artists in the present time explore the interconnection of the senses, these activities are not strictly modern. The Greek word mousik!, the etymological ancestor to many modern languages word for music, referred to both songs and poems, and indeed any art presided over by the muses. Poetry often employs inter-sensory elements that
12 contribute to expressive power: color, timbre, rhythm, and sound are core poetic devices that are simply unable to exist on the page, but are critical elements of the poetic art. Many approaches to cross-modal interactions appear in philosophical and scientific writings that span much of recorded history: from ancient Greek and Roman societies, through the Victorian era, to todays neuroscientific and psychological approaches. Cross-modal relationships in the arts appear frequently as well, with musicians and visual artists crossing the boundaries of their home disciplines to create visual music or sound paintings. When coupled with historical and modern investigation into synaesthesia, a rare sensory phenomenon where individuals consciously experience events in one sensory mode due to the presence of stimulus in another, investigations into synaesthesia and cross-modal associations can be conducted on an even deeper level. Chromaesthesia, a type of synaesthesia where individuals experience colored visual percepts in response to musical pitches, is one of the most common types, and offers a direct tie to music in the realm of cross-modal research. To contribute to the exploration of cross-modal perception and its possible role in musical contexts, the present study explores the interconnectedness of the auditory and the visual, with a particular focus on color and sound. The core topic is approached from several angles: first, by reviewing the available literature on sound-color relationships in science and the arts; second, with an experimental study designed to collect quantitative data on how musicians and non-musicians associate sound and color; and third, by composing a new work that draws on both the long history of cross-modal interactions in human society and utilizes the collected data to musically elaborate on the subject at hand. The interdisciplinary nature of these goals will bring together a wide array of knowledge from musicology, neuroscience, cognitive psychology, and composition. While most of the individual components of this study are not unique (i.e.,
13 scientific research on sound-color matching has been conducted before, and many composers have used cross-modal associations and synaesthetic visions as inspiration), the present study is distinct in its merging of previously disparate areas. A major goal of this work is to encourage further interdisciplinary collaboration by showing how each area of research and creativity can contribute to the understanding of this captivating human capacity.
14 CHAPTER 1 INTRODUCTION Victorian-Era Curiosity into Human Sensory Systems Beginning in the second half of the nineteenth century, a flurry of articles appeared in the scientific literature regarding a strange sensory condition called synaesthesia (Calkins, 1893, 1895; Galton, 1880, 1883; Langfeld, 1914, 1915; Rose, 1909). Individuals with synaesthesia reported to researchers of their colored hearing, colored alphabets, and even colored days of the week. These were not mere associations or mixed memories, but actual sensations experienced by the synaesthete: stimuli in one sensory domain triggered vivid perceptions in an atogether different modality. These perceptions occurred rapidly and consistenly, appearing to be unavoidable and seemingly hard-wired in the individual who experienced them. Further, synaesthesia was persistent across the lifespan: many synaesthetes reported that they had always experienced such perceptions, and that it was not linked to focal events such as stroke or emotional trauma. Prior treatises in a host of disciplines had drawn comparisons between the senses, but the introspective phenomenon uncovered by Victorian-era scientists exceeded what could be understood as mere analogy: deeper investigation on a scientific level was warranted. Credit for bringing synaesthesia into the focus of pre-20th century psychology is attributed to the noted English polymath Francis Galton, who meticulously tabulated cases of synaesthesia and cross-modal associations, and is credited with introducing the topic to a wide audience in his 1883 book Inquiries into Human Faculty and Its Development. This publication drew much attention, and research began in a variety of locations in Europe and the United States. Unsure of how to study the internal states of the human mind, much of this early empirical research was conducted through testimonial reports either by the researchersynaesthete himself or gathered through correspondence with synaesthetes.
15 Music was often a subject of interest in these early studies, with several researchers explicitly asking if respondents saw colors when they heard music, a question that yielded high numbers of chromaesthetes, that is, individuals who experienced color perceptions in response to music. One such survey was conducted at Wellesley College in 1895, with researcher Mary Whiton Calkins documenting 64 cases of music-color synaesthesia among a total of 212 purported synaesthetes in the all-female campus population (Calkins, 1895). The ensuing years of the earlyand mid-twentieth century saw the downturn of Victorian-era inquisitiveness and the rise of behavioral psychology, which deemphasized internal mental states. Scientists largely abandoned scientific work on synaesthesia in the years following World War II. Artists and composers, however, continued to explore the expressive possibilities of cross-modal associations, acquiring further empirical and subjective evidence that the five senses, at least when it came to aesthetics, are, to some extent, interactive and not entirely modular. From the late 1970s to today, the relationship between the senses has continued to fascinate artists and has resurfaced as a topic for scientific exploration. Many human faculties previously thought to be unresearchable are now accessible to scientists through firmly established methods in cognitive psychology and neuroimaging. Living tissue can be examined through functional magnetic resonance imaging, which can be used to infer in vivo mental processes without invasive procedures. Cross-modal experience (and synaesthesia in particular) has reemerged with the rise of these techniques, and has recently been approached as an important window into human experience. For recent reviews on this wide-ranging literature, see Hochel (2008), Ward (2008), and Smilek (2008).
16 Contemporary Notions of Cross-Modal Perception Several truisms bear mentioning when approaching synaesthesia and cross-modal perception through the lens of modern science. First, human beings experience the world around them through finely developed sensory systems which are traditionally described in five distinct domains: the visual, auditory, gustatory, olfactory, and tactile. Second, these senses give each individual access to the surrounding environment, the ability to communicate with others, and the means to acquire knowledge and experience. Third, the human body is constructed to both send and receive signals in each of these sensory modes, as evidenced by the dualities of speech/hearing, sight/movement, smell/pheromones, and so on. Finally, the human brain has the capacity to continuously process multiple, parallel sensory information streams. For example, one might focus solely on the voice of another individual on the telephone or engage several senses at once while watching a movie: sight (toward the screen), hearing (to the sound), and perhaps taste (popcorn and soda) and tactile (holding hands with a date; feeling low frequency sounds from the theater subwoofer). Each of the sensory modalities contributes to the experience as a whole, and is critical in shaping that experience. As Andrew King writes, The capacity of the brain to coordinate the different sensory signals that arise from a common source provides us with a unified percept of the world (King, 2004). What is not immediately apparent, though, is that the senses have more complex characteristics, perhaps the most striking of which mirrors what fascinated Victorian-era psychologists about synaesthetes: human beings appear acutely capable of integrating and associating their sensory perceptions. For example, human beings can generally view printed text and hear it internally, while simultaneously being able to produce the motor movements necessary to generate the corresponding sounds aloud. Musicians with high levels of training often engage in armchair score reading, where a full orchestral score can be heard internally
17 though no music is sounding in the external world. This skill, called audiation (Gordon, 1989) as well as other capabilities present in musicians such as the ability to sight-read, are forms of sensory integration involving visual, auditory, motor, and cognitive facilities. For example, Hubbard and Ramachandran (2001, 2005, 2006) have argued that crossmodal connections are not only inherent in mans sensory system, but also essential in forming the capacity for creativity and artistic expression. For these scientists, mans creative drive arises from latent, biological interconnections between sensory modes, and that each of the discrete senses, when coupled, form a single gestalt system that exceeds the capacity of each individual element. Further, they argue that this single system is responsible for humanitys ability to engage in metaphor, which is the logical precursor to creativity and artistic expression (E. M. Hubbard & Ramachandran, 2005; Vilayanur S. Ramachandran & E. M. Hubbard, 2001; Ramachandran & Hubbard, 2006). Ramachandran and Hubbards theory is also capable of examining associative relationships. For example, literary theory views poetry as much more than words: sound, rhythm, timbre, and imagery are core poetic devices that transcend mere orthography and are part of essence of poetic art. Eric Zillmer and colleagues inclusion of Arthur Rimbauds poem Vowels1 when discussing synaesthesia demonstrates this poignantly. In the poem, Rimbaud describes the vowels of the English language as having both tactile and color characteristics. The authors inclusion of this poem is revealing, and offers a powerful example of the intimate relationship between cross-modal association and the arts. While Rimbaud himself may have been an actual synaesthete, it is the readers ability to connect the seemingly disparate sensations of touch and color perception to the text. Rimbauds presentation of these inter-sensory elements 1 Rimbauds Vowels is readily available on the Internet, one such reproduction of the poem can be found at http://www.doctorhugo.org/synaesthesia/rimbaud.html.
18 enhances the aesthetic enjoyment of the poem and raises the text to poetic levels. In Ramachandran and Hubbards framework, biological interconnections underlie this ability, providing the gateway to synaesthetic association. Given humans ability to create, experience, and perceive sensory information across modalities, music, an artistic medium traditionally thought to be solely auditory, may include deeper characteristics that can be uncovered through systematic inquiry. The existence of chromaesthesia, a condition that causes individuals to experience colored photisms in response to music, as well as the wide array of common, cross-modal musical terms like the Blues, andante (walking speed), and dolce (sweet), present strong evidence that can motivate more systematic study of the relationship between sound and other sensory modes. About This Study The present study explores the interconnectedness of sight and sound, with a particular focus on color and music. This core topic will employ three separate approaches: first from a historical standpoint, including a discussion of how artists, musicians, and others have engaged the relationship between sound and color; second, through an experimental study aimed at collecting quantitative data regarding sound-color associations in musicians and non-musicians; and third, by using the collected data as a compositional tool to create a new work intended to more deeply explore sound-color relationships from an artistic standpoint. In order to focus on the particular topic of the relationship between sound and color, it is necessary to frame the boundaries of the present study. The central topic of this document is cross-modal association; a generic term intended to acknowledge a cognitive mechanism that enables human beings to process and gain awareness of the relationships between sensory modes (J. Ward, Huckstep, & Tsakanikos, 2006). By contrast, multisensory integration is understood to be an automatic process that arises due to physiological characteristics of the human body,
19 namely the nervous systems ability to modulate and couple incoming sensory information from disparate modes (E. M. Hubbard & Ramachandran, 2005; King, 2004). Cross-modal perception refers to specific neurological situations where two senses couple into a single percept or where one sensation substitutes for another (Brugger & Weiss, 2008). Synaesthesia is the extreme example of both cross-modal perception and association: synaesthetic individuals experience sensory responses in one modality due to input from another modality (Baron-Cohen & Harrison, 1997; Galton, 1883). Synaesthetic sensations, though they are not actually present in the exterior world, are vivid and consciously perceived by the synaesthete and are typically systematic, highly consistent, and automatic (Cytowic, 1997), as well as tightly bound to other cognitive systems like emotion and memory (Bargary & Mitchell, 2008; Cutsforth, 1925; Kadosh & Henik, 2007). Cross-modal perception, multisensory integration, neurological synaesthesia, and soundcolor associations are separate, specific topic areas that are tightly connected. A review of the literature of any single one would reveal intertwined histories and common features among them. As a matter of course, it is most logical to encompass these four areas under a single broad term that acknowledges their interconnectedness yet allows individual discussion. In this document, that term is synaesthetic behavior. This term, specific to the present study, is intended to encompass the repertoire of human capabilities that span the senses, and uses the term synaesthetic as an analogy and not to specify a neurological condition. Further, use of the word behavior is not intended to indicate physical action (i.e., hunting, writing, making music, etc.) but as a descriptive word, e.g., the human sensory system behaves synaesthetically. To maintain a streamlined approach to synaesthetic behavior, the present study will not focus on theoretical neuroscience, philosophy, or abnormal psychology. Though these are
20 certainly areas of interest with much relevant information, the present study assumes that some aspects of synaesthetic behavior (e.g., chromaesthesia) reflect an innate, normal human capacity. This capacity appears to varying degrees in many works of art and has been documented in several studies that will be outlined in Chapter 2. Further, synaesthetic behavior (in its variety of forms) has contributed greatly to the production of art, and arises due to experiential, educational, physiological, and cultural factors. Though the approaches offered by other disciplines above are equally valid, this study will examine synaesthetic behavior for its own sake, and not attempt to explain or track potential roots in philosophy or physiology. The present study does not intend to theorize about how neurological synaesthesia might contribute to artistic expression, nor how perceiving art might be helped, hindered, or otherwise modified in synaesthetes who vividly experience their unique sensations, though it does consider cross-modal association in terms of music perception, as discussed in Chapter 6. Likewise, acknowledging the probable cultural influences on cross-modal association is critical to deeper understanding of this topic, but this area is far too broad to cover in this document. However, it is hoped that this document will prompt social scientists to begin studying synaesthetic behavior outside of the realm of Western society to obtain a richer view of the interactions between culture, creativity, the brain, and the body. By engaging the topic in this way, it is possible to give historical, artistic, and scientific merit to the answering of several questions: first, are normal cross-modal associations subject to training effects in a particular aesthetic realm (in this case music); and second, to what degree does a persons latent cross-modal associations influence artistic expression (such as musical behaviors like composition and performance) and perception (i.e., listening to a work incorporating actual scientific data)? Finally, the present study will attempt to contextualize both
21 the scientific and musical findings in terms of how synaesthetic behavior may fit into existing models of musical perception, specifically the framework of primary and secondary illusions in art and music proposed by Susanne Langer, discussed in Chapter 6. While similar research has been conducted in all of the aforementioned areas of study (as described in Chapter 2), the present study contributes to the current literature by gathering a wide array of knowledge in a single document, by comparing the sound-color matching responses of both musicians and nonmusicians, and by attempting to frame its findings in terms of both a philosophical model and a new creative work. In this way, the present study aims to merge previously disparate areas of inquiry, and encourage further interdisciplinary engagement in synaesthetic behavior which, as this document demonstrates, is an important human capacity that warrants deeper collaborative exploration. Overview of the Chapters The present study will address synaesthetic behavior across the space of history, engage in a quantitative study of cross-modal associations, and attempt to utilize both the artistic and scientific oeuvre as the inspiration for a new musical work. Chapter 2 will review the available literature and focus on music-related topics such as chromaesthesia and the extensive history of artistic fascination with and scientific gravitation toward synaesthetic behavior. This literature acts as the impetus for the quantitative study outlined in Chapter 3, which presents the experimental procedures. Chapter 4 presents the results of the study, and outlines the data analysis and statistical correlations. Chapter 5 describes the creative process used to compose the new musical work. Finally, Chapter 6 will attempt to interpret the compositional, historical, and scientific findings presented in this study into a common framework, and discuss possible ramifications of these findings.
22 CHAPTER 2 REVIEW OF THE LITERATURE The Ancient World Etymologically, the term music arrived in 12th-century English via the French language. The word was systematically borrowed into disparate language families from French and Spanish, and arrived in non-European languages (such as Arabic) by the seventeenth century, though a number of cultures that adopted the term had no previous concept of music as it existed in Western culture (Nettl, 2001). Today, the bulk of the worlds languages contain some cognate of this word, all of which derive from the ancient Greek mousik#, which referred to any work inspired by the nine muses, musical or otherwise (Nettl, 2001). The notion that the five human senses (the visual, auditory, olfactory, gustatory, and tactile) are not entirely discrete mirrors the ambiguity of the Greek term. Much in the same way that Greek society recognized common threads among seemingly disparate arts, at least enough so to warrant using a single term for them, notions about sensory intermixing consistently reappear in many scientific and artistic works across several centuries. Beginning with Pythagoras documentation of the mathematical equations responsible for ancient musical tuning, the relationship between sound and numbers offers a surface-level glimpse into cross-modal relationships (Day, 2001; Van Campen, 1999). Ancient philosophers expanded rudimentary mathematics into larger-scale perceptual constructs, the most prominent of which is Pythagoras music of the spheres, an idea later adopted by Plato that uses musical nomenclature to describe the motion of celestial bodies. Effectively, Pythagorean and Platonic philosophers established cross-modal views between motion, numbers, and sound. These philosophers also described musical intervals in terms of color quite specifically: black and white pigments were blended along the same ratios as musical intervals, resulting in shades of
23 grey that corresponded to each possible sonority (Jewanski, 2001). Influenced by these conjectures, the philosopher Democritus would propose a doctrine of sensory unity in the 5th century BCE, when he theorized that all senses operate in the same way: that is, via atoms coming into contact with the body (Marks, 1978). Though this notion ultimately proved to be false, the idea of uniformity in sensory perception would persist throughout history. The ancient view that analogous relationships existed between the senses, and that those senses might operate in similar ways, would remain present in every future cross-modal endeavor. Aristotle continued the trend, noting the apparent similarity between colors and sounds. In particular, he observed that colors have the tendency to agree or disagree with one another much like musical notes which, when sounded simultaneously, create consonant and dissonant intervals. This colored view of musical harmony appears in De sensu et sensibilibus, where the great philosopher directly correlates musical and visual harmony. Clearly, even in the earliest documented period of human civilization, philosophers, mathematicians and astronomers saw metaphorical relationships to music in decidedly non-sonic contexts (Day, 2001), further solidifying the notion that human perception is inherently cross-modal. Many ideas formulated in the ancient world would persist in Western society well into the 17th century, and sound-color relationships are no exception. Since the ancient Roman and Greek civilizations, it appears that thinkers have found much to explore in the relationship between color and sound. Historical Examples of Synaesthetic Behavior in Western Culture The notion of interaction between the senses repeatedly reappears at the crossroads between art and science over several centuries of Western culture. John Locke, likely following the notions of Aristotle, asserted how the senses assist one another (Locke, 1685; Marks, 1978), and offered an early example of purported synaesthesia in 1690 while describing a blind man
24 who came to understand the color scarlet as like the sound of a trumpet (Baron-Cohen & Harrison, 1997; Marks, 1978). The English ophthalmologist Thomas Woodhouse reported another blind individual with sound-color sensations in 1710 (Larner, 2006; Marks, 1978). Evidence of touch-smell -color cross-modal perception also appears in Jonathan Swifts Gullivers Travels, published in 1726. Though a work of fiction, it is believed that Swift was aware of the scientific trends of the time and may have taken inspiration from them (Larner, 2006). One possible source for this inspiration may have been a report of a blind individual who could differentiate between pigment colors based on touch and smell by Robert Boyle, a 17th century fellow of the Royal Society. This report, which arrived to Boyle from fellow physician Sir John Finch, predates both Lockes and Woodhouses reports (Larner, 2006), indicating that some knowledge of synaesthesia was present in the scientific community before 1700. Certainly Swift and many others would have been aware of such reports. Though this particular circumstance has been challenged in terms of its relevance as actual synaesthesia (Brugger & Weiss, 2008), it is clear that synaesthetic behavior and attention to the cross-modal was indeed part of the Zeitgeist from the 17th to the 19th centuries and beyond (Marks, 1978). Musicians have also shown a keen interest in topics such as color-music, further demonstrating that intersensory relationships were present in the collective consciousness before the 17th century. Spurred by Aristotles influence, composer/theorist Gioseffo Zarlinos 1558 treatise Le Istitutioni Harmoniche related musical unisons and octaves to white and black, with the intermediate colors of red, green, and blue assigned to the consonances interspersed therein (Jewanski, 2001). Jesuit scholar Athanasius Kircher advocated deeper associations in the ensuing century, formulating complex tables to show the relationship between pitches, colors, and brightness, as well as a system of musical interval to color relationships (Jewanski, 2001).
25 Even Louis XIVs physician, Marin Cureau de la Chambre, developed these ideas further by the mid-17th century with his Systme des Couleurs et des Harmonies, which assigned pairs of notes to pairs of colors based on mathematical proportions (Jewanski, 2001). Because Western Europe revered Aristotles ordered models of the world, and Greek society in general was admired and imitated to varying degrees of accuracy, many of these artistic and philosophical approaches arose out of ancient mathematicians and philosophers desire to view the world as ordered and logical (Jewanski, 2001). Western culture was merely following suit, striving to discover relationships in the physical world that upheld these qualifications. Analogical explanation of nearly everything from astronomy to musical tuning can be found during the development of Western philosophy. The Music of the Spheres was not merely poetic, but a serious attempt to systematize the ratios underlying celestial rotations and musical intervals into a single codified construct. Efforts toward mathematical order continued to trump aesthetic relationships leading up to the turn of the 18th century. Even Isaac Newtons Opticks contained reference to the similarities between colors, musical scales, and physical motion, mirroring of Aristotles systematic model (Jewanski, 2001; Marks, 1978; Van Campen, 2008). However, Newtons work catalyzed a slow shift, leading to new notions about the relationship between sound and color in Western society. The ancient geocentric, highly ordered view of the universe had disappeared, and when coupled with Jean-Phillipe Rameaus 1722 assertion that the chord (rather the scale or mode) was musics central building block, musicians and scientists began to examine the relationship between sound and color from fresh perspectives. The cosmic and mathematical reference points described by earlier philosophers and scientists were gone, shifting the foundation of centuries-old philosophies that had influenced the arts and science up to that point (Jewanski, 2001).
26 These changes freed music theorists like Louis-Bertrand Castel to draw new kinships between music and visual art. Castel proposed that single colors, when used in tandem, created images much in the same way that single pitches created harmonies (Jewanski, 2001). Castel, living in the era of Newton and Rameau, in combination with the knowledge of synaesthesia he may have garnered from his close friend Thomas Woodhouse (Marks, 1978) and the influence of Kircher (Franssen, 1991), constructed an optical harpsichord around the year 1720. Each key of the instrument operated a pulley to move a small curtain, allowing a beam of light to pass through a piece of stained glass (or possibly paper) and shine through to the audience (Brougher, Strick, Wiseman, & Zilczer, 2005; Van Campen, 1999). Both Rameau and Georg Philip Telemann followed suit, working on their own optical harpsichords. Telemann found the commonalities between color and sound poignant, observing that a fugue in sounds will make up a fugue in colors (Jewanski, 2001). Though documentation on Castels clavecin oculaire is limited, and no surviving instrument exists, his experiments represent the first serious engagement by a musician to directly relate sound and color. Castels instigation of color music may have had a dramatic influence on the eventual rise of Romanticism and todays interest in visual music, notions that are extensively well-documented by Franssen (1991). As the 19th century approached, artists exhibited further interest in synaesthetic behavior, since it seems to validate the belief in the primacy of imagination in human cognition, as well as to ratify the original wholeness, continuity, and interfusion of immediate experience (Dann, 1998), concepts that lend directly to the central tenets of Romanticism. According to Barry Stein and M. Alex Meredith, during the 19th century it was fashionable among those in polite society to have had synaesthetic experiences (Stein & Meredith, 1993). Composer Robert Schumann, among other 19th century artists, revisited Castels ideas, arguing that composition and painting
27 share a common aesthetic and are only variable in the nature of the physical materials (Jewanski, 2001). Perhaps Frederick Kastners pyrophone color organ exhibits the Romantic-era interest most poignantly in its very construction: the instrument used gas lamps to illuminate crystal tubes, capturing the Romantic spirit quite literally. The pyrophone used the very fire of the 19th century to engage audiences (Van Campen, 1999). Clearly, in musical realms, connections between the visual and auditory domains existed in the Baroque, Classical, and Romantic periods. This historical canon would continue to spark interest as the turn of the 20th century approached. Scientific Angles: from Francis Galton to Behaviorism Medical student George Sachs 1812 dissertation included a description of his personal synaesthesia in the closing chapters, marking the beginning of interest in synaesthesia among scientists in the 19th century (Dann, 1998; Van Campen, 1999; J. Ward, 2008). Sachs commentary was particularly poignant, as he himself was an albino: the young student saw the world in vivid colors while he himself lacked pigment. Sachs writing, possibly influenced by Goethes 1810 Theory of Colors (J. Ward, 2008), marks a turning point in cross-modal exploration when the subject would begin to move from a mere curiosity and artistic fascination to a topic ripe for scientific study. By the 1870s, medical journals had presented several cases of synaesthesia, with physicians positing theories as to the origin of such a phenomenon. The dominant theory of the time was that synaesthesia was a rare pathology of the visual system (Dann, 1998). Research into cross-modal sensory perception and synaesthesia would not entirely congeal until a half century later when Francis Galton began his work on the subject. Galton, a cousin of Charles Darwin and noted polymath of the Victorian era, is credited with the invention of fingerprinting and noted for his extensive work in meteorology, statistics, and psychology.
28 The common thread in all of Galtons work was the activity of measurement, and much of his work concerns itself with quantitative documentation of various human faculties. This activity, now known as psychometrics, forms a cornerstone of modern psychological study. Galton collected qualitative evidence of cross-modal interactions, beginning with his own personal color associations and branching into written correspondence with his subjects, who documented their intersensory perceptions. Galtons body of work includes a myriad of examples, including letters from individuals who saw simple visual forms as colored though no color was present. This type of synaesthesia, now commonly known as grapheme-color synaesthesia, opened the door for Galtons further exploration of sensory interplay. In Inquiries into the Human Faculty and Its Development, Galton discusses this and several other types of synaesthesia he had observed in preceding years, including number-color, form-color, and date-color, finally progressing to sound-color (Galton, 1883). Galton notes that instantaneous association of colour with sound characterizes a small percentage of adults, and it appears to be rather common, though in an ill-developed degree, among children (Galton, 1883). Galton makes striking observations on this subject that culminate in his summary: As my present object is to subordinate details to the general impression that I wish to convey of the peculiarities of different minds, I will simply remark First, that the persistence of the colour association with sounds is fully as remarkable as that of the Number-Form with numbers. Secondly, that the vowel sounds chiefly evoke them. Thirdly, that the seers are invariably most minute in their description of the precise tint and hue of the colour. They are never satisfied, for instance, with saying blue, but will take a great deal of trouble to express or to match the particular blue they mean. Fourthly, that no two people agree, or hardly ever do so, as to the colour they associate with the same sound. Lastly, that the tendency is very hereditary (Galton, 1883). The impetus to connect sound and color in Galtons work arose primarily from his observations regarding vowel sounds. Mirroring Rimbauds poem Vowels (discussed later) in which the poet describes the tactile and colored characteristics of his native French vowels,
29 several of Galtons subjects describe phoneme-color associations with poetic detail. Galton acknowledges the connection to poetry in documenting how one of his correspondents mapped speech sounds to colors, writing The patterns are to him like words in poetry, which call up associations that any substituted word of a like dictionary meaning would fail to do (Galton, 1883). Perhaps Galtons most significant contribution to the study of synaesthesia appears in his discussion of how cross-modal mappings arise in individuals. Galton noted that the perennial nature/nurture problem, where psychologists strive to determine if genetic or environmental factors are responsible for human behavior, was difficult to solve with regard to synaesthesia. The widespread incidence of color associations indicated some degree of natural influence, whereas the variance in perceptions between individuals seemed to indicate some cultural, familial, or contextual contributions (Galton, 1883). The whole of Galtons publications on the subject served to fuel curiosity across Europe and the United States, carrying synaesthesia and cross-modal associations from mere analogies and rare pathologies to a legitimate topic for scientific inquiry. The flurry of articles published in the period following Galtons initial observations spans the range of possible psychological study during that era. Published articles include accounts of single subjects that simply documented and retested the synaesthetes perceptions after a considerable time interval (Langfeld, 1914), to statistical and qualitative study of large numbers of individuals in search of synaesthetic tendencies (Calkins, 1893, 1895; Rose, 1909). One author even meticulously documented his personal, wide-ranging (and difficult to believe) synaesthesia, whose onset occurred after a year of being physically rundown and nervously unstable (Ginsberg, 1923).
30 Researchers continued to examine the nature of synaesthetic percepts and perpetuated Galtons search for an answer to the nature/nurture problem by examining synaesthetic behavior in children. The three resulting theories (Langfeld, 1915, 1926) persist in contemporary views on why human beings possess the ability to interrelate their senses. More specifically, synaesthesia could be viewed as a physiological phenomenon of neural cross wiring, an adaptive advantage that contributed to survival over the course of evolution, and a mechanism that contributes to communication, meaning, and understanding. Subjects like these, which would later become cornerstones of cognitive psychology, also foreshadowed the synaesthesia and cross-modal sensory perception research that occurs today. For example, Thomas Cutsforths subject Miss E reported complex, multifaceted synaesthesia, including chromesthesia tied strongly to the timbre of the sound. Each of Miss Es color perceptions was mediated by an emotional response to the various stimuli. As Cutsforth reported, the same colors, under different interpretative mental sets, function as emotion, as tactual, or as auditory perceptions (Cutsforth, 1925). He surmised that Miss Es emotionallymediated color perceptions harkened back to early childhood, and that the responses seen in Miss E could be equally true of all individuals (Cutsforth, 1925). Cutsforth also studied synaesthesias role in the process of reasoning, contrasting two synaesthetic subjects with a nonsynaesthetic blind control subject (Cutsforth, 1924). Ultimately, Cutsforths work focused on meaning and how synaesthesia might assist in its resolution, a notion that persists today in discussions of the cognitive advantages afforded by cross-modal perception. Early 20th century researchers also turned to music as a vehicle for uncovering the characteristics of synaesthetic behavior. A host of articles mentioning the color of music appeared during these decades (Calkins, 1893, 1895; Ginsberg, 1923; Langfeld, 1914), with
31 some rather idiosyncratic descriptions arising. Mary Collins synaesthetic subjec t S reported that Bach and Beethoven are brown in addition to the typical pitch-color mapping that chromaesthetes report (Collins, 1929). Clearly, even decades before the cognitive revolution, topics such as attention, memory, mental imagery, reasoning, and emotion played into synaesthesia and cross-modal research as interpretive themes. The trend toward behavioral analysis that would arise in the early 20th century, with its strict avoidance of internal mental processes and subjective experience, would result in a downturn in the study of synaesthetic behavior as engaged by Victorian and early 20th century researchers. The number of published articles on the subject is representative of the decline, with less than forty articles published between 1930 and the late 1970s (Van Campen, 1999). However, while the number of psychological investigations declined, artistic interest in crossmodal perception as a means of expression persisted as the 20th century progressed. Synaesthetic Elements in Music and Art: The 20th Century Despite the downturn in psychological study of synaesthetic behavior, fascination with cross-modal perception has lingered in music and other arts since the turn of the 20th century. The topic almost cyclically reasserts itself over the last 150 years, likely as a result of scientific interest during the 19th century and the desire of artists and musicians to find new modes of expression after the turn of the 20th century. Romantic-era artists championed synaesthetic perception as an example of heightened consciousness, and exalted those affected with the condition as proof that typical ways of experiencing the world could be exceeded. Synaesthesia, in such a cultural climate, appeared to validate the search for paranormal, transcendental experiences (Dann, 1998) and evoked the possibility of greater wealth of sensory and emotional experience (Ione & Tyler, 2004). Richard Wagners concept of the Gesamkunstwerk, a kind of mega-opera that incorporated
32 music, dance, architecture, poetry, and visuals, also sought to break free from compartmentalized forms and offer these transcendental experiences to concertgoers. Attention to the cross-modal elements of art were cultivated as a result, codifying concepts like Wagners leitmotif, where musical snippets are used to identify characters over the space of an opera. All of these elements sought to glorify the sensory experience, and foreshadowed the infatuation of later Romanticism, from French Symbolism to Haight-Ashbury psychedelic culture (Dann, 1998). Romanticism began to equate the capacity for color hearing with artistic sensitivity and even genius, a notion that would reemerge as a component of 1960s counterculture (Dann, 1998). Examples of composers exhibiting synaesthetic behavior in the late Romantic and early 20th century abound. Alexander Scriabins work Promethius: Poem of Fire is often mentioned as evidence of the composers synaesthesia: Scriabin included a line in the score dictating how colored lights should be presented as an accompaniment to the music. While this example may demonstrate Scriabins sensory perceptions, the notion that he was a true synaesthete has been challenged, citing the odd arrangement and octave equivalence in his color-pitch system (Dann, 1998; Peacock, 1985), features that nearly no other chromaesthete shows under controlled research conditions. Eventually Romantic ideals would dissolve, yielding to shifts in artistic doctrine during the 20th century. As the arts became increasingly non-representational, fascination with synaesthesia only amplified. Arnold Schoenberg, arguably the most influential composer of the early 20th century, included a color crescendo sequence in the third scene of his 1910 work Die glckliche Hand. The sequence is painstakingly planned in the composers notes, progressing from pale red to light blue (Brougher et al., 2005; Van Campen, 1999). Abstract musical
33 compositions such as these would make use of not only sound-color agreement, but disagreement as well, adopting synaesthetic dissonance (Van Campen, 1999) much in the same way it adopted musical dissonance. Sounds relationship to color remained of interest to musicians and composers throughout the 20th century as artistic doctrines trended toward extreme abstraction. Much in the same way that Romantic-era artists saw synaesthesia as a rich, sublime inner perception, artists in the 20th century saw it as a way to achieve complete non-representation. Considering the breakdown of tonality, movements such as aleatoric and serial composition, and the injection of technology into the creative arts, the exalted, often imaginary inner synaesthetic world offered a powerful space in which to create aesthetic experiences. These movements created circumstances where musicians became more deeply interested in timbre (also known as tone color), often using it as the driving force behind compositions (as opposed to pitchor harmony-centered music). Schoenbergs term klangfarbenmelodie, or tone-color-melody, is a prime example of this. Works that demonstrate klangfarbenmelodie typically distribute contiguous musical materials rapidly between instruments, creating a single line that shifts in color as it progresses. Abstract approaches to sound are not always synaesthetic in nature. Some composers, particularly ones working in acousmatic or spectral styles, adopt the doctrine that sound and how it changes over time. Proponents of this style refer to the transformation of sound over time as spectromorphology (Smalley, 1997), and argue that it is the only relevant aspect of music: visual imagery and other points of reference that the music draws up are irrelevant. At the same time, such music often pairs well with abstract video presentations and animation, and composers presently working with technology are not limited to working with sound alone, increasing the salience of cross-modal thought in the contemporary mindset.
34 The visual art world saw many of the same dramatic changes that music experienced in the 20th century. Abstraction offered artists an opportunity to forego representational work and focus on kinetic nonrepresentational art akin to pure instrumental music (Brougher et al., 2005). Much nonrepresentational art adopts the principles of harmony, texture, color, and hue in much the same way that music does. Though the non-representational tradition represents an alternative thread in the history of abstract art as a whole, it remains an important one given the influence imparted by the artists who practiced it. The term visual music was coined by Roger Fry in 1912, described visual art that (gave) up all resemblance to natural form, and (created) a purely abstract language of form a visual music (Brougher et al., 2005). The many artists purveying visual music offer further insight into synaesthetic thinking as both a neurological and analogical condition: Wassily Kandinsky, likely a true synaesthete (Ione & Tyler, 2004; Kemp & Blakemore, 2006), wrote of the influence of Schoenberg on his work, and often named his paintings with musical titles such as Fuga (fugue), Concert and Composition. Kandinsky, also a sometime composer, frequently concerned himself with the musical concepts of consonance and dissonance, and incorporated synaesthetic agreement and disagreement in several of his visual and theatrical works (Van Campen, 1999) much in the same way that Schoenberg and his followers liberated musical dissonance. Paul Klee, an artist who would influence the composer Gunther Schuller in his work Seven Studies, often used musical analogies as well, with works like New Harmony and Nocturne for Horn. Piet Mondrian explored the synaesthetic relationship between static images and rhythm in works like Broadway Boogie Woogie (Van Campen, 1999). Georgia OKeefes works included musical titles, as did the works of a host of 19th and 20th-century artists like Arther Dove, Thomas Wilfred, Hans Richter and many others (Brougher et al., 2005; Zilczer, 1987).
35 In-depth discussion of these artists is beyond the scope of this document, but demonstrates that the interplay between the visual and auditory remains very much alive in the world of painting. Brougher and colleagues book Visual Music (2005) presents a complete and fascinating exploration of this topic. The book presents countless works of visual art whose intentions are musical in nature, by artists who both experienced neurological and otherwiseinspired synaesthetic perceptions. Recent exhibitions at art galleries and museums around the world have presented the work of synaesthetic artists, two examples of which include a 2006 exhibition of Kandinskys work at the Tate Museum in London (Kemp & Blakemore, 2006; O. Ward, 2006) and a collection of synaesthesia-inspired work at the Hirshhorn Museum in Washington, D.C. (Brougher et al., 2005). Contemporary Scientific Research: Synaesthesia Reemerges Contemporary research into synaesthesia and cross-modal relationships commonly approaches the topic in much the same way the present study has: by framing it in historical, artistic, and scientific terms. Current researchers have revisited synaesthetic behavior in terms of both their predecessors and through new research techniques informed by cognitive science and medical imaging technology. The convergence of history and these new techniques have resulted in both new data and new theories that mark a significant resurgence in research into synaesthetic behavior. Synaesthetic behavior, in its variety of forms, is currently theorized to be formative in a variety of contexts including artistic expression, the development of language, and the nature of aesthetics. Humans have access to rich interconnections among their senses, and this capability can result in powerful aesthetic experiences. As Lawrence Marks writes, it is little wonder, then, that descriptions and expressions of sensory correspondence appear, occasionally even a s doctrine, in the arts and in literature as well as science (Marks, 1978).
36 Marks seminal work The Unity of the Senses: Interrelations among the Modalities explores this topic deeply and emphasizes how the study of synaesthesia might provide a neurological basis for metaphor. He approaches the subject through analysis of how sound acts as a symbolic mechanism in poetry, as well as proposing five distinct doctrines that underlie cross-modal processes, ranging from information equivalency gleaned through separate sensory modes to common neural and psychophysical substrates (Marks, 1978). Marks theories, though not universally agreed upon2, represent the beginning of the contemporary resurgence of scientific interest in synaesthetic behavior. His work catalyzed serious scientific inquiry into cross-modal relationships and synaesthesia and marked a rediscovery of many of the same mysteries elucidated in previous eras. Anatomical Substrates, Brain Plasticity, Psychedelic Drugs, and Neonatal Synaesthesia Contemporary explorations into the individual experiences of synaesthetes have brought to light several possible anatomical correlates of synaesthesia (Hnggi, Beeli, Oechslin, & Jncke, 2008; E. M. Hubbard & Ramachandran, 2005; Nunn et al., 2002; Vilayanur S. Ramachandran & E.M. Hubbard, 2001; Sperling, Prvulovic, Linden, Singer, & Stirn, 2006). Based on this work, it is generally accepted that true synaesthetes, while possessing curious neural interconnections in their sensory systems, are not hallucinating, addicted to drugs, or expressing themselves in an excessively metaphorical way, nor do they belong in mental institutions (Day, 2005). Much to the contrary, some researchers have proposed that synaesthesia arises as a remnant of an important developmental process that begins at birth, when, as the theory states, the human brain is infinitely cross-wired and everything is connected to everything else. This so-called neonatal synaesthesia begins to fade during either the synaptic pruning process (in which 2 For counterpoints to Marks work, see Ward (2006), Cytowic (1997), and Van Campen (2008).
37 sensory neurons differentiate and tune themselves to incoming stimuli) or as a result of apoptosis (i.e., programmed cell death) (J. Ward, Huckstep et al., 2006). Researchers have proposed that this pruning process is only partially successful and we all, as adults, still possess interlinked senses and remnants of the type of synaesthesia that we experienced in infancy (J. Ward, 2008; J. Ward, Huckstep et al., 2006). Still other researchers have noted that neonatal synaesthesia appears to fade and reemerge as the infant becomes habituated to the environment around them (Maurer & Mondloch, 2005), lending credence to contemporary notions of brain plasticity: that is, the ability of the brain to reorganize itself based on both internal and external factors. Both theories strongly support contemporary theories on plasticity: either synaesthesia arises from interconnections that never stripped away in the infant brain, or new, specialized connections arise during adolescence (or perhaps both). These studies appear to verify Galtons initial observations about synaesthesia in children, showing that adult synaesthetes share some characteristics with infants and younger children (Kadosh, Henik, & Walsh, 2009) and offering insight into the possible anatomic underpinnings of synaesthetic behavior. Studies of other groups including blind individuals, both congenital and acquired, also support notions of plasticity both with regard to the senses and otherwise (Maurer & Mondloch, 2005), reasserting the plausibility of ancient observations by John Locke and others some three hundred years prior. On a more poignant note, researchers working outside of the realm of synaesthesia have noted morphological changes in the brains of musicians based on the age music training commenced, demonstrating substantial evidence that music training results in profound and durable effects on the brain (Habib & Besson, 2009). Researchers propose that the plasticity effects shown during music training occur through the study and repetition of
38 musical activities, which reinforces neural connections in auditory, visual, and motor areas. Cross-modal connections are among those that are reinforced, resulting in demonstrable anatomical and cognitive adaptation that may prove relevant in the study of cross-modal perception and synaesthetic behavior as well as rehabilitative therapies, particularly those that make use of multi-modal and cross-modal paradigms. While widely accepted that synaesthetes are not exhibiting some sort of neurosis, contemporary researchers have also recognized that the hallucinations experienced by psychedelic drug users strongly resemble the experiences of true synaesthetes. Several researchers have proposed that the effects of such drugs on the nervous system can inform study of synaesthesia and cross-modal relationships (Van Campen, 2008). From a cultural standpoint, 1960s and 70s psychedelia advocated that hallucinogenic drugs can open the door to enhanced perception, essentially unblocking the walls that separate the senses and expanding ones world view (J. Ward, 2008), mirroring the Romantic ideal of enhanced consciousness (Dann, 1998). In this way, drug use can be seen as much more than a cultural phenomenon. Documenting the experiences of drug users might contribute directly to both the potential biological roots and neural communicative capacities of synaesthesia. Psychedelic drugs like LSD, mescaline, and other hallucinogenic substances modulate the nervous system, potentially enhancing, exciting, inhibiting, or otherwise artificially modulating existing connections between differentiated sensory systems (Vilayanur S. Ramachandran & E. M. Hubbard, 2001). This drug-induced, overmodulated state of the brain often results in normal individuals experiencing perceptions that resemble synaesthesia (Ione & Tyler, 2004). However, both Cytowic and Ramachandran have pointed out that synaesthetic experience in drug users is not universal and may not occur during every episode of drug use (Cytowic, 2002; Vilayanur S. Ramachandran & E. M. Hubbard, 2001).
39 The Relevance of Synaesthetic Behavior With the wide ranging literature on the subject, it is clear that humans engage in synaesthetic behavior in both specific and generalized ways. Several theories on the relevance of this capacity have arisen from recent studies and the revisiting of available historical publications. For example, theories on cross-modal interactions and sensory integration have laid out a host of possible adaptive advantages provided by synaesthetic perception. As Jamie Ward writes, When a large number of [these] multisensory neurons work together they can influence our behavior and give us certain advantages that would not be possible if our senses were strictly segregated (J. Ward, 2008). The fusing of auditory and visual stimuli may aid in language perception, movement perception, and other sensory phenomena (Parault & Schwanenflugel, 2006; Ramachandran & Hubbard, 2006; Saenz & Koch, 2008; J. Ward, 2008). As a specific example, multisensory integration may prove to carry evolutionary adaptive advantages in that it allows rapid assessment of potential predators based on salient aspects of sound and color (Reilly, Rodriguez, McCabe, Biun, & Altmann, 2009). Studies of intermodal sensory characteristics are not limited to human beings, with much neuroanatomical and behavioral research on other species behavior under cross-modal circumstances. For an overview of this subset of contemporary cross-modal inquiries, see Barry Stein and M. Alex Merediths The Merging of the Senses (1993), which offers insights into several of the theories previously mentioned using cats, rodents, and reptiles. The Term Synaesthesia in Modern Research Even with anatomical models and the theorized adaptive advantages offered by sensory interconnections, many scientists remain hesitant to apply the term synaesthesia to all but the most specific type of cross-modal perception. To conclude that we are all synaesthetes is, to some, more of a cop-out than an explanation since synaesthesia has similarities to normal
40 multisensory perception, yet synaesthesia is not the norm (J. Ward, 2008). Still others posit that the synaesthetic metaphors present in everyday language and in the arts are the result of latent cross-wiring in particular cortical areas (Bargary & Mitchell, 2008; Vilayanur S. Ramachandran & E.M. Hubbard, 2001), and therefore may deserve to be referred to as true synaesthesia rather than synaesthetic or pseudo-synaesthetic. Richard Cytowic has offered diagnostic criteria specific to neurological synaesthesia that relies on five qualifications (Cytowic, 2002; Hochel & Miln, 2008), likely intended to establish a line beyond which the term can be properly applied: Table 2-1. Cytowics elements of synaesthesia (Cytowic, 2002; Hochel & Miln, 2008) 1. It is involuntary and automatic. 2. It is consistent and generic. 3. It is spatially extended. 4. It is memorable. 5. It is affect-laden. While these criteria appear straightforward, all five easily apply to everyday cross-modal interactions. The continued disagreement on terminology is most markedly demonstrated by Cretien van Campen, who reports that he has found it hard to distinguish synaesthetes from non-synaesthetes by scientific methods (Van Campen, 2008). Van Campen, with this statement, acknowledges that while there are clear distinctions between synaesthesia as a neurological condition, synaesthesia as an artistic doctrine, and normal cross-modal sensory integration as many have asserted (Baron-Cohen & Harrison, 1997; Cytowic, 1997; J. Ward, 2008), it is difficult to disconnect these topic areas from one another (Stein & Meredith, 1993). In an attempt to resolve the disconnect, Jamie Ward has proposed a model of synaesthesia that takes into account a variety of contributing factors including drug use, acquired changes in the brain, and components of the learning proces s (J. Ward, 2008). Wards model offers a compromise that balances the potential influences of cultural context (Van Campen,
41 2008) with both developmental influences and proposed neural models (Hnggi, Beeli, Oechslin, & Jncke, 2008; E. M. Hubbard & Ramachandran, 2005). Other authors have found little resolve in attempts to unify synaesthesia under a single system of substrates, and suggest that competing, complimentary perspectives are more productive than monolithic theoretical models (Smilek & Dixon, 2008). The complimentary perspectives approach may indeed prove the most useful. When taken separately, neurological synaesthesia, artistic metaphor, and latent cross-modal associations are somewhat soporific: linguistic metaphor and Telemanns color fugues reveal little about cognitive substrates, while neurological synaesthesia, with its rarity, explains little about the nature of art, music, language, or creativity. However, in many cases throughout history, the three are all but indistinguishable, since all but the most recent accounts of synaesthesia do not provide solid neurological or behavioral evidence. Works of art that appear to exploit crossmodal tendencies contain little biographical, much less medical, information about the creators of those works. A true relationship between neurological synaesthesia, cross-modal perception, and artistry remains elusive, though increased artistic inclination has been found in individuals with soundcolor synaesthesia based on standardized creativity assessments (J. Ward, 2008). Synaesthesia is apparently more likely in artistic individuals than the general population (Vilayanur S. Ramachandran & E. M. Hubbard, 2001), and synaesthetes are more likely to engage in creative activities than the remainder of the population (J. Ward, Thompson-Lake, Ely, & Kaminski, 2008). For a complete and specific review of contemporary approaches to synaesthesia, including a review of neuroimaging findings, see Hochel and Miln (2008) and Ione and Tyler (2004).
42 Potential Problems with Synaesthetic Behavior Even with synaesthesia and cross-modal perception so closely linked in contemporary scientific work, problems do arise when attempting to directly relate two sensory modes. Specific to the present study, problems with directly relating sounds and colors have been illuminated by scientists and artists alike. Castels work with the clavecin oculaire and his subsequent writings on the subject drew criticism from thinkers of the time (Franssen, 1991), centering around the notion that color agreement is a product of fashion, whereas pitch agreement stems from immutable laws of consonance and dissonance based in nature (i.e., the harmonic series). Additionally, philosophers and theorists of the time also argued that musical perception always centered around a variable tonic note, where color perception is fixed (Jewanski, 1999, 2001). These ideas are not entirely accurate, since musical notions of consonance and dissonance also shift, albeit much more slowly than in visual realms. The presence of tuning systems and non-pitch centered music that differ from the Western tradition also weakens this argument, with different cultures offering much variation on what is perceived as consonant and dissonant, to say nothing of what is musically acceptable in the context of a complete work. As Jrg Jewanski writes, one has to have knowledge of the respective color theories as well as of music theory and physics (especially optics), physiology, astronomy, and even alchemy (Jewanski, 1999) to gain a clear picture of what tone-color relationships have meant at various points in history. Contemporary scientists have also approached much of the early literature with skepticism. For example, John Lockes report of the blind man associating scarlet with the sound of the trumpet: the question remains if this perception was literal or acquired from discussions with others (Cytowic, 1993), or indeed merely a hypothetical parable given as a component of Lockes philosophical endeavors (J. Ward, 2008). Others have noted that while blind individuals
43 may possess increased capacities in non-visual sensory modes, this does not necessarily mean their visual system has been recruited as an inner synaesthetic visual realm (Brugger & Weiss, 2008). Further still, the synaesthesia claimed by artists of the past is impossible to prove, for they may have been actual perceptual experiences, a product of the artistic culture of the time, or simply an en vogue way of expressing oneself (Jewanski, 1999; Stein & Meredith, 1993). Specific Work on Chromaesthesia Specific research into how individuals, both synaesthetes and non-synaesthetes, map one sense onto another have been conducted, though published material on the subject is sparse, particularly in non-synaesthetes. Chromaesthesia, or the specific form of synaesthesia where sounds elicit colored photisms or associations, has been a subject of repeated study among music psychologists. Several researchers study chromaesthesia by way of absolute or perfect pitch, since specific pitches drive the perceived colors (i.e., C natural is blue, F sharp is purple, etc). The phenomenon of chromaesthesia arises in the endeavors of musicians who paired single pitch classes to colors such as Messaein, Schoenberg, Kandinsky, etc. (Bernard, 1986; Brougher et al., 2005; Jewanski, 1999). George Rogers (1987) studied this phenomenon in four musicians with absolute pitch and noted the variations therein. For example, Rogers noted the tendency of his four subjects to label the white keys on the piano as primary colors, with one participant combining two adjacent white notes into a blended color assigned to the black key between them. Rogers also reported that his subjects were unable to divorce their absolute pitch from their chromaesthesia, and noted that all four complained of troubles in transposing music and success in playing by ear and memorization because of their chromaesthesia. Rogers concluded that evidence of early childhood color-pitch pairings suggests that for certain individuals, chromaesthesia may be related to early experience (Rogers, 1987).
44 Lenore Block (1983) investigated the tone-color responses of non-synaesthetic trained musicians while not limiting her subjects to those with absolute pitch. Block determined that those with absolute pitch were more consistent in their tone-color matches than those with high relative pitch abilities, and concluded that chromaesthesia may be more normal in the sense of natural sensory processing, but may dissipate in many people because of lack of reinforcement and socio-cultural expectancies (Block, 1983). While both Rogers and Block, among others conducting similar studies, noted the specific tone-color matches of small numbers of individuals, other researchers have attempted to quantify both pitch and color in an effort to reveal the underlying properties of such matches. Timothy Hubbard (1996) performed such a study, and discovered that both pitch height and melodic intervals contributed to color selections, namely in terms of color brightness. Hubbard describes his findings as supporting a simple mediation based on a common intensity value while acknowledging that the complexity of real-world sensory input is far greater than those used in the experiments (T. L. Hubbard, 1996). Phineas de Thornley Head (2006) studied six chromaesthetic individuals, three with perfect pitch and three without, in an attempt to establish a localized isomorphism between sound and color. In the study, the synaesthetes mapped both in-tune pitches and out-of-tune pitches to specific colors. The responses indicated a covariant pitch-color relationship that occupies contiguous areas of both pitch and color space. This study appears to be the first that allowed participants to match colors and notes on an open-ended scale, and indicates that chromaesthesia is in fact systematic and may not be a fixed one-to-one mapping between specific pitches and colors.
45 Richer experimental techniques examining the relationship between sound and color have also been developed, and have shown results reaching beyond chromaesthesia as an isolated phenomenon. Jamie Ward and colleagues studied synaesthetic responses to musical notes in three individuals and found uniformity among the color responses to sound, written note letter names, and musical notation (J. Ward, Tsakanikos, & Bray, 2006). The study found Stroop-like interference with mismatched stimulus/response pairs, indicating that the color response to a given sound is able to migrate or be shared among modalities. The same group of researchers examined color responses to musical notes and other sounds varied by pitch, timbre, and composition, and found that both synaesthetes and controls use the same heuristics for matching between auditory and visual domains (e.g. pitch to lightness) (J. Ward, Huckstep et al., 2006). These findings mirror results reported by previous researchers (including Galton, Hubbard, etc), but the researchers propose that the mechanism used is shared among all adults, albeit recruited differently by synaesthetes and non-synaesthetes. The studies by Ward and his colleagues suggest that synaesthetic responses are, above all, conceptual in nature, an idea that can potentially unify the entire history of synaesthesia research and artistic endeavors, and that crossmodal mechanisms inherent in the sensory system may play a critical role in not only music perception, but perceptual experiences as a whole. Continuing Engagement with the Cross-Modal: A Hypothesis Above all, synaesthesia and the relationship between the senses have remained a broad topic of concern to scientists, artists, and philosophers. From Aristotles ordered models of the universe to the genesis of contemporary thought regarding synaesthesia in the 19th century, the common thread of breaking down boundaries seems to persist in every mention of cross-modal phenomena. Clearly, the mechanisms of how, when, and why these boundaries break down, to say nothing of our meta-cognitive drive to explore them, are reason enough to produce both
46 objective research and engage in creative activities related to the topic. On a deeper level, Jamie Wards assertion that synaesthesia and normal cross-modal interactions are drawn from the same conceptual pool (J. Ward, Huckstep et al., 2006) leads to the notion that one can engage synaesthetic behavior as both a distinct neurological condition and a shared capability possessed by all. From a research standpoint, the present study adopts the view of Ward, Van Campen, and several other researchers that the relationship between sounds and colors is not arbitrary, and is, to some degree, ubiquitous. That is to say, normal adults show systematic sensitivity to the relationship between sound and color (Reilly et al., 2009; J. Ward, Huckstep et al., 2006). With the persistent interest in cross-modal relationships that history has shown, particularly in the arts, this study proposes that sound-color relationships are, in fact, ubiquitous, and largely based on the apparent height of the sensory information (lightness in the visual realm, pitch height in the auditory realm). Further, despite the notion that artists have increased access to such a relationship, as demonstrated by the repeated study of chromaesthesia in musicians and artists (Block, 1983; Day, 2001; de Thornley Head, 2006; T. L. Hubbard, 1996; Peacock, 1985; Rogers, 1987; Van Campen, 1999; J. Ward, Huckstep et al., 2006; J. Ward et al., 2008) it is proposed that systematic sound-color matching can be found in all adults irrespective of training (in this case, musical training). The relevance of such a finding will be discussed in terms of music in Chapter 6. In proposing that all individuals have access to some form of synaesthetic behavior, the present study challenges a part of the current literature, in particular Ramachandran and Hubbards notion that artistically inclined people are more likely to be true synaesthetes than the remainder of the population (Vilayanur S. Ramachandran & E. M. Hubbard, 2001). The
47 quantitative study outlined in the following chapter will demonstrate these hypotheses through a series of experiments where musicians and non-musicians will be asked to create matching pairs of sounds and colors in the same vein as previous studies by Hubbard (1996), Block (1983), de Thornley Head (2006), Ward (2008), and Rogers (1987), among others. From a creative standpoint, given the revitalized interest in synaesthesia and its relevance to human perception, understanding, and evolution, new creative endeavors influenced by the subject seem particularly appropriate. Chapter 5 of this study will engage the historical, creative, artistic, and scientific elements of synaesthetic behavior by adopting the quantitative data gathered in Chapter 3 as well as the historical interest in the subject as the conceptual basis for a new musical composition entitled Color Shifts.
48 CHAPTER 3 EXPERIMENTAL METHODOLOGY AND PROCEDURES Introduction Undeniably, connections between sensory modes, and to larger extent artistic modes of expression, have engaged cross-modal phenomena repeatedly. Though notions of intersensory relationships and possible connections between the arts and mathematics existed for centuries, the work of scientists and artists from Francis Galton forward offers much evidence that continues to fuel interest in the subject. The contemporary resurgence in synaesthesia research (in works by Marks, Ward, Ramachandran, Cytowic, Hubbard, Dann, van Campen, and many others) and the insight it offers is promising, but systematic study of sound-color associations has thus far been limited (Jewanski, 2001), particularly in musical terms. Inspired by the historical and resurgent interest in cross-modal sensory perception, individuals latent cross-modal associations were explored in a series of four experiments. In these experiments, musicians and non-musicians subjectively matched pairs of sounds and colors. The purpose of this study is to further the understanding and advance the literature on the subject of cross-modal associative relationships and to reverse the notion that synaesthesia and cross-modal perceptual abilities are limited to creative individuals and/or individuals with a rare neurological condition. While perceptual and anatomical changes do occur in those with high levels of music training, this study proposes that the ability to systematically match sounds and colors is not one of those changes. This study aims to collect data that could give insight to the nature of music perception, the origins of music, and the relationship between auditory and visual phenomena in those with music training and those without. In this way, the present study can effectively address both the expressive and receptive sides of music, that is, from the perspective of both the musician (who transmits) and the listener (who receives).
49 Participants Participants were recruited into two groups consisting of musicians and non-musicians. Non-musicians were defined as individuals with no formal music training who did not regularly play an instrument or sing. More specifically, non-musicians must not have participated in college-level music study nor had they been exposed to a significant amount of private training. The musicians group was defined as individuals recruited from the music department at the University of Florida who were undergraduate music majors and had achieved a passing grade in at least two semesters of music theory coursework. This coursework, as is the case in many academic music programs, includes a rigorous aural skills component designed to improve the students utilitarian musical abilities. Graduate students and faculty members were also eligible under the assumption that they shared a common training background in music. The musicians and non-musicians groups were age and gender-distribution matched. Thirty-eight of the forty participants were right handed. All participants were native speakers of English, though several had acquired other languages as a product of their background. All participants had been educated from childhood in the English language, eliminating the possible confounding effect of linguistic differences in color perception (Kay & Regier, 2006). To eliminate the possible confounding effect of individuals with perfect pitch and the possible coincidence with chromaesthesia, individuals with perfect pitch were excluded from the study. Participant Compensation For their participation in this experiment, subjects were compensated with class credit where applicable. Several non-musicians were drawn from the participant pool of the Communication Sciences and Disorders department, and were issued a document verifying their
50 participation in order to complete class requirements. Participants who were ineligible to receive class credit were compensated with a one-time honorarium of $10 US. Procedures Stimulus Preparation In order to reduce confounding variables, all stimuli were carefully prepared to focus on the relationship between pitch and color and eliminate possible interference from other characteristics, such as variations in perceived amplitude (Suzuki et al., 2003). The visual stimuli consisted of 15-degree divisions of the 360-degree color wheel, with two color swatches created at each hue level. One swatch consisted of each hue degree at one hundred percent brightness, and the other at a forty percent brightness level (See Figure 3-1 below). Similar procedures were used to create auditory stimuli. Since participants would be exposed to both pure tones and piano tones during the matching exercises, tones were created from the frequency range available on a standard 88-key piano. Stimuli ranged from 27.5000 Hz to 3951.0664 Hz, corresponding to the fundamentals of the lowest note and the highest available note on an equal-tempered modern piano using whole-step divisions (see Table 3-1 below). A byproduct of selecting tones in this way is that participants were not exposed to a tonal pitch center since the resulting notes formed a whole-tone scale (which lacks, by its nature, a central pitch). Additionally, pure tone stimuli were filtered through an inverse A-weighted equalizer to account for psychoacoustic amplitude effects (Radocy & Boyle, 2003; Suzuki et al., 2003), and processed to fade in and out over 30ms at the beginning and end of each file. These fades were applied to avoid any additional frequency transients from interfering and adding spectra to the pure tone, but are short enough to be imperceptible. Since each audio file contained only a single spectral component, phasing effects of these filters were negligible.
51 Figure 3-1. Color swatches used in the experimental procedures, with their respective RGB values. Swatches are sorted by hue ranging from 0 to 345 Table 3-1. Pitches used in the experimental procedures, in ascending order using Acoustical Society of America pitch convention (Young, 1939) A0 B0 C#1 D#1 F1 G1 A1 B1 C#2 D#2 F2 G2 A2 B2 C#3 D#3 F3 G3 A3 B3 C#4 D#4 F4 G4 A4 B4 C#5 D#5 F5 G5 A5 B5 C#6 D#6 F6 G6 A6 B6 C#7 D#7 F7 G7 A7 B7
52 Pre-Experimental Procedures Participants first received a description of the experimental battery in plain language, and read and signed an informed consent document. Following consent, all participants completed a color blindness screening using a short naming task consisting of 10 color swatches (e.g., red, yellow, green, blue, etc.). Participant hearing acuity was also examined with a computerized tone detection paradigm, wherein tones were presented at pseudorandom intervals in either the left or right ear at frequencies 250Hz, 500Hz, 1kHz, 2kHz, 4kHz, and 8kHz (Reilly, Troiani, Grossman, & Wingfield, 2007). Participants indicated when they detected a sound by pressing the space bar on the computer. Participants who could detect all of the tones passed the hearing screening since the ability to hear these tones indicated intact hearing across a significant frequency range. Specific deficits such as reduced acuity in a particular frequency range was not assessed, though deeper examination of hearing loss and cro ss-modal associations offers yet another avenue for deeper exploration on this subject. As a verification measure, all participants completed a computerized version of the Advanced Measures of Music Audiation (AMMA), a short adult-oriented music aptitude measure that yields tonal, rhythmic, and composite scores (Gordon, 1989). This measure is a commonly administered examination used in Universities and other settings to assess an individuals audiation abilities, that is, the persons ability to hear and manipulate music in their mind. The AMMA highly correlates to the larger Gordon Musical Aptitude Profile as well as other standardized music aptitude assessments (Gordon, 1989) and offers a relatively compact way of verifying musical abilities. The AMMA also excludes musical sensitivity measures, eliminating the potential confound associated with emotional affect. Finally, all participants completed a brief questionnaire intended to gather information on their musical activities, age, gender, languages, and handedness.
53 Experimental Procedures Participants subjectively matched tones and colors in four separate experiments. In all experiments, visual and auditory elements were presented using SuperLab 4.0 stimulus delivery software (Cedrus, 2009). Visual stimuli appeared as 250x250 pixel bitmap images centered on a 20 Apple Cinema Display with a black (no-color) background. The standard Apple sRGB color profile provided uniform color presentation and computer display calibration. Participants wore AKG K-240 professional headphones for auditory stimuli, and were allowed to adjust the volume to a comfortable listening level following the hearing screening. In all experiments, participants used a USB dial device (a Griffin PowerMate) or the computer mouse to search for their auditory or visual response until finding an appropriate match. Participant selections were stored in Max/MSP and matched offline to the corresponding stimulus presentation generated by SuperLab. In all experiments, participants were allowed a practice period to familiarize themselves with the stimulus and explore how the USB dial device and mouse affected their responses. Experiment I: Color Swatch to Pure Tone Cross-Modal Matching Participants viewed 48 color swatches in random sequence and turned the USB dial to the frequency that most closely matched each presented color. The tone generator was implemented using Max/MSPs cycle~ object, and limited to the range of a traditional 88-key piano, 27.5000Hz (A0) to 3951.0664Hz (B7). The tone generator moved across the frequency spectrum in units of one semitone (the distance between adjacent keys on a piano) based on the dial movement. Before each color swatch, the tone generator returned to the lowest pitch (27.5000Hz or A0). An inverse A-weighted equalization filter applied to the tone generator controlled for psychoacoustic amplitude effects. When the participant found a suitable matching frequency, they depressed the dial, storing the selected frequency and continuing to the next
54 color swatch. After each response, the tone generator was reset to the 27.5000 Hz starting point, which was below the threshold of perceptible sound in the bulk of the participants. Experiment II: Pure Tone to Color Swatch Cross-Modal Matching Participants heard 44 pure tone audio files in random sequence and selected a color that most closely matched each tone by moving the mouse cursor and clicking a location on a color palette. Participants could repeat each 1-second tone as many times as they liked by pressing the Return key on the keyboard. Upon finding a particular color that the participant felt best matched the tone, the participant depressed the space bar, which stored the RGB value of the selected color and advanced to the next tone after a 1000ms delay. Experiment III: Color Swatch to Piano Tone Cross-Modal Matching Like Experiment I, participants viewed 48 color swatches in random order and were asked to turn the USB dial device to find a match for each. However, instead of a pure tone, participants heard synthesized piano tones in order to assess the effect of timbre on the selection. The dial device allowed the participant to move across the range of the piano in units of one half step, the distance between adjacent keys, for 88 possible responses whose fundamentals matched those possible in Experiment I. When the participant found a matching pitch, they depressed the space bar, which stored the response and advanced to the next color swatch. Experiment IV: Piano Tone to Color Swatch Cross-Modal Matching Like Experiment II, participants heard a tone and used the mouse to find a matching color. However, in this experiment, the tones consisted of computer-generated, synthesized piano tones to assess the effect of timbre on the selection. There were 44 total pitches presented in random order in this experiment as well, with each tones fundamental frequency matched to those of the pure tones used in Experiment II.
55 CHAPTER 4 EXPERIMENTAL RESULTS Introduction Participant Demographics Table 4-1 below shows the demographic and metric information collected from both the musicians and non-musicians groups. All participants included in the study passed the hearing and color vision screenings given at the start of the experimental protocol. The table shows the intended age and gender matches, as well as the differences in level of music education, evidenced by the semesters of coursework in music theory and scores earned on the Advanced Measures of Music Audiation. A one-way ANOVA comparing group AMMA scores showed a significant difference between the Musicians and Non-Musicians (F(1,38) = 10.3, p = 0.003), validating the group selection process. Table 4-1. Participant demographics and assessment scores Characteristic Musicians Group Non-Musicians Group Total Participants 20 20 Male / Female 10 / 10 10 / 10 Mean Age 23.4 years 23.25 years Semesters of Music Theory Coursework Mean: 4.4 N/A Handedness 19 Right, 1 Left 19 Right, 1 Left Additional Languages Participant 110: ASL Participant 111: Spanish Participant 207: Tagalog Participant 210: Spanish Participant 212: German Participant 213: Russian Composite Score, AMMA Highest Possible Score: 80 Mean: 58.45 Median: 57 Range: 43 to 71 Mean: 51.65 Median: 50.5 Range: 45 to 61
56 Data Conversions During the experiments, participants were able to search for matching visual and auditory stimuli as presented from the computer terminal. Since both musical notes and colors have a multiplicity of features, mathematical conversions allowed the separating out of salient elements in both the sounds and colors. For auditory responses, participant selections were standardized to the 127-note MIDI pitch numbers to simplify data analysis. The MIDI protocol, since it specifies pitches and not frequencies, strongly resembles the linear system upon which humans perceive pitch, thus eliminating the confounding effect of examining fundamental frequencies, which are perceived as non-linear in the human auditory system (Radocy & Boyle, 2003; Young, 1939). Effectively, the MIDI system of pitch designation creates an interval scale upon which to analyze the collected data without assigning arbitrary numbers: MIDI is an industry-standard system that is commonly used in computer music hardware and software (MMA/AMEI, 1982-2001). Color selections were stored using the standard RGB color triplet system commonly used for computer displays. However, this color system is not perceptually uniform and contains litt le useful data unless converted to a more accessible format. Given that the pitch system (using MIDI note numbers) is effectively perceptually linear, conversions were necessary on both the colors presented to and selected by the participants to maintain the integrity of the comparison. Equations developed by the International Commission on Illumination (CIE) and readily available online (Lindbloom, 2007) were used to write short computer programs that would perform the conversions. Participant selections were converted using these programs to the CIE LCH (Lightness, Chromaticity, Hue) system, which is a best-fit uniform color system that mirrors the perceptual response of the human eye. Using these conversions, individual elements of color including hue, lightness, and !E, or distance from a reference point (in this case
57 black/no-color) could be analyzed separately. LCH color space is best described as a sphere, with lightness on the vertical axis, chroma on the horizontal axis, and hue degrees around the circumference of the space (Lindbloom, 2007). Inside this spherical space, !E is a manifestation of Euclidean distance between two points (e.g., the distance from black to bright red is equal to the distance from black to bright blue). Finally, a simple transformation matrix was applied to the stored RGB colors to achieve conversion to the LMS color system, also called cone space, which was developed to mirror the response of the cones in the human eye to the long, medium, and short wavelengths of light (Reinhard, Ashikhmin, Gooch, & Shirley, 2001). This allowed the stored RGB color to be examined in terms of total stimulus to the eye (by summing the calculated value of each cone) as opposed to one single set of cells. Both the CIE LCH and LMS color systems are interval scales, wherein the space between adjacent integer values is effectively equal. Both systems aim for perceptual uniformity, that is, to mimic the way humans perceive color in a linear fashion. Data Analysis Following conversion of the participant responses, means were calculated for each group and matched to corresponding values extracted from the stimulus. The three analyses performed were pitch vs. lightness3, pitch verses distance from black (!E), and pitch to cone-space (" LMS) Statistical comparisons between groups and within groups are described below under each analysis heading. 3 N.B.: In this case, the terminology used is based on those used by the CIE for lightness. Brightness is a specific term used in some computer color systems, whereas lightness refers to the value assigned to each color in the CIE LCH color system, wh ich aims to be perceptually uniform. The two terms are effectively interchangeable, though the L and B values do not precisely correspond between the HSB and LCH color spaces.
58 Analysis of Pitch vs. Lightness Participants strongly associated lighter color swatches with higher pitches, and darker color swatches with lower pitches. In Experiments 1 and 3, where the color swatches were the presented stimulus, the strength of the association is best shown graphically: Lighter colors, on the whole, (labeled b. in the graphs below) elicited higher-pitched responses than darker colors (labeled a.). In mapping the colors to pitches, both groups showed similar correlations between the lightness of the stimulus and the pitch height of the chosen tone. Further, the standard deviations demonstrate that the selected pitches tended to be within a one-octave range (12 half-steps). The musicians group showed slightly more adherence to the linear trend than the non-musicians in Experiment 1, and the reverse in Experiment 3 (see Table 4-2). A one-way ANOVA performed on the results indicated no significant difference between the groups in either experiment (F(1,94) = 0.24, p = 0.63 and F(1,94) = 0.92, p = 0.34 respectively). Table 4-2. Statistical correlations of lightness vs. pitch, Experiments 1 and 3 Group and Experiment Lightness to Pitch Correlation Musicians, Experiment 1 Pearson r = 0.86, standard deviation: 9.5 Non-Musicians, Experiment 1 Pearson r = 0.83, standard deviation: 10.05 Musicians, Experiment 3 Pearson r = 0.84, standard deviation: 11.17 Non-Musicians, Experiment 3 Pearson r = 0.86, standard deviation: 10.92
59 Figure 4-1. Light and dark bias in pitch selection, Experiment 1 Figure 4-2. Light and dark bias in pitch selection, Experiment 3
60 In Experiments 2 and 4, where participants were presented with a tone and asked to choose a matching color, an even stronger systematic relationship was found in both groups. Table 4-3. Statistical correlations for pitch vs. lightness, Experiments II and IV Group and Experiment Pitch to Lightness Correlation Musicians, Experiment 2 Pearson r = 0.97, Standard Deviation: 23.08 Non-Musicians, Experiment 2 Pearson r = 0.97, Standard Deviation: 26.01 Musicians, Experiment 4 Pearson r = 0.97, Standard Deviation: 22.46 Non-Musicians, Experiment 4 Pearson r = 0.97, Standard Deviation: 22.65 As the figures below demonstrate, the mapping of pitch to brightness approaches a 1:1 correspondence, appearing nearly linear and with high degrees of similarity between the musicians and non-musicians groups. With the stimuli changed to a pitch rather than a color, the bright and dark bias observed in Experiments 1 and 3 is absent:
61 Figure 4-3. Linear mapping of pitch to lightness, Experiment 2 Figure 4-4. Linear mapping of pitch to lightness, Experiment 4
62 Statistical comparison of participants responses in these experiments demonstrates that again, little discernable difference can be found between the two groups. A one-way ANOVA comparing lightness selections Experiments 2 and 4 respectively yielded no significant difference between the musicians and non-musicians (Experiment 2: F(1,86) = 0.17, p = 0.68; Experiment 4: F(1,86) = 0.02, p = 0.87). Further analysis revealed significant differences between experiments, however. Experiments 1 and 2 used pure tone auditory responses, where Experiments 3 and 4 used synthesized piano tones. The presented color swatches were identical in Experiments 1 and 3. A comparison between these two sets of responses shows that they highly correlate (i.e., the trend of bright-high and dark-low remained, r = 0.95 for both groups) with the responses significantly shifting in reaction to the timbre change: Table 4-4. Statistical comparisons between experiments for pitch and lightness Comparison Results Experiment 1 vs. Experiment 3: Pitch selections in Musicians (selected pure tone vs. selected piano tone) Paired samples t-test, within group t = 10.97, p < 0.0001 (significantly different) Experiment 1 vs. Experiment 3: Pitch selections in Non-Musicians (selected pure tone vs. selected piano tone) Paired samples t-test, within group t = 18.35, p < 0.0001 (significantly different) Experiment 2 vs. Experiment 4: Lightness selections in Musicians (given pure tone vs. given piano tone) Paired samples t-test, within group t = -4.88, p < 0.0001 (significantly different) Experiment 2 vs. Experiment 4: Lightness selections in Non-Musicians (given pure tone vs. given piano tone) Paired samples t-test, within group t = -5.34, p < 0.0001 (significantly different)
63 Analysis of Pitch vs. Color Difference (!E) Since color is multifaceted, examining how participants matched color to sound in terms of lightness alone would yield results in only one specific realm of color perception. To gain a deeper view into how participants selected matching colors and tones, another approach is warranted that incorporates a multifaceted interpretation of color. In color-critical industries such as textile manufacturing and the printing of physical media, measuring color shift is critical to product development. The International Commission on Illumination, or CIE, provides a method of measuring how far a color drifts between input and output (i.e., a computerized blanket design vs. actual color rendered with dyes). This system is known as !E, with conversion equations readily available online (Lindbloom, 2007). For the purposes of the present study, !E is adopted as a way of measuring colorfulness: while the !E measure is typically used to discern the shift between similar colors, in this case the values were calculated with respect to an all black reference point (RGB 0,0,0 or no color on a computer display). In this case !E can be used to show how colorful a participants selection was based not only on lightness, but on hue and saturation as well, along a single colorfulness scale. In this analysis, Experiments 1 and 3 were omitted since the stimuli (color swatches) were not prepared based on this system. Still, the results show that participants mapped pitch height to colorfulness in a highly systematic way, as in the previous analysis of lightness. In general, both musicians and non-musicians selected more colorful responses as a function of pitch height, as pictured below. A one-way ANOVA performed between groups for Experiments 2 and 4 revealed no significant differences between groups for either experiment (Experiment 2: F(1,86) = 1.45, p = 0.23; Experiment 4: F(1,86) = 0.39, p = 0.53). In contrast with the analysis of lightness, trends in the analysis of the colorfulness of participant selections fell along a third-order polynomial regression, with high degrees of
64 correlation in both groups in both experiments. This would appear to indicate a more complex interaction in the mapping of sound to color as a whole, with brightness as the primary factor in determining how participants selected their responses. Figure 4-5. Pitch to colorfulness trend, Experiment 2 Figure 4-6. Pitch to colorfulness trend, Experiment 4
65 Like the lightness analysis, significant differences were found within groups and between experiments. The comparison tested participant color selections under pure tone auditory stimuli vs. selections under synthesized piano tone stimuli, and indicated significant differences between the experiments. Table 4-5. Statistical comparisons between experiments for pitch and colorfulness Comparison Results Experiment 2 vs. Experiment 4: Color selections in Musicians (selected pure tone vs. selected piano tone) Paired samples t-test, within group t = -3.18, p = 0.0027 (significantly different) Experiment 2 vs. Experiment 4: Color selections in Non-Musicians (selected pure tone vs. selected piano tone) Paired samples t-test, within group t = -4.87, p < 0.0001 (significantly different) Analysis of Pitch to Cone Space (" LMS) The final analysis of the collected data examines how participants matched pitches to colors using the LMS, or long-medium-short, color space. This color space, like any other including the RGB system used for computer displays or the LCH system used in color-critical industries, has particular uses in digital photography and commercial applications (Borer & Ssstrunk, 2002; Ford & Roberts, 1998; Lindbloom, 2007; Reinhard et al., 2001). It also serves to generalize sensory response in that it represents the cone responses in the retina (Borer & Ssstrunk, 2002). In particular, the LMS color space is used to estimate the appearance of colored images under variable lighting conditions by examining the total stimulus to the three types of cone cells in the human eye, sometimes referred to as the red (long wavelength), green (medium wavelength), and blue (short wavelength) (Reinhard et al., 2001). Since each cone in
66 the eye responds differently based on the wavelength of incoming stimulus, summing the three values in the LMS systems yields a potential total stimulus value (" LMS) that can be likened to pitch height. In this sense, the LMS system offers perhaps the best tool for mapping pitch to color, since it gives a specific value in the range of 0 (no stimulation of the eye) to 300 (100% activation of each cone), corresponding to net impact on the visual system. Like the !E analysis, the LMS analysis was only performed with data from Experiments 2 and 4, since the color swatches used in the other two experiments were not prepared based on this system. As with the brightness and !E analyses, participants mapping of sound to color was highly systematic within this color space. As pitch height increased, so too did the net cone response as calculated by LMS. A one-way ANOVA performed between the groups (musicians and non-musicians) within Experiments 2 and 4 again showed no significant differences between them (Experiment 2: F(1,86) = 0.08, p = 0.77; Experiment 4: F(1,86) = 0.01, p = 0.89).
67 Figure 4-7. Pitch to LMS trend, Experiment 2 Figure 4-8. Pitch to LMS trend, Experiment 4
68 By contrast, a comparison between experiments showed no significant differences between experiments, which may demonstrate that mapping a sound to a color does indeed fall along a linear scale, with variations in timbre affecting the choice of hue or chroma. Table 4-6. Statistical comparisons between experiments for pitch vs. LMS Comparison Results Experiment 2 vs. Experiment 4: Color selections in Musicians (given pure tone vs. given piano tone) Paired samples t-test, within group t = -1.01, p < 0.3143 (no significant difference) Experiment 2 vs. Experiment 4: Color selections in Non-Musicians (given pure tone vs. given piano tone) Paired samples t-test, within group t = -1.33, p < 0.1874 (no significant difference) Summary of Results After confirming the validity of the groups in the experiment through the use of the Gordon Advanced Measures of Music Audiation (Gordon, 1989), three separate analyses were performed on the collected data. Though much of the current and historical perspective on synaesthetic behavior indicates that musicians have increased access to such associations, no such difference was found in these series of experiments. However, the data indicates that, as proposed in Chapter 2, both musicians and non-musicians appear to possess the capability to systematically map the seemingly unrelated senses of color perception and musical pitch, at least under the present conditions. Specific variations on this ability were discovered, with participants selections significantly changing based on timbral variation in the auditory stimuli. Specifically, the colors selected to match pure tone stimuli were significantly less bright and less colorful, though the total stimulus to the cones of the eye appear to be equivalent. Further, the systematic mapping was more pronounced when participants were given a pitch and asked to choose a corresponding
69 color, as opposed to the reverse (given a color and asked to choose a pitch). This consistency remained even under timbral variation. These results confirm the notion that when making a cross-modal mapping between sound and color, the most salient elements in making the match appear to be brightness/lightness and pitch height for color and sound respectively, mirroring the results found by other researchers (Block, 1983; de Thornley Head, 2006; T. L. Hubbard, 1996; Jewanski, 1999). In addition, a secondary interaction was found stemming from timbral elements in the auditory stimuli, likely arising from the presence of overtones in the synthesized piano stimulus. Finally, the experiments address the problem elucidated by Jewanski that systematic study has not yet been conducted on sound-color phenomena (Jewanski, 2001), and provide a possible answer to the question of why artistic fascination with synaesthetic behavior exists, as put forth by several researchers (including Jewanski, Ramachandran, Ward, etc.). In Jewanskis words: The fundamental difficulty in assessing the artistic significance of synaesthesia is that in the case of many musicians and artists it is impossible to be sure whether they are experiencing synaesthesia, have a heightened sensitivity to interdisciplinary associations and/or are seeking new ways of expressing themselves by deliberately blurring the frontiers between the arts (for instance, in sound-sculptures). (Jewanski, 2001) Based on these findings, it would appear that musicians do not have heightened sensitivity to cross-modal associations under these conditions. While some musicians may lament this finding, it may prove to be a positive contribution to music perception, as will be discussed in Chapter 6.
70 CHAPTER 5 COMPOSITIONAL METHODOLOGY Introduction Many artists across the space of history have recruited cross-modal perception/association and synaesthesia as a point of inspiration. Often, these artists use sound-color relationships specifically, whether it is a composer (like Scriabin) with apparent neurological synaesthesia (Bernard, 1986; Day, 2001; Ione & Tyler, 2004; Jewanski, 2001), a visual artist creating musical analogues (Brougher et al., 2005; Kemp & Blakemore, 2006; Van Campen, 1999; Zilczer, 1987), or composer-inventors such as Castel and Kastner attempting to create visual musical instruments (Dann, 1998; Franssen, 1991; Van Campen, 1999), cross-modal pursuits and interest in synaesthetic behavior has remained of interest for centuries of artistic endeavor. Translating between or pairing visual phenomena with sound, either directly or indirectly, has exemplified the artistic gravitation towards synaesthesia in recent years. The work of Clarence Barlow, who has undertaken the direct sonification of images using computerized algorithms (Jette, 2009), and Wayne Slawson, who created a systematized set of sound materials grouped into color classes (Slawson, 2005), offer complimentary perspectives on cross-modal musical elements. In addition, many electroacoustic composers and video artists have begun creating singlular works incorporating abstract sound and video, opening new possibilities for cross-modal experiences in the arts. In contrast, the approach to music making in the present study involves the use of colors as pivot points analogous to the tonal modulations, changes in orchestration, or spectral shifts found in various musical styles. This approach allows composers (both the author of this study and others) to derive inspiration from not only broad notions of synaesthetic behavior, but also from actual scientific data. In terms of electroacoustic music, sound materials, filter settings, and
71 points of resonance can be derived from the collected data and used to drive digital signal processing algorithms. Above all, this approach provides a window through which sound can be manipulated and changed: the primary activity in the composition of electroacoustic music. A New Composition: Color Shifts A new composition, Color Shifts, written specifically to accompany the present study, was composed using a set of three filters whose characteristics were, in part, derived from the collected data described in Chapter 4. The piece consists of three movements, presented attaca, and is composed for 5-channel digital medium. The works inspiration stems from synaesthesia itself: put simply, in the mind of a synaesthete, a transformation takes place wherein one sensory mode becomes another. Sound materials with strong visual components were chosen to reinforce the opening sections of each movement, which then transform into new sound materials, using the constructed filters as a synaesthetic tint. Movement I: Red Pipes The first movement of Color Shifts begins with the sound of an organ, specifically, the Jordan Memorial Organ housed at Columbus State University in Georgia, manufactured by Orgues Ltourneau Lte of Quebec. Samples were collected during the 2nd Jordan International Organ Competition, held in 2009. Over the course of the movement, the organ segments into smaller and smaller bursts of activity, and eventually the sound materials converge into those of a thunderstorm. In one sense, the tactile and somatosensory experience of hearing an organ performance is similar to that of experiencing a thunderstorm, as the organ is capable of producing a huge range of frequencies, from the lowest bass to the highest treble. However, the movement was constructed primarily using granular synthesis techniques. The granulation settings were derived from the participant data, specifically, those fundamental frequencies that the participants identified as red.
72 Table 5-1. Filter example: granular settings for Red Pipes Frequency selected by participants as red Granular synthesis settings example 99.2 Hz Initial density: 99.2 grains/sec Grain size: 10.08ms (1000ms / 99.2) The above table shows one example of how frequencies were used to drive the granular synthesis engine. Settings such as these were generated and used to process the organ samples in the Amber granular synthesis environment (McCabe & Merkowitz, 2006). Hundreds of such processes were executed, and the author selected the sound files used in the final version. Movement II: Green Voices After initial planning on the second movement, the voices of several university students were recorded with the intention of using these voices as the sound materials in a second set of filters. However, after some consideration of the recordings, a new motivation for naming the movement Green Voices was discovered. As previously mentioned, the musical style known as the blues describes the particular moods associated with the music and its performers (Oliver, 2001). Instead of constructing digital signal processing algorithms as initially planned, the author instead opted to present the materials in their raw form, while still adhering to notions of transformation. In this case, green takes on a more metaphorical meaning (as does the blues) derived from the multiplicity of definitions in the dictionary. The term green literally refers to a color, but can also be used as an adjective to refer to something that is pleasantly alluring. In contrast, green can also refer to something that is unripened or not fully matured, or a person that is not yet fully trained in his or her discipline. Finally, in an environmental sense, green is currently a popular term referring to the
73 conservation movement or to items that are recyclable, biodegradable, or nonpolluting (Merriam Webster, "Green," 2010). All of these terms (with the exception of the environmental sense) metaphorically match with the nature of the voices of the amateur singers recorded for the movement. Logically, to connect the movement to the transformative goals of the piece, the green voices at the opening give way to the sounds of the ocean in the work, thus bringing together all possible meanings of green. While not directly synaesthetic in its execution, the very nature of the word is reflected in the piece by bridging the gaps between meanings, referring to a word with much cross-modal power, and uncovering yet another way that synaesthetic thinking can influence the production of musical work. Movement III: Blue Orchestra The composition of the third movement, Blue Orchestra, utilizes similar procedures to those used in Red Pipes. A resonance filter was constructed using stacked pure tones and Apples Space Designer software. Participants selections of frequencies matching the color blue effectively became the driving force behind the resonance in the movement, which operated on recorded samples of the Columbus State University Philharmonic. Table 5-2. Filter Example: The resonance filter in Blue Orchestra Frequency selected by participants as blue 118.07 Hz 129.81 Hz 146.80 Hz 164.06 Hz 187.82 Hz 278.10 Hz 299.17 Hz 228.47 Hz 428.98 Hz 514.59 Hz 525.42 Hz 574.21 Hz
74 Figure 5-1. Screen shot of Apples Space Designer with blue filter loaded Several samples were processed through this filter, and used to create the transformation in the movement, wherein the orchestral sounds transform into those of a freight train. The author captured the train recordings outside of Columbus State Universitys Schwob School of Music, which lies directly on a well-traveled rail line. Like Red Pipes, the transformational element of this movement is synaesthetic in the sense that the tactile and somatosensory experience of listening to an orchestra is not unlike the experience of a train moving past: both are capable of producing high volumes, explosive tones, and ground-shaking bass frequencies. Summary Color Shifts draws its inspiration from the transformational element of synaesthesia. Put simply, ideas about one thing changing into another inspired the use of both the data-driven filters and the metaphorical and aesthetic choices present in the piece. Much in the same way that chromaesthesia allows tones to transform into colors, electroacoustic music is uniquely
75 capable of facilitating transformations between disparate materials. Color Shifts expands on these notions of transformations by linking sonic materials together through filters that reflect color itself, as well as tone color. It is hoped that this work will provide not a literal synaesthetic experience (though a simple poll of an audience with no knowledge of the titles of the movements might be conducted), but rather a metaphorical one that touches on the very nature of our sensory, creative, and aesthetic capacities as human beings. Object 5-1. Composition: Color Shifts (AIFF, 127.5MB)
76 CHAPTER 6 DISCUSSION AND CONCLUSIONS Introduction With Greek and Roman philosophys consistent influence on European culture, particularly during the Renaissance and Baroque periods, scientists, philosophers, artists and musicians sought to discover harmony in nature and in their creative output. The notion of harmony, indicating some degree of aesthetic or systematic agreement in a general sense, can be viewed as a synaesthetic concept. The constant attention paid to notions of harmony likely drove the rise of synaesthesia as both a scientific and artistic endeavor: agreement and correspondence between seemingly disparate ideas, things, and sensory experiences lies at the core of much creative work as well as scientific research. Further, in music, harmony has been the driving force behind stylistic development in music in several eras, ranging from Monteverdis new ideas regarding dissonance and text settings in the 16th century to Schoenbergs complete reorganization of harmonic systems in the early 20th. Contemporary scientific views on synaesthetic behavior indicate that deeper study on the topic may result in further understanding of not only musical changes over time, but also cortical connectivity, the origins of language, possible substrates of learning mechanisms, and insight into the human developmental process. Further still, framing the study of creativity and human artistic expression in cross-modal terms may help elucidate these mysterious human capabilities that many scientists identify as uniquely human. The present study has approached synaesthetic behavior from both scientific and artistic angles, and attempted to justify each in terms of the other: the composition of a new work, inspired by historical context and actual data, and a study inspired by the work of both scientists
77 and musicians. If continued exploration of these topics is to result in new understanding of synaesthetic mental processes, this dual approach may serve as a model for future research. Discussion of Findings Analysis of the data collected in the experimental study outlined in Chapter 3 would appear to indicate that musicians and non-musicians use the same process when creating matching pairs of sounds and colors. Given the high level of correlation and lack of differences between these two groups, generalizing the findings to the larger population is at least somewhat valid: all individuals appear to map sound and color the same way (that is, primarily based on brightness, with interactions of hue, chroma, and timbre coming into play as well). While this discovery matches the findings of others (Block, 1983; de Thornley Head, 2006; T. L. Hubbard, 1996; Langfeld, 1914; Marks, 1978; J. Ward, Huckstep et al., 2006), it is unique in the realm of comparisons: no previous study examined comparative tone-color responses in musicians versus those of non-musicians. In addition, the present study offers support for theories presented by other researchers: the notion that sound-color synaesthesia is an exaggerated form of a cross-modal mechanism possessed by all human beings (J. Ward, Huckstep et al., 2006) would appear correct given the data presented here. It also reinforces findings by researchers not working specifically with arts-related cross-modal matching, such as Reilly et. al.s finding of consistent size-tone and size-color matches in novel pseudo-animals (Reilly et al., 2009). It also lends validity to Cretien Van Campens assertion that it is difficult to distinguish synaesthetes from non-synaesthetes (Van Campen, 2008). From the standpoint of musical composition, particularly in electroacoustic music, these findings establish a solid foundation upon which to base new creative activities. The work composed as a result of the present study, Color Shifts, adopts synaesthetic inspiration as the driving force behind a purely sonic composition and one that removes the element of live
78 performance in that it is composed for digital medium. Electroacoustic music is uniquely capable of addressing synaesthetic elements, particularly in fixed-media forms, where no visual interference is present during performance. In the case of video works, synaesthetic matching between sound and image may contribute to the expressive power of such works. The present study also would appear to validate the decision of many electroacoustic composers to perform their works in total or near-darkness to allow the internal reactions of the audience to operate uninhibited. In this sense, synaesthetic behavior would seem to validate contemporary notions of sound symbolism in music: that musical sound can be representational in its materials, and it is the nature of those representations that raise music to the level of fine art. Sound Symbolism in Music A corollary to musical sound symbolism is the sound symbolism present in poetry. Synaesthesia researcher Lawrence Marks offers a comprehensive view of poetic cross-modal perception in The Unity of the Senses: Interrelations Among the Modalities (Marks, 1978). Marks (as many others) asserts that the poetic art is enhanced and enriched through its use of sound symbolism. In a psychological sense, language offers a broad window into cognitive activity, which is why it is often used as the vehicle through which cognitive psychology research is conducted. Much the same can be said of music: it offers a vehicle for superlinguistic expression, the sonification of emotional states, and the opportunity for soundsymbolic creative output, all of which offer a window into mental states and processes that both overlap and exclude language. For Marks, language (and in particular, poetry) contains intersensory elements in three areas: mimetic, analogical, and symbolic. Though music lacks the semantic specificity of language (Patel, 2008), these three areas are also visible in the musical literature.
79 Mimesis in music is perhaps the most noticeable of these, with a host of examples including Mozart and Messiaens birdcall music, as well as other musical representations of animal sounds (such as the clomp of horse hooves portrayed by woodblocks). Natural sounds such as thunder can be produced by large drums, and water sounds by instruments such as the rain stick. The human bodys sounds can also be imitated, with a heartbeat easily formed by deep, thudding drums. The human voice is also capable of imitating instrumental sounds, exemplified by the scat singing style in jazz, wherein a vocalist uses the syllables of language to mimic a saxophone or trumpet. Imitation of speech using musical materials also appears in much 20th-century music, such as Steve Reichs Different Trains, which uses melodies derived directly from vocal recordings. In other musical styles, such as electroacoustic music, literal representations (i.e., sound recordings) replace mimesis. Recording technology mediates animal vocalizations, natural sounds, etc. into musical sounds. Though sampled sounds are not necessarily mimetic, their use contributes to sound symbolism through direct reference rather than imitation. Frequently, the use of materials such as these can give strong visual, emotional, kinesthetic and tactile information to a listener though only auditory information is present during performance. In this way, music can operate as mimetic and symbolic at the same time, such as Richard Strauss Don Quixote, a tone poem that communicates the plot of Cervantes story using no language whatsoever, or Seung-Ah Ohs DaDeRimGil, which explores the ironing ritual performed in traditional Korean villages. Music also operates in the analogical realm, with the most striking example of the blues, itself a synaesthetic metaphor. While blues is a formal musical structure that dictates a particular number of bars and a general harmonic outline, it also operates in the realm of analogy. The
80 blues scale, with its blue notes, is frequently used to communicate a particular mental state. The notion of certain notes of the scale being somehow off, coupled with the rough timbres typically used by a blues musician, contributes to its expressive power. The blues frequently communicates the condition of melancholy or depression (Oliver, 2001), which the performer can rid himself of through the act of singing (ibid). This analogy, perhaps the most synaesthetic of all musical styles, demonstrates a specific cross-modal symbolism in music: the blues is blue, and not, for example, orange. Synaesthetic behavior, whether it exists in the form of neurological cross-wiring, artistic metaphor, or mental association, can be interpreted as a possible functional element in the resolution of sound symbolism in the mind of a listener. Since music is frequently abstract, establishing a process through which musical works have meaning can be difficult. Synaesthetic perception is one potential mechanism through which meaning is discovered, whether color (in a literal sense) is part of the equation or not. For example, it is via synaesthetic metaphor that the blues actually becomes blue, while works by composers such as Michael Torke rely on their own views of colors. Torkes synaesthesia directly influences the creation of works such as Ecstatic Orange and Bright Blue Music. Associations about particular colors attach themselves to non-visual elements, and are used to impart meaning or give inspiration. With regard to the present study, this mechanism may be particularly important in the realm of spectral content and musical timbre. The experimental data showed significant changes in color selections under timbral manipulation. It may hold true that variations in timbre are what gives music its wide expressive capability much in the same way that visual art makes use of light, shadow, and color. The art of orchestration is, at its core, the selection of particular sonic colors using combinations of instruments. Orchestration is often judged for quality
81 based on how those colors transform, and shift, and interact with one another. This phenomenon of transformation is strikingly similar to the sensory transformation that occurs directly in neurological synaesthetes, and, based on the data presented here and by others (Reilly et al., 2009; J. Ward, Huckstep et al., 2006), in all individuals. While in most scientific studies, one searches for differences, in this case access to sound-color relationships appears ubiquitous, meaning that all individuals can recruit and understand tones in terms of color and lightness. Jamie Reilly has proposed that the apparent uniformity with which individuals map sound to color represents a mental map that may provide specific adaptive advantage: large, predatory animals make deeper, richer tones, and prey animals higher, less rich ones (Reilly et al., 2009). If sound-color uniformity is important in the perception of music, it is therefore an exaptive trait: an ability that evolved for purposes other than aesthetic enjoyment, but that contributed to humans creative capacities later in the evolutionary progression. This theory mirrors other nonadaptive theories of musical evolution, which are well-articulated in the music psychology literature and textbooks (McDermott & Hauser, 2005; Thompson, 2009). Regardless of the adaptive advantages to sound-color mapping, models of musical perception must encompass all individuals. After all, enjoyment of the arts is not limited to those with training, and deficits in music perception are the exception and not the rule. Music psychologists and philosophers have articulated a wide variety of models, most notably Susanne Langer, who defines artistic perception in terms of internal illusions. Susanne Langers Concept of Secondary Illusion in the Arts Across the space of her work, Susanne Langer developed an illusory model of arts perception. In her writings, Langer discusses arts perception in a two-tiered system consisting of primary and secondary illusions. The term illusion is not used in a hallucinatory or delusional sense, but rather in terms that can be traced back to the work of Immanuel Kant, who conceived
82 of illusions as acknowledged through a play of the mind with perceptual appearance (Reichling, 1995). In other words, illusions are a cognitive capacity which may enhance the expressiveness of a work of art as well as provide a kind of inward vision (ibid). Music, Langer argues, gives the listener an auditory apparition of time (ibid) where movement through the musical materials does not occur according to a clock, but rather in terms of the music itself, which exists in time. This temporal illusion serves as the primary illusion in Langers framework. Musical movement through time, Langer argues, is illusory in the sense that no physical objects are in motion, but the sound itself moves on a virtual time-plane. As a complimentary example, Langer argues that paintings primary illusion is one of space, since a flat canvas is rendered into an image that is not real, but is vividly perceived nonetheless. For Langer, space and time are the primary realms in which visual artwork and music exist, respectively. Secondary illusions in music, according to Langer, arise within the primary illusion and consist of the aesthetic responses encapsulated therein. As Mary Reichling writes, Rather than remaining constant like the primary illusion, secondary illusions arise and dissolve, sometimes suddenly and surprisingly, other times more gradually. They appear to be in a state of constantly unfolding, often so subtle that they seem subjective and personal (Reichling, 1995). Of the many possible manifestations of secondary illusion, Langer indeed identifies colors among many others, including texture, weight, power, etc. (Langer, 1974; Reichling, 1995). Within this framework, synaesthetic perceptions and associations take on an important role: they become one of many mechanisms used by the human mind in the understanding of formal structure. In compositional terms, Langers illusion model offers insight into the formation of masterworks: if the depth, quality, and construction of these secondary illusions take on certain
83 qualities, the resulting musical experience is thus deepened as well. The convergence of both natural talents and extensive training in music, when combined, may be the underlying feature that gives rise to artistic genius. Synaesthetic behavior in composers and artists is but one facet of secondary illusion that can provide enriched experiences for listeners. Conclusions Across the space of history, humans have utilized a particular ability to relate seemingly disparate topics to one another. Beginning with the ancient Greek and Roman notions of harmony in the world, through Western civilizations artistic endeavors and the rise of the scientific method, people have found much depth and beauty in the interrelation of the senses. The present study has addressed this fascination from a variety of angles, and has attempted to link a multiplicity of approaches in one common research project. The continued study of synaesthesia and cross-modal associations by neuroscientists and psychologists, as well as the artistic endeavors of contemporary composers, can only continue to shed light on this unique facet of human experience. Only by approaching this subject from both angles can scientists and artists begin to understand what role synaesthetic behavior might play in human perception and creative activity. While the present study has elucidated but one element of this complex topic, it is hoped that through continued engagement from both scientific and creative angles that humans might begin to further understand the processes that give rise to deep, meaningful aesthetic experiences. The visions of synaesthetes are often puzzling and seem unbelievable, but these unique individuals can undoubtedly reveal critical insight into how all human beings might process the world around them. If the research presented here is any indication, there is much yet to be discovered about the linking of the senses, how artists and musicians express themselves, and what new modes of expression might be uncovered based on cross-modal experiences.
84 WORKS CITED Bargary, G., & Mitchell, K. (2008). Synaesthesia and Cortical Connectivity. Trends in Neurosciences, 31(7), 335-342. Baron-Cohen, S., & Harrison, J. E. (1997). Synaesthesia: An Introduction. In S. Baron-Cohen & J. E. Harrison (Eds.), Synaesthesia (pp. 3-16). Cambridge, MA: Blackwell Publishers, Ltd. Bernard, J. W. (1986). Messiaen's Synaesthesia: The Correspondence between Color and Sound Structure in His Music. Music Perception, 4(1), 41-68. Block, L. (1983). Comparitive Tone-Colour Responses of College Music Majors with Absolute Pitch and Good Relative Pitch. Psychology of Music, 11, 59-66. Borer, S., & Ssstrunk, S. (2002). Opponent Color Space Motivated by Retinal Processing. Proceedings of the First European Conference on Color in Graphics, Imaging, and Vision(April 2002). Brougher, K., Strick, J., Wiseman, A., & Zilczer, J. (2005). Visual Music. New York, NY: Thames and Hudson. Brugger, P., & Weiss, P. H. (2008). Dermo-Optical Perception: The Non-Synaesthetic "Palpability of Colors", A Comment on Larner (2006). Journal of the History of the Neurosciences, 17, 253-255. Calkins, M. W. (1893). A Statistical Study of Pseudo-Chromaesthesia and of Mental Forms. American Journal of Psychology, 5(4). Calkins, M. W. (1895). Synaesthesia. The American Journal of Psychology, 7(1), 90-107. Cedrus. (2009). SuperLab (Version 4.0): Cedrus Corporation. Collins, M. (1929). A Case of Synaesthesia. Journal of General Psychology, 2, 12-27. Cutsforth, T. D. (1924). Synaesthesia in the Process of Reasoning. The American Journal of Psychology, 35(1), 88-97. Cutsforth, T. D. (1925). The Role of Emotion in a Synaesthetic Subject. American Journal of Psychology, 36(4), 527-543. Cytowic, R. E. (1997). Synaesthesia: Phenomenology and Neuropsychology. In S. Baron-Cohen & J. E. Harrison (Eds.), Synaesthesia (pp. 17-41). Cambridge, MA: Blackwell Publishers, Ltd. Cytowic, R. E. (2002). Touching Tastes, Seeing Smells -and Shaking Up Brain Science. Cerebrum, 4, 7-26.
85 Dann, K. T. (1998). Bright Colors Falsely Seen: Synaesthesia and the Search for Transcendental Knowledge. New Haven: Yale University Press. Day, S. A. (2001). A Brief History of Synaesthesia and Music [Electronic Version]. ThereminVox. Retrieved February 12, 2009, from http://www.thereminvox.com/article/articleview/33/1/5 Day, S. A. (2005). Some Demographic and Socio-Cultural Aspects of Synaesthesia. In L. R. a. N. Sagiv (Ed.), Synaesthesia: Perspectives from Cognitive Neuroscience (pp. 11-13). New York: Oxford University Press. de Thornley Head, P. (2006). Synaesthesia: Pitch-colour isomorphism in RGB-space? Cortex, 42(2), 164-174. Ford, A., & Roberts, A. (1998). Color Space Conversions. Retrieved 12/1/2009, 2009, from http://www.poynton.com/ColorFAQ.html Franssen, M. (1991). The Ocular Harpsichord of Louis-Bertrand Castel: The Science and Aesthetics of an Eighteenth-Century Cause Clbre. Tractrix, 3, 15-77. Galton, F. (1880). Visualized Numerals. Nature, 21, 494-495. Galton, F. (1883). Inquiries into Human Faculty and Its Development. Retrieved October 2009, from http://www.galton.org/ Ginsberg, L. (1923). A Case of Synaesthesia. American Journal of Psychology, 34, 582-589. Gordon, E. E. (1989). Advanced Measures of Music Audiation: GIA Publications. Green. (2010). Merriam Webster Online, from http://www.merriamwebster.com/dictionary/green Habib, M., & Besson, M. (2009). What Do Music Training and Musical Experience Teach Us About Brain Plasticity? Music Perception, 26(3), 279-285. Hnggi, J., Beeli, G., Oechslin, M., & Jncke, L. (2008). The Multiple Synaesthete E.S. -Neuroanatomical basis of interval-taste and tone-colour synaesthesia. NeuroImage, 43, 192-203. Hochel, M., & Miln, E. G. (2008). Synaesthesia: The Existing State of Affairs. Cognitive Neuropsychology, 25(1), 93-117. Hubbard, E. M., & Ramachandran, V. S. (2005). Neurocognitive Mechanisms of Synaesthesia. Neuron, 48, 509-520. Hubbard, T. L. (1996). Synaesthesia-Like Mappings of Lightness, Pitch, and Melodic Interval. The American Journal of Psychology, 109(2), 219-238.
86 Ione, A., & Tyler, C. (2004). Neuroscience, History, and the Arts: Synaesthesia: Is F-Sharp Colored Violet? Journal of the History of the Neurosciences, 13(1), 58-65. Jette, C. (2009). Decompartmental: Conversing with Clarence Barlow. Computer Music Journal, 33(4), 10-22. Jewanski, J. (1999). What is the Color of the Tone? Leonardo, 32(3), 227-228. Jewanski, J. (2001). Colour and Music [Electronic Version]. Grove Music Online. Retrieved February 17, 2009, from http://www.oxfordmusiconline.com/subscriber/article/grove/music/06156 Kadosh, R. C., & Henik, A. (2007). Can synaesthesia research inform cognitive science? Trends in Cognitive Sciences, 11(4), 177-184. Kadosh, R. C., Henik, A., & Walsh, V. (2009). Synaesthesia: Learned or Lost? Developmental Science, 12(3), 484-491. Kay, P., & Regier, T. (2006). Language, thought and color: Recent developments. Trends in Cognitive Sciences, 10(2), 51-54. Kemp, M., & Blakemore, C. (2006). Science in Culture: Hearing colours, seeing sounds. Nature, 442(7102). King, A. (2004). Development of Multisensory Spatial Integration. In C. Spence & J. Driver (Eds.), Crossmodal Space and Crossmodal Attention. Oxford: Oxford University Press. Langer, S. (1974). Mind: An Essay on Human Feeling. Baltimore, MD: The Johns Hopkins University Press. Langfeld, H. S. (1914). Note of a Case of Chromaesthesia. Psychological Bulletin, 11, 113-114. Langfeld, H. S. (1915). Synaesthesia. Psychological Bulletin, 12, 121-122. Langfeld, H. S. (1926). Synesthesia. Psychological Bulletin, 23, 599-602. Larner, A. J. (2006). A Possible Account of Synaesthesia Dating from the Seventeenth Century. Journal of the History of the Neurosciences, 15(3), 245-249. Lindbloom, B. (2007). Useful Color Equations. 2009, from http://www.brucelindbloom.com/ Locke, J. (1685). An Essay on Human Understanding [Electronic Version]. Retrieved June 18, 2004 from http://humanum.arts.cuhk.edu.hk/Philosophy/Locke/echu/ Marks, L. E. (1978). The Unity of the Senses: Interrelations among the Modalities New York: Academic Press.
87 Maurer, D., & Mondloch, C. J. (2005). Neonatal Synaesthesia: A Reevaluation. In L. Roberton & N. Sagiv (Eds.), Synaesthesia: Perspectives from Cognitive Neuroscience (pp. 193213). New York: Oxford University Press. McCabe, M., & Merkowitz, J. B. (2006). Amber: A Granular Synthesis Tool. Proceedings of the 2006 International Computer Music Conference. McDermott, J., & Hauser, M. D. (2005). Probing the Evolutionary Origins of Music Perception. Annals of the New York Academy of Sciences (1060), 6-16. MMA/AMEI. (1982-2001). MIDI 1.0 Specification [Electronic Version], from http://www.midi.org/techspecs/midispec.php Nettl, B. (2001). Music [Electronic Version]. Grove Music Online. Retrieved February 2, 2009, from http://www.oxfordmusiconline.com.lp.hscl.ufl.edu/subscriber/article/grove/music/40476 Oliver, P. (2001). Blues [Electronic Version]. Grove Music Online. Retrieved June 21, 2009, from http://www.oxfordmusiconline.com/subscriber/article/grove/music/03311 Parault, S. J., & Schwanenflugel, P. J. (2006). Sound-symbolism: A Piece in the Puzzle of Word Learning. Journal of Psycholinguistic Research, 35(4), 329-351. Patel, A. D. (2008). Music, Language, and the Brain. New York: Oxford Unviersity Press. Peacock, K. (1985). Synesthetic perception: Alexander Scriabin's color hearing. Music Perception, 2(4), 483-506. Radocy, R., & Boyle, J. D. (2003). Psychological Foundations of Musical Behavior (4th ed.). Springfield, IL: Charles C. Thomas Publisher, Ltd. Ramachandran, V. S., & Hubbard, E. M. (2001). Psychophysical Investigations into the Neural Basis of Synaesthesia. Proceedings: Biological Sciences, 268 (1470), 979-983. Ramachandran, V. S., & Hubbard, E. M. (2001). Synaesthesia--a window into perception, thought and language. Journal of Consciousness Studies, 8(12), 3-34. Ramachandran, V. S., & Hubbard, E. M. (2006). Synesthesia: What Does It Tell Us about the Emergence of Qualia, Metaphor, Abstract Thought, and Language? van Hemmen, J. Leo, 23, 432-473. Reichling, M. J. (1995). Susanne Langer's Concept of Secondary Illusion in Music and Art. Journal of Aesthetic Eduction, 29(4), 39-51. Reilly, J., Rodriguez, A., McCabe, M., Biun, D., & Altmann, L. (2009). Sound Symbolism as a Window into Cross-Modal Perception and Language. Paper presented at the Annual Meeting of the International Neurological Society.
88 Reilly, J., Troiani, V., Grossman, M., & Wingfield, A. (2007). An introduction to hearing loss and screening procedures for behavioral research. Behavior Research Methods, 39(3), 667-672. Reinhard, E., Ashikhmin, M., Gooch, B., & Shirley, P. (2001). Color Transfer between Images. IEEE Computer Graphics and Animation, 21(5), 34-41. Rogers, G. L. (1987). Four Cases of Pitch-Specific Chromesthesia in Trained Musicians with Absolute Pitch. Psychology of Music, 15, 198-207. Rose, K. B. (1909). Some Statistics in Synaesthesia. The American Journal of Psychology, 20(3), 447. Saenz, M., & Koch, C. (2008). The Sound of Change: Visually-induced Auditory Synaesthesia. Current Biology, 18(15), 650-651. Slawson, W. (2005). Color-Class and Pitch-Class Isomorphisms. Perspectives of New Music, 43(1), 54-119. Smalley, D. (1997). Spectromorphology: Explaining sound-shapes. Organised Sound, 2(2), 107126. Smilek, D., & Dixon, M. (2008). Two Complimentary Perspectives on Synaesthesia. Trends in Cognitive Sciences, 12(10), 364-366. Stei n, B. E., & Meredith, M. A. (1993). The Merging of the Senses. Cambridge, MA: MIT Press. Suzuki, Y., Mellert, V., Richter, U., Mller, H., Nielsen, L., Hellman, R., et al. (2003). Precise and Full-Range Determination of Two-Dimensional Equal Loudness Contours [Electronic Version]. IS-01, from http://www.nedo.go.jp/itd/grant-e/report/00pdf/is01e.pdf Thompson, W. F. (2009). Music, Thought, and Feeling. New York: Oxford University Press. Van Campen, C. (1999). Artistic and Psychological Experiments with Synaesthesia. Leonardo, 32(1), 9-14. Van Campen, C. (2008). The Hidden Sense: Synaesthesia in Art and Science. Cambridge, Massachusetts: MIT Press. Ward, J. (2008). The Frog Who Croaked Blue: Synaesthesia and the Mixing of the Senses New York: Routledge. Ward, J., Huckstep, B., & Tsakanikos, E. (2006). Sound-colour synaesthesia: To what extent does it use cross-modal mechanisms common to us all? Cortex, 42(2), 264-280. Ward, J., Thompson-Lake, D., Ely, R., & Kaminski, F. (2008). Synaesthesia, creativity and art: What is the link? British Journal of Psychology, 99(1), 127-141.
89 Ward, J., Tsakanikos, E., & Bray, A. (2006). Synaesthesia for Reading and Playing Musical Notes. Neurocase, 12(1), 27-34. Ward, O. (2006). The Man Who Heard His Paintbox Hiss [Electronic Version]. The Telegraph Online. Retrieved 1 June 2009, from http://www.telegraph.co.uk/culture/art/3653012/The-man-who-heard-his-paintboxhiss.html Young, R. W. (1939). Terminology for Logarithmic Frequency Units. Journal of the Acoustical Society of America, 11(1). Zilczer, J. (1987). Color Music: Synaesthesia and 19th Century Sources for Abstract Art. Artibus et Historiae, 8(16), 101-126.
90 BIOGRAPHICAL SKETCH Matthew McCabe grew up in New Jersey and Virginia and attended Woodberry Forest School, the University of Richmond, and Bowling Green State University before arriving at the University of Florida. His compositions have been performed across the United States and Europe, and his work can be found on the Centaur and Everglade record labels as well as in Computer Music Journal. Mr. McCabes interests include electroacoustic music, music perception and cognition, audio engineering, graphic design, and craft beer. He has served on the technical staff of the University of Richmond, as technical director of the Florida and Third Practice electroacoustic music festivals, and on the Executive Committee of the Society of Composers, Inc. In 2009, he began service as Visiting Assistant Professor of Audio Technology at Columbus State University in Columbus, Georgia.