Intermedial Cognition

Towards a method for analyzing electroacoustic intermedia through an interpretation of the topoi in Jarosław Kapuściński’s “Oli’s Dream”

by Ryan Olivier

Hearing Electronic Voices

During a post-concert question-and-answer session at the 2012 Workshop on Computer Music and Technology in Taiwan, the partner of a local electroacoustic composer asked the panel how they explained their profession to their spouses and children. She admitted that she had a hard time understanding the music her partner created and that she often found it boring. The audience laughed awkwardly, but the question came from a place of genuine curiosity. Unfortunately for her, no one had a good answer. So then, what does it say about a musical tradition if those who are closest to the creators — their family members — cannot understand what they are trying to achieve through their art? How then are electroacoustic composers going to present their music in a highly ritualized space, such as a concert hall, to an audience of strangers? The composer and the audience must somehow find a common ground from which to start; they must develop a common language for musical discourse.

Enabling Discourse Through Multimedia

The temporal connection between electronic music and a more recognizable form of discourse allows for the transference of meaning from one medium to another. In electroacoustic concert works where the semiotic discourse might be unfamiliar to the listener, the addition of another sensory experience creates fertile ground for the negotiation of meaning. Due to the expectation of visual action in the concert hall and the apparent lack of human activation in electroacoustic production, electroacoustic sound has an artistic complement when paired with visual media.

The widespread availability of video projection in concert halls in recent years has enabled the incorporation of visual media into traditional concert works much more easily than was the case in the past. Many orchestras around the country have faced declining concert attendance, and some have sought to lure the public back through the addition of visual projection. The Philadelphia Orchestra, long associated with visual media through their historic collaboration with Disney for Fantasia, filed for bankruptcy in 2011. In 2012, they added to their 2012–13 concert season a live screening of the entirety of West Side Story film in Verizon Hall. In July later that year, thanks to a variety of factors, the orchestra emerged from bankruptcy, but continued the tradition of visual accompaniment by adding four concerts with projection (Pixar in Concert, West Side Story, Gladiator and Star Trek Into Darkness) to their 2014 summer concert series. Additionally, the orchestra has premiered two works by Tan Dun that used live projections (The Map and the NuShu Concerto), and their most recent foray into mixed-media presentation in the concert hall was the addition of an application for smartphones and tablets, which was synced to a live orchestral performance and provided programmatic information. Whether or not these types of ventures will result in a sustained increase in interest in the symphonic repertoire is yet unknown, but it is possible that the addition of visual projections is more in line with the audience’s expectation of a fully engrossing artistic experience created through the assimilation of multimedia within 21st-century Western culture.

With the use of visual representation comes the potential to synchronize musical themes or motives with visual signs, creating another level of musical cognition through sight. The subsequent unfolding of the visual signifiers offers a direct visual complement to the transformational processes of aural themes in “absolute” electroacoustic compositions. I will explore the heightened experience of metaphorical exchange in multimedia concert works, giving particular attention to the increased perception of musical discourse in electroacoustic composition through the use of visual representation and the emergence of a contrapuntal multimedia experience created by the interaction between potentially contrasting visual and auditory narratives.

Topic Theory

Leonard Ratner affirmed in Classic Music: Expression, Form and Style (1980) that through its contact with poetry, drama, entertainment, dance, ceremony, ritual and popular culture, music develops a thesaurus of figures based on rhythm, contour, texture and harmony to which we ascribe various emotional, episodic and semantic associations. Ratner examined the highly public and codified music of the early 18th century, but theorists such as Kofi Agawu have shown how these musical signifiers — musical topoi or topics — operate within less codified and more private cultures, such as those of the Romantic period and the early 20th century (Agawu 2009). Before analyzing the discourse that arises from the addition of non-auditory media to electroacoustic concert works, it is important to understand how music gains a language of its own through topical discourse. By applying Ratner and Agawu’s models to electroacoustic music, I will explore how listeners come to know and recognize topics in the 21st century, how new topics are formed, why topics are difficult to identify in electroacoustic music, what musical topics can be heard in electroacoustic works and how composers can create topics in their works through the utilization of multimedia.

How Topics Are Created

Topics are a construct of culture (Agawu 2009). There is no inherent meaning in the fluctuation of particles resulting in our perception of sound (Scruton 1983). Rather, physical stimuli are interpreted by our brains and assigned meaning based on image schema stored in our semantic memory (Zbikowski 1998). As such, for topical discourse to exist, a listener must be able to identify the cultural connections between the features of a work and the associations they suggest. This requires some prior experience with the style of a given work in order to recognize suggestive features within the music. These features are known as “signifiers”. The recognition of topics also requires some level of prior acquaintance with these signifiers through cultural association to enable them to point to what they signify (Agawu 2009). The clarinet passage (the signifier) in Messiaen’s Quartet for the End of Time, which represents bird song (the signified), is an example of a musical topic. Further meaning can be ascribed to this passage through the intertextual significance of natural sounds in Messiaen’s music and their relation to his spiritual beliefs. It is clear in this example that prior acquaintance with bird song (the signified) is a prerequisite for its sonic identification. Also important for identification would be some prior knowledge or experience with how Western music represents bird song (the signifier) through imitative gestures in acoustic instruments (how the music signifies). A more in-depth knowledge of Messiaen’s music would allow for further attribution of meaning through an intertextual understanding of what bird song represents in his other works. Topics create a language for musical discourse that the listener interprets through her own personal web of associations based on her knowledge of cultural signification.

We come to know topics through previous exposure to them, but as stated earlier, topics are not objects out in the world waiting to be identified; they are constructs of culture. It is through our participation in culture that we come to learn, manipulate and create topics. However, culture is highly unstable. Certain cultural associations, and our perception of them, can change rapidly. For example, during my first encounter with Richard Strauss’ Also sprach Zarathustra, I associated the music with space. I had never seen Stanley Kubrick’s film, 2001: A Space Odyssey, nor heard the entirety of Strauss’ work, yet the association was so prevalent within my culture that I seemed to “just know” the reference. Later, I encountered the work as a music student. I learned how it utilized the harmonic series in the opening measures and referenced Nietzsche’s text. Then I saw the film and realized the connection between the sunrise scene from Nietzsche’s work as depicted through Strauss’ music and Kubrick’s images depicting the dawn of man. When I finally heard the Philadelphia Orchestra perform the work live a few years ago, all of these references created an immediately identifiable musical topic. The harmonic series in the brass accompanied by the sound of the timpani signified the sunrise, nature, creation and mankind.

Many topics are ready-made through our experience with culture. My associations with the opening bars of Also sprach Zarathustra, to my knowledge, appeared out of nowhere. Mike Rugnetta and Kornhaber Brown refer to this ready-made cultural production in their video on Internet culture as a cultural singularity (2012). Cultural singularity is the idea that culture is continually creating and manipulating itself at a rate that is beyond our ability to remain knowledgeable of its source. This is most easily seen through the phenomena of Internet memes, which so rapidly manipulate cultural references it is hard to stay aware of the independent cultural references that constitute their composite meanings. For most cultural participants, Internet memes are produced and evolve independently of any means of production or authorship (Rugnetta and Brown 2012). The Internet meme thus serves as a visible representation of the continual manipulation of our cultural landscape.

Signification can also be created through our subconscious interaction with culture. The utilization of Classical music in television commercials is a perfect example, which Nicholas Cook has thoroughly analyzed in Analyzing Musical Multimedia (1998). While we might not be consciously paying attention to the music while viewing a commercial, it is, nonetheless, performing signification. For example, in a series of commercials Gatorade released in 2008, the on-screen physical intensity and excitement of sports players was paired with Carl Orff’s Carmina Burana. Signification occurs in our subconscious through encoding the association between the music and the images. Another process of signification is conscious signification. My study of Strauss’ work as a music student and my understanding of the harmonic series and its relation to Nietzsche’s work was a conscious connection I made that I later consciously drew on during the Philadelphia Orchestra’s performance mentioned above.

Topical Cultures

Agawu states that, theoretically, topics can be found in all works (Agawu 1991). However, in his discussion of the difference between the Classical and Romantic uses of topics, he notes that the culture surrounding a work dictates the ontological purpose of the music, and how, if at all, the music participates in discourse (Ibid.). In the Classical period, music was used as a very public form of discourse, communicating what were believed to be universal truths furthering the goals of a collective cultural project known as the Enlightenment. In contrast, during the Romantic period, music was a much more private form of discourse, conveying personal beliefs and ideas furthering the project of Modernity (Ibid.). In order to understand the relevance of topic theory in the contemporary genre of electroacoustic music, one must first be aware of the cultural projects from which the music came, and how the music works within that discourse.

Since the 1980s, the West has been preoccupied with the project of Postmodernity. Philosophers and critical theorists have characterized the goals of Postmodernity in many ways. One applicable manifestation of these goals in the arts has been the levelling of authority (Cobley 2001). It allows for the equality of high culture and low culture, resulting in an extremely polystylistic artistic landscape. Postmodernity also promotes the idea that contemporary art should somehow be concerned with the identification of the artistic participant, be that a composer, performer or listener, within their culture. This polystylistic acceptance has led to a form of discourse in contemporary music that is both private and public, reflecting the Postmodernity project of equality.

Within this framework, topics can be both immediately recognizable and difficult to discern, depending on the particular work and the listener’s knowledge of the style of the work. Contemporary composers operate in both overtly public and private forms of discourse. They utilize easily recognizable cultural topics as well as purposefully obscure musical figures, depending on how they want their music to operate within their particular culture to make an association with the topic. This polystylistic language can be difficult for listeners to understand, because they would have to know both the private language of the composer and the public language of the culture. This would require a high level of exposure to a vast amount of music. Paradoxically, our postmodern interest and participation in multiple cultures makes it unlikely that an average listener would ever achieve the high level of specialized exposure necessary to understand the musical topics being utilized in order to participate in polystylistic musical discourse.

Multimedia As An Enabler

Polystylism poses difficulties for an outsider to participate in the discourse of electroacoustic music on a surface level, however, I believe multimedia can increase the perception of musical topics and allow the general listener the ability to hear and understand the musical discourse of electroacoustic music. Cognitively speaking, musical topics are a manifestation of cross-domain mapping. In Conceptualizing Music (2006), Lawrence Zbikowski has shown that through our cultural interactions with music we ascribe meaning to various musical figures through the process of cross-domain mapping across image schemas. George Lakoff and Mark Johnson first proposed this model in their cognitive restructuring of the linguistic concept of metaphor in their book Metaphors We Live By (1980). They define image schemas as abstract constructs of recurring bodily experiences. Through image schemas, various actions and experiences are understood through their relationship to other similar experiences that point to the same abstract concept.

Building on Lakoff and Lawrence’s model, Zbikowski demonstrates that music also creates associations metaphorically. Through metaphor, these image schemas can be mapped onto one another through an enabling invariant link. These metaphorical mappings thus create a gamut of new relations. An example from linguistics would be the maxim “Love is blind.” Here the word “is” is the enabling link between the concept of love and blindness. Through this link, one’s associations with blindness are mapped onto the concept of love. These mappings are strongest where there is a high level of invariance between our understanding of the source domain, in this example “blindness”, and the target domain, “love”.

Nicolas Cook, in his theory of analysis proposed in Analyzing Musical Multimedia (1998), notes that in multimedia works this enabling link is most often a temporal relationship. When auditory stimuli are temporally aligned with visual stimuli, our brain enables a link between the two domains. Sandra K. Marshall and Annabel J. Cohen (1988) noted that after this initial link, the continuation of a particular auditory stimulus or visual stimulus, even when experienced in isolation, can evoke the meaning previously locked in the other domain. In essence, through the intermingling of media via the temporal link video affords, musical discourse can be translated through the cross-domain mapping of visual and auditory stimuli.

Topical Analysis of “Oli’s Dream”

In order to show how multimedia can aid in discerning musical discourse within an electroacoustic work, I will analyze Jarosław Kapuściński’s work Oli’s Dream (2010). In order to organize my analysis, it will be helpful first to catalogue the musical topics an experienced listener may encounter in electroacoustic music. Agawu combines several catalogues of topics from sources such as Ratner, Allanbrook, Hatten, Monelle, Dickensheets, Brabócz and Mirka to cover the music of the Classical and Romantic periods as well as the 20th century (1991 and 2009), however there is no listing of contemporary electroacoustic musical topics. I have therefore identified, based on my own experiences with the genre, a short list of topics found in electroacoustic works (Table 1). The titles, through verbal signification, point to their cultural associations and the musical descriptions show how they are most often represented. Through first analyzing the musical topics an experienced listener would identify in Oli’s Dream, I will then explore how visuals aid in translating that discourse to a listener unfamiliar with the genre.

Kapuściński’s work is a single-movement multimedia piece for piano, electroacoustic sound and projection, in which imagery is cued by the piano in real time. Because of the integration of the performer’s actions with the electroacoustic sound and imagery, Kapuściński refers to his work not as a multimedia concert piece, but as a work of interactive media, or “intermedia”. The work was originally written for a Disklavier, an acoustic piano with electronic sensors to sense key depression. Later, Kapuściński utilized a Moog Pianobar attachment to outfit concert pianos with sensors to detect key depression and report MIDI data to a control computer. The data was used to control the visuals projected on the screen above the piano. Kapuściński is currently in residence at IRCAM (Institut de Recherche et Coordination Acoustique/Musique) in Paris, where he is developing a score-following program that analyzes the sound of the piano using a microphone and cues the video based on auditory data.

Oli’s Dream is a great example of a work steeped in the project of Postmodernity. It was the result of a collaboration between Kapuściński and poet Camille Norton. As written in the preface to the score:

When composer Jarosław Kapuściński and poet Camille Norton began talking about the rapport between music and poetry, they discovered that as children they had shared the same dream, the dream in which a piano becomes a typewriter and in which a typewriter becomes a piano. (Kapuściński 2010)

While the image is central to understanding this intermedia piece, I will start my analysis by interpreting what one hears at the beginning of the work (Audio 1).

Auditory Analysis

The first phrase in the piano part consists of three short gestures, which settles any apprehension as to what kind of electroacoustic piano music is to come. This is not a mechanical or thorny work for piano but something simpler and more familiar. Each gesture opens with a compound major third. The thirds in the opening phrase (Fig. 1) immediately inform the listener that while the work will feature electronic sounds, the piano part is not divorced from the expectations set forth by the piano repertoire of the Common Practice era, dominated by tertian harmonies. The second half of the second gesture continues the downward two-note motif, but instead of a compound major third, the dyad completes the C-major triad by including the G and doubling the C an octave lower, confirming the tonal language insinuated by the opening. The piano is not, however, the only pitch-based material in the opening phrase.

Even before the pianist plays, the consonant sound of a compound major third emanates from the speakers. The timbre is similar to that of a sustained piano note, but swells from nothingness without an attack, as if the ghost of a piano were singing from the darkness. The swell ends with the strike of the first compound third from the performer. After each subsequent gesture, the swell returns to lead to the next attack. When the pitch material of the singing piano is combined with the pitches printed in the score, a tonal language of carefully voiced harmonies arises that was previously only insinuated by the major thirds and C major triad.

When the electroacoustic and piano parts are transcribed from the printed score into a composite, the harmonic reduction reveals a closed harmonic progression in B-flat major (Fig. 2). After the initial opening in B-flat major, the music passes through a B/D-sharp dyad leading to the secondary dominant, setting up the cadence.

Historical Topics

Agawu notes that topics in the 20th [and 21st] century are, “in part, a repository of 18th- and 19th-century usages” (Agawu 2009, 48). Patrick McCreless set a precedent of identifying historical uses of Romantic topics in contemporary electronic works through his gesture-based analysis of the music of Mario Davidovsky (McCreless 2006). Following McCreless’ lead, we can look to the musical material of the opening phrase of Oli’s Dream for signs of historic musical topics. The opening thirds in the piano and the simple harmonic progression in B-flat major, analyzed above, evoke a pastoral topic. Robert Hatten, in his comprehensive analysis of topics in the music of Beethoven, Musical Meaning in Beethoven: Markedness, correlation and interpretation (1994), notes that the pastoral topic is supported by the characteristic use of parallel thirds and the major key. He says that “any movement in a major key with a simple character would qualify as pastoral,” and furthermore that “melodic parallel thirds are a ‘sweetener’… often encountered in pastoral movements from the Baroque, not only for their sweetness, but because of the simplicity they suggest.” Moreover, the use of parallel thirds supports the implication of a vocal duet where “the two singers are in complete accord” (Hatten 1994, 82 and 98). In addition to the pastoral topic, the harmonic progression implied by the alternating rising thirds evokes a striving topic similar to those identified by Hatten in his analysis of Beethoven (Ibid., 18). After the second major third gesture, the piano fills out the C major harmony and then yields to a circle of fifths progression from C major to F major and finally to B-flat major to close.

Having identified the musical material as signifying a pastoral topic as well as both the striving and yielding topics, we can look to what the pastoral signifies in other musical works and related art forms to interpret its usage in Oli’s Dream. Hatten states that the pastoral alludes to the idyllic, the peaceful, the happy, the picturesque and the unsullied, thus we can interpret the opening as confidently happy music for piano and that through the historical use of the pastoral in the 21st century the music points temporally to music from a simpler, unsullied time. The allusion to a vocal duet set up by the parallel thirds at the beginning of each gesture in the piano coincides nicely with the timbral duet between the piano and the “singing piano” in the synthesized electroacoustic voice that fills out the progression. There is a more distant connection between the pastoral as a representation of the picturesque and Oli’s Dream as a work of visual music where the image is equally important as the music itself.

Electroacoustic Topics

Having identified the topics found in the piano music, let us now turn our attention to the electroacoustic sound. I alluded to the voice of the “singing piano” above, but it is not the only electroacoustic presence in the soundscape. At the opening, not only do we hear the ghostly, synthesized voice, but we also hear the sound of a typewriter. An annotated transcription shows all of the sounds heard in the opening 20 seconds and how they are linked temporally (Fig. 3). As the opening phrase progresses, we hear various typewriter sounds, strikes, spaces and a carriage return that coincide with notes from the piano. Additionally, there are vocal utterances from a baby. Unlike the typewriter, the baby sounds do not accompany every note from the piano. While they are only aligned at the opening and the closing of the phrase and thus are not directly linked by cause and effect, the infantile attempts to communicate align with similar attempts made by the typing piano.

Unlike the singing piano, these sounds act more like found objects that immediately point to physical references and images. While the pitch material of the singing piano adds to the pastoral character of the piano part, the timbre calls to mind the technological mutation topic. The timbre does not sound too far removed from that of a sustained piano, but the onset of each note and its subsequent crescendo can not be created by a traditional piano. As the two timbres trade off gestures of the phrase, their tones blend together, to evoke the electroacoustic blending topic. While the typewriter, the baby and the piano never timbrally transform from one into the other or blend together to form a new, unified timbre, their temporal connection does align them, thus musically supporting the blending topic.

Combing Auditory Topics

By combining the pastoral topic of the piano part and the harmonies of the singing piano with the icons of the typewriter and the baby’s voice, as well as with the electroacoustic topics of technological mutation and blending, we arrive at nuanced interpretation of the opening narrative. The opening sets the music in the simpler and unsullied past. This interpretation is reinforced by the innocence of the baby’s voice, somehow facilitated through the use of enhanced technology. Additionally, the parallel thirds at the beginning of each gesture in the piano allude to a vocal duet, which coincides nicely with the timbral duet between the piano and the singing piano in the synthesized electroacoustic voice that fills out the progression. A more distant connection is made between the pastoral as a representation of the picturesque and Oli’s Dream as a work of visual music wherein the image is equally important as the music itself.

Visual Signification

The visuals, projected above the piano, further facilitate signification to classically trained listeners even if they have limited experience with the electroacoustic genre, which in turn allows for a more nuanced level of discourse. If we take a look at the opening 20 seconds (Video 1), the implied blended signification is immediately confirmed by the linkage of text with each keystroke of the piano and the sound of the typewriter. The image moves on the screen as if it were being typed by the piano, confirming the acoustic implication that these two timbres are somehow one through the use of technology. An audience member with no understanding of these common tropes in electroacoustic concert works can easily understand this connection through the linked images of the pianist’s movements and the projected image.

Leading up to the opening dyad from the piano, an indiscernible collage of text scrolls across the screen accompanied by the opening singing piano sound. The lack of acoustic source nicely coincides with the ambiguous appearance of the words, which are not typed onto the screen but scroll past at different speeds as if waiting to be grabbed out of the air and enacted. The piano acting as typewriter decisively puts an end to the uncertain image and wills the words onto the screen. This wilful enactment of communication is juxtaposed by the baby’s voice attempting to communicate at the start and end of the phrase.

Wheel of Association

Taken together, the visuals and the sound create a web of interpretation with varying levels of clarity as to what is being signified and how it achieves signification. Intermedia concert works with live performers, interactive electronic sound and projected imagery operate in four planes of signification: the sound of the instrument or the voice of the performer, the electronic sound, the projected image and the movements of the performer. These four planes are interconnected and interact across each domain. Through these connections they inform and transform one another. It is best not to think of this interaction in a linear sense but as a continuum (Fig. 4).

In addition to the four levels of signification found in most intermedia concert works, Oli’s Dream adds two additional layers of signification through the use of found sound objects in the electroacoustics — the typewriter and the baby — and the words that appear on the screen that immediately imply signification through the direct use of language. Thus, both media contain a form of direct signification in addition to the more nuanced narrativity of the piano music and the movement of the images projected on the screen. While these narrative objects directly point to their signification, their signifiers are less connected to the embodiment on stage because the objects are not visually present, but imaginatively depicted in our minds. Figure 5 combines the four planes shown in Figure 4 with the two additional planes found in Oli’s Dream, that of the “Found Sound” and the “Text.”

Oli’s Wheel

Returning now to the intermedial interpretation, I will combine the printed words with their implied signification. I have already shown how the found electroacoustic objects of the typewriter and baby’s voice are linked with the typing piano, but the addition of the imagined imagery that arises from the text adds another layer of meaning that is essential to how we interpret the work. In Figure 6, underneath the music from the printed score and my own transcription of the singing piano part, I have mapped the found sounds, the projected text and the visible movements projected on the screen onto three one-line staves. Since the notation implies that the pianist will have to move to play the notes, I did not create a separate staff for the pianist’s movements; however, the pianist’s actions on stage are in no way less important to the drama of the work than any other type of media. Along the left side of the staff, I have listed the associative categories found in the intermedia wheels in Figures 5 and 6. Viewed temporally, the continuum between media is hidden, but the associative links between the media come to the fore.

The opening swell in the singing piano, not enacted by any motion of the performer, coincides with an indiscernible collage of text that scrolls across the screen by an invisible force. The singing piano timbre, invisible typewriter and scrolling text are not only linked temporarily but also topically as the disembodied ghostly timbre of the singing piano voice evokes a sense of mystery or ambiguity — as does the collage of words that scroll by at different speeds and whose movements are enacted by an unseen force. In addition to the sound of the synthesized piano voice there is the sound of an unseen typewriter, but what is typing is not yet clear. Both the scrolling text and typewriter seem to move at speeds that are not consistent with our expectations of a real typist. These sounds and images are temporally non-diegetic and suggest a time not consistent with the present. The mix of scrolling objects moving at different speeds across the screen and the fast sound of the typewriter suggest a shift in time at the opening to a time removed from the present.

In contrast, at Rehearsal 2, the pianist visibly plays the first major third dyad and immediately the text is thrust upon a white screen with black ink accompanied by the striking sound of a typewriter and the sound of a baby trying to communicate. The first letter and word plucked from the ethos of scrolling text is “I”. These sounds and images are happening in the present, but due to the temporal ambiguity set forth by the opening sound and images it is difficult to ascertain whether this is happening in the present or if we are somehow witnessing something occurring in the present that started from a different point in time.

Even in the first pianistic gesture there is a collision of topics and meaning. The disembodied and temporally ambiguous opening suggests a time and place removed from our own. With the first gesture, the pianist begins in the present and brings the text and music to life through their actions. The pianist strives to communicate through the harmonic progression, which is reinforced by the attempts of the infant to communicate. This musical topic helps place the present action of the pianist in the past, thus informing the opening sound and scrolling text as travelling back in time. The strike of the typewriter coinciding with the notes of the piano and the clearly displayed text on the screen solidifies the blending topic of the piano and the typewriter. The blending topic is often enacted through a technological mutation. In this instance, the projected imagery of text is very similar to what many of us have experienced when viewing an author type at a laptop and then seeing the image projected onto a screen. However, we do not see an author typing on a laptop, but instead see a pianist at the keyboard. The sound we hear is not that of keyboard keys being depressed but that of a typewriter, a writing tool of the past that is often romanticized by storytellers and authors. The pianist is playing a pastoral typing instrument made possible by technology. Finally, the visible object we associate with a typewriter, a letter, is displayed on the screen after we see the pianist excite the strings of the piano. 1[1. Kapuściński is currently in residence at IRCAM (Institut de Recherche et Coordination Acoustique/Musique) in Paris, where he is developing a score-following programme that analyzes the sound of the piano using a microphone and cues the video based on auditory data.] All of these signs coalesce when we view the first unambiguous sign, “I”.

The use of the first person immediately sets up narrative tension as we try to determine where this newfound agency is coming from. Who is this “I”? The temporal alignment of aural and visual signifiers offers a few clues. Because the text is typed on the screen, we can assume “I” represents the typist; however, we do not see a typist but instead a pianist. With the linkage between the typed words and the actions of the pianist, we can now interpret the pianist as the typist, but we cannot assume he is also the author. 2[2. Although in the recording referred to here the performer is indeed the composer of the work and therefore, presumably, also the author of the text, broadly speaking this assumption is not a given.] The vocal utterance of the baby further complicates our understanding of the pianist as author, since we can assume that no matter who is performing the work, it is presumably not an infant. However, the pianist’s attempts to communicate to the audience through music are similar to a baby trying to communicate for the first time.

The opening reads like a multimedia narrative told through music, words and action — a technological Gesamtkunstwerke. The opening signifies travel to a time when music was simpler. This travel enacts a striving attempt at dialogue in the first person. The “voice” of the narrative agent emerges through the pianist typing on the screen. With each note from the piano, another letter appears. Then with an audible space from the typewriter, the image on the screen is adjusted and the singing piano, which earlier symbolized time travel, now returns to symbolize a pause in dialogue. As the text continues, “I am a,” the author begins to feel more comfortable in his newfound voice. The striving tension eases as the harmony yields to a circle of fifths progression and a cadence in B-flat major closing the progression and signalling a return of the opening. While the harmonic progression is closed, the text is left open.

From this very brief example, we can see that while the music of Kapuściński’s work provides a vast resource of musical topics for the negotiation of meaning, even for the trained Classical listener not steeped in the language of electroacoustic music, engagement with the work and its discourse is greatly enhanced through the incorporation of the visual domain. This cross-domain mapping allows for the identification of musical topics that an experienced listener may hear in the music. When we add the found sounds as well as the blending and technological mutation topics to the narrative language of the work, we open up new possibilities for interpreting the music, adding depth and nuance. Topics imbue music with a voice. One simply has to learn the language to know what it is saying, and multimedia affords an artistic method for quickly learning the vocabulary.

Contrapuntal Intermedia

With the utilization of multimedia in electroacoustic concert works comes the added benefit of a new plane of cognition. In addition to the enhancement of topical discourse in unfamiliar musical languages and the inherent visual discourse in the imagery, a third plane of cognitive discourse emerges in the mixing of the media. For example, in visual music works, not only are we listening to the discourse of the music we hear, and viewing the discourse in the images we see, but we are also closely following the discourse between the two sensory planes, creating a third rhetorical plane.

In Analyzing Musical Multimedia (1998), Nicholas Cook goes so far as to say that a multimedia work is only truly understood as a mix of various types of media when their meanings are not the same. Cook argues that if the images are so closely linked with the sound that they point to the same topical meaning, the medium is more closely akin to a synesthetic experience where two perceptual senses interpret the same stimuli than a multimedia one. Since the term “multimedia” most commonly refers to the presence of more than one medium at a given time regardless of their cognitive interactions, I will use the term “intermedia” instead of redefining multimedia.

In Audio-vision: Sound on screen (1994), Michel Chion describes this linked experience as not truly existing in either one medium or the other, but in a single unified meaning which he calls “audio-vision”. Chion describes the work of the Foley artist on a film set as the archetypical example of audio-vision. Building on Chion’s ideas of audio-vision and rendered sound, I propose that 21st-century listeners, having the potential benefit of a lifetime’s worth of experience with audio-vision, view it as more real than the true recorded images and sounds heard in archival footage. For instance, we believe we are seeing a more accurate capture of a horse riding on soil when we hear its hooves clopping along, even if the source material for the sound is not created by the hooves of a galloping horse striking the ground but in fact the clapping together of coconut shells. While the source material for the creation of the audio-vision object differs, they both point toward the same meaning; however, when the expectations built up by one medium differs from the presentation in the other, the interaction gives rise to a new plane of understanding, the intermedial plane. Monty Python captured this idea brilliantly in their parody of horseback riding in Monty Python and the Holy Grail (1975). In the film, King Arthur travels the countryside accompanied by the sound of a galloping horse. However, he is not on horseback. Instead, as Arthur prances along as if riding an actual horse, his squire accompanies him, clapping coconut shells together to simulate the sound of the horse. The humour of the third intermedial plane is caused by the disconnect between the expectations built up by the sound and the images displayed on screen.

Analyzing Multimedia

In order to analyze these divergences, Cook sets forth a method he calls “instances of multimedia” (IMM) that are found in commercials, film music, popular music videos and traditional multimedia art forms such as opera, but he does not go so far as to analyze electroacoustic multimedia concert works. In his method of analysis, he looks for the meaning embedded in both media and interprets how they are similar or different using terminology from Lakoff and Johnson’s similarity and difference tests for the relationships among metaphors. He describes three possible interactions for each instance of multimedia: conformance, complementation and contest. Conformance is where both media point to the same metaphorical relationship. Complementation occurs when one medium lends its potentially dissimilar meaning to another and informs it. Finally, contesting multimedia occurs when the two media are in conflict, vying to impose their meaning onto one another. For Cook, multimedia only exists in the complementary and contesting forms of interaction, what Kapuściński calls intermedia.

Kapuściński describes the interaction between various media as a counterpoint of linked attributes. If a visual attribute such as the direction of an object moves upwards and it is temporally linked to a melody that also ascends, he states that these two attributes move in similar motion. If both move in different directions, the two attributes move in contrary motion, and if one moves up while the other stays the same, the attributes move in side motion. Kapuściński notes that since contrapuntal intermedia affords the opportunity for change in domains other than direction, such as density, brightness and colour, the term “motion” no longer applies and the relationships should be analyzed in terms of change. He even goes so far as to say that new terminology is needed to describe these relationships (Kapuściński 2001). I suggest keeping as many similarities as possible while taking out the implication of motion. For instance, an increase in density for both the projected image and the musical texture would be a similar change. An increase in density in one medium accompanied by a decrease in the other would be contradictory change, and a change in density in one domain while the other remains static would be considered to be a divergent change.

Analyzing Contrapuntally Interactive Media

By combining the two theories, I will set forth a method for analyzing the intermedial experience. Like Kapuściński, I will use familiar contrapuntal terms to describe the interaction of motion between musical voices to show the movement of meaning from one topic to the next between various media. Intermedial counterpoint analyzes the movement from one instance of intermedia to the next but instead of independent musical lines working together in harmony, a work of contrapuntal media deals with independent lines of meaning working together to produce a cognitive intermedial experience.

In order to analyze intermedial counterpoint between varying media there must be an established cognitive link that insinuates a connection between the meanings embedded in each form of media. As mentioned earlier, Cook notes that this initial link is often temporal. After the initial link, if the meaning of the image remains consistent with that of the sound when progressing from one section to the next, the lack of contradictory change creates intermedial stasis. If, however, both forms of media move to new meanings but maintain the same metaphorical cognitive relationship, the interactive change is parallel. If the topical meaning in both voices moves to differing meanings, the interactive change is contrary. Finally, if the topical meaning in one medium changes but remains constant in the other, then the interactive change is divergent or oblique. In order to distinguish between this type of interaction and a contradictory change in meaning also implied by the term “divergent”, I will use the term “oblique”; although as Kapuściński pointed out, this term refers primarily to motion and not a change in meaning.

With oblique change there also arises the potential for the transference of meaning across cognitive planes as described by Marshall and Cohen (1988). The meaning embedded in one medium can be maintained in the static medium while the other medium changes. For example, if we are viewing a running horse and hear the sound of it galloping, and then as the galloping sound continues we see an image of a knight prancing without a horse, our expectation of a horse is carried over to the image of the knight even if we do not see the horse. Likewise, contrary change can create a cross modal link by transferring characteristics of one medium to the other through metaphorical exchange. If after viewing the prancing knight and hearing the galloping horse, we see the knight slowly start to move faster and faster, accompanied by the sound of a train beginning to leave a station, we transfer the implied increase in the train’s momentum to the knight picking up speed. We can then assume that the knight is also gaining the same kind of powerful momentum as the train, even if it appears as if he is only prancing faster and faster.

Contrapuntal Intermedia Analysis of “Oli’s Dream”

Having established a method for analyzing contrapuntal intermedia, let us return to Kapuściński’s Oli’s Dream. Instead of looking at the notes on the staff, I will identify the music using descriptive text as I did with the projected imagery. When nothing is happening in a given plane, I have provided a rest to indicate the inaction and silence. When the topical idea is continued, I have used repeat signs. Most intermedia concert works operate in the four planes described in the wheel of association above (Fig. 4), but due to the use of found sounds and text-based imagery, I have analyzed six cognitive planes for Oli’s Dream (Fig. 5). First it will be important to see which planes are related through initial temporal links. The following table shows the initial intermedial links in Oli’s Dream at Rehearsal 1 (beginning of the piece) and at the first gesture following Rehearsal 2.

While the shift from Rehearsal 1 to Rehearsal 2 is a marked narrative moment due to the introduction of new linked intermedial modalities, the contrapuntal change is parallel across existing modalities. However, starting at Rehearsal 2 not all of the linked modes continue in parallel. After filtering out the parallel relationships to focus on the oblique and contrary changes, two deviations arise. As seen in the final table below, the linked strike of the piano, the sound of a typewriter keystroke and the first discernible word “I” diverge from the indiscernible utterance of the baby after the first gesture. While the linked media all return in the next gesture, the baby’s attempt to communicate is absent; however, it is still possible to equate the baby’s attempt at speech with the striving motive in the piano and the typed text on the screen. Thus, through the initial link, the baby’s attempt is embedded in the other sounds and images even as the contrapuntal meanings diverge in oblique motion. The indiscernible utterances return at the end of the third gesture to close the phrase.

Additionally the onset of each note of the striving topic in the piano and the synthesized singing piano is linked to the sound of the typewriter and the typing of text or the movement of a space on the screen. While the onsets continue to be linked to the manipulation of text on the screen throughout the phrase, the trajectory of the text and the progression contradict one another. The striving progression of rising thirds eventually yields to a circle of fifths progression and cadences at the end of the phrase back on tonic. The text seems to follow the same trajectory at first, starting with a forced attempt at speech and then continuing to add characters to construct a meaningful phrase, but the text does not reach a “cadence” when the music does. The music returns to tonic with the final notes from the piano and the cadence is punctuated by a repeat of the tonic in the synthesized singing piano at the break. However, the article “a” that ends the text implies continuation: “I am a…”

These two divergences give new narrative meaning to the work. The baby tries to speak through the language of the piano via the agency of the pianist. In the first gesture the attempt appears to be successful, as the pianist types the words on the screen through the keys. After the baby’s utterances begin to convey meaning with the typing of the first word, the utterances cease, but when the meaning of the text is left incomplete — “I am…” — at the musical cadence, the indiscernible utterance of the infant returns to signal a failed attempt at communication. The successful attempts of the pianist and the synthesized singing piano, which together strove to create discourse through their music via the technological mutation of the blended typing piano, fail because the text does not reach its conclusion at the end of the closed musical phrase. Fortunately, the story is not over and the media will again assemble in the second phrase to try to complete their task.

Conclusion

In Oli’s Dream, Kapuściński not only succeeded in creating a work of multimedia in which multiple media are utilized to create the artistic experience, but he also succeeded in creating a work of intermedia in which the continual manipulation and transference of metaphors among the various medial planes creates a web of discourse and narrative unfolding that is equally as important as the unfolding in each individual plane. Thanks to this interactive multimedia, the audience viewing Kapuściński’s work can look forward to a contrapuntal web of discourse and interchanging signification that creates a new sense of tension and stability. This web of discourse enables contemporary electroacoustic composers to speak to their audiences through many different forms of signification, thus developing a common artistic language with which to convey meaning and engagement in their concert works. Additionally, it adds a new cognitive layer to the concert hall experience, one that is familiar to, and in many ways expected by 21st-century listeners. Our current multimedia-based society grows continually more demanding of its members’ ability to interpret and interact with various forms of media. As our cognitive abilities develop and change, one can only hope that artists will mine our expanding capabilities to experience art and reach out to their audience in new and imaginative ways that breathe new life into the concert hall and usher in an age of reformation and renewal.

Bibliography

Agawu, Kofi. Playing With Signs: A semiotic interpretation of Classical music. Princeton NJ: Princeton University Press, 1991.

_____. Music As Discourse: Semiotic adventure in Romantic music. New York: Oxford University Press, 2009.

Chion, Michel. Audio-Vision: Sound on screen. Trans. Claudia Gorbman. New York: Columbia University Press, 1994.

Cobley, Paul. Narrative. New York: Routledge, 2001.

Cook, Nicholas. Analyzing Musical Multimedia. Oxford: Oxford University Press, 1998.

Hatten, Robert. Musical Meaning in Beethoven: Markedness, correlation and interpretation. Bloomington IN: Indiana University Press, 1994.

Johnson, James. Listening in Paris: A cultural history. Berkley: University of California Press, 1995.

Kapuściński, Jarosław. “Basic Theory of intermedia: Composing with sounds and images.” 2001. Available online at http://jaroslawkapuscinski.com/writing.html [Last accessed 5 April 2016]

_____. Oli’s Dream. [Score] 2010.

Lakoff, George and Mark Johnson. Metaphors We Live By. Chicago IL: University of Chicago Press, 1980.

Marshall, Sandra K. and Annabel J. Cohen. “Effects of Musical Soundtracks on Attitudes Toward Animated Geometric Figures.” Music Perception 6/1 (Fall 1998), pp. 95–112.

McCreless, Patrick. “Anatomy of a Gesture: From Davidovsky to Chopin and back.” In Approaches to Musical Meaning. Edited by Byron Almen and Edward Pearsall. Bloomington: Indiana University Press, 2006, pp. 11–40.

New Oxford American Dictionary. Oxford: Oxford University Press, 2013.

Pierce, John R. An Introduction to Information Theory: Symbols, signals and noise. New York: Dover, 1980.

Rugnetta, Mike and Kornhaber Brown. “Are Memes & Internet Culture Creating a Singularity?” YouTube video (6:29) posted by “PBS Idea Channel” on 22 August 2012. https://youtu.be/eNBOkp346G8 [Last accessed 5 April 2016]

Ratner, Leonard. Classic Music: Expression, form and style. New York: Schirmer Books, 1980.

Scruton, Roger. “Understanding Music.” Ratio 25/2 (1983) pp. 97–120.

Zbikowski, Lawrence. “Metaphor and Music Theory: Reflections from cognitive science.” Music Theory Online 4/1 (January 1998), pp. 1–14.

_____. Conceptualizing Music: Cognitive structure, theory and analysis. Oxford: Oxford University Press, 2002.