Investigating Audience Reception of Electroacoustic Audio-visual Compositions

Developing an effective methodology

by Andrew Hill

What is Electroacoustic Audio-visual Music?

Brief Background

Much like the frequent discussions that take place surrounding the definition of sound-based music there are many discussions about what to call and how to define multimedia works exploring the combination of the two time-based media, sound and moving image. 1[1. Some of these discussions are available as archives from the Centre for Visual Music's discussion list visualmusicplus.]

Discussion of sound and image interaction is not new. The Ancient Greeks discussed it, Newton had a theory on the subject and numerous people devoted their lives to the construction of colour organs in the 18th and 19th century in an attempt to realize an art form that brought together sound and light. 2[2. The following texts provide an introduction to the history of sound and image association within western society: Brougher 2005, Collopy 2000, Ox and Keefer 2006. Many other articles and links are available from the Centre for Visual Music’s online library.] The most liberating technical development for the genre was the invention of tools to capture sound and image, and most importantly, to play them back again alongside one another.

The advances in, and recent affordability of digital technology has encouraged this genre to expand rapidly with VJ (visual-jockey) performances in clubs becoming regular and visualization software on music media players being almost ubiquitous. While there has been an explosion in the prevalence of audio-visual media and visuals to accompany sound there is still a fairly limited selection of theoretical texts analysing the subject field and even less work has taken place in the field of audience study for these works.

Visual Music? Lumia? Visualisation? Or Audio-visual music?

There are plenty of terms used to describe works that make use of sound and image and in the context of this paper it is necessary to outline a set of definitions and distinctions for the various forms and types of electroacoustic audio-visual music.

There are a lot of divided opinions over what qualifies a work to be classed as visual music. The subjective nature of classification means that each individual will define works in reference to a different ideal of visual music and thus a certain degree of flexibility is necessary within the categorisation system so as to make it as useful as possible to the widest range of scholars or artists. Any work containing characteristics that might qualify or suggest it to be visual music from a specific perspective should be able to be defined within the classification system. I find it most useful to separate the overall art form of visual music into four main sub categories.

1. The purely visual approach to Visual music, for example Thomas Wilfred’s Lumia, or the works of Kandinsky or Klee. Works that aim to emulate music, or contain structures and forms inspired by those within music but contain no sonic content themselves.
2. Visual composition to pre-existing musics such as in some of the early works of Oskar Fischinger, the artistic interpretations of Walt Disney’s animators in the 1940 film Fantasia or music videos of the type found on MTV.
3. The composition of sound and image informed by traditions of music in which materials are structured within time. This form is here defined as audio-visual music because works contain both sonic and image elements. The sound and image are regarded as equal components joined in the context of a work and are both structured musically. A work itself would be an audio-visual composition. 3[3. From a Cageian perspective “audio” and “music” in the same term could seem tautological but this definition operates with a distinction between sound and music.]
4. The synthesis of visual materials from sound and the representation of sound visually. This includes visualization software such as those within media players, oscilloscopes and computer algorithms that render visually spectral and waveform images of sonic material.

My own research investigates what I define as electroacoustic audio-visual music, classified within category C. In each of the above categories the artistic practice involved in the creation of and the characteristics of the final work will differ. Thus the way that audiences act to interpret works in each category will also differ, “We never see the same thing when we also hear; we don’t hear the same thing when we see as well” (Chion 1994, xxvi).

So where does the electroacoustic part come in?

In the current project, the works selected contain sonic elements exploring and utilising the timbral nature of their audio materials and that are influenced heavily by the genres of musique concrète, elektronische Musik and electroacoustic music; music genres for which sound and not the musical note are the basic unit (but perhaps not exclusively). 4[4. See Understanding the Art of Sound Organization (Landy 2007) in which the term “sound-based music” is introduced and discussed.] The visual elements of these works explore the texture, colour, form and motion of their materials; but again may not do so exclusively. Further to this, the works that have been chosen for investigation exist in a fixed form, i.e. they are not greatly altered by repeat performance other than through the equipment and setup used for projection. 5[5. Projection here refers to both sound and image.]

The type of works that have been selected can be described using a definition adapted from Simon Emmerson and Denis Smalley’s definition of electroacoustic music cited on the ElectroAcoustic Resource Site (EARS): 6[6. Simon Emmerson and Denis Smalley, “Electroacoustic Music” in The New Grove Dictionary of Music and Musicians, edited by Stanley Sadie (Second edition, 2001). [Last accessed 2 August 2010]]

An electroacoustic audio-visual music work could be defined as a cohesive entity in which audio and visual materials are accessed, generated, explored and configured, primarily currently with the use of computer-based electronic technology, in the creation of a musically informed audio-visual expression. Electroacoustic audio-visual music works explore the possibilities that the combination of their two time-based media (sound and moving image) allow. (Hill 2010)

Such works could be more succinctly described as being works of “organized sound and image”. 7[7. See note 5. Also observe the use of and to designate the equal balance between sound and image within an electroacoustic audio-visual music work as opposed to the use of the word with which might designate or suggest the primacy of one element over the other.]

Aims of the Current Research Project

The current research project seeks to investigate the following questions:

1. Can audiences previously unexposed to electroacoustic audio-visual music compositions understand and appreciate them?
2. How do variations in the style of composition (i.e. selection and arrangement of materials) affect the ability of audiences, previously unexposed, to understand and appreciate electroacoustic audio-visual compositions?
3. How does the volume and content of contextual information from the composer affect the ability of audiences, previously unexposed, to understand and appreciate electroacoustic audio-visual compositions?

In order to develop an effective methodology that is able to address the above questions it is necessary to investigate and evaluate the methodology of previous research projects investigating similar questions or in similar fields.

Please note this paper deals only with the issue of audience reception of works and so issues relating to the contextual information and the impact that this has on audience reception will not be covered. Further information about this and other aspects of the project can be found in the author’s article “Desarrollo de un lenguaje para la música audiovisual electroacústica: investigación sobre su comunicación y clasificación” (Hill 2010).

Influential Research Projects

Making Sense of Contemporary Dance

At first glance a study investigating audience reception to contemporary dance performances might seem to be rather distant from the audience investigation of electroacoustic audio-visual music but the aims of the project and the nature of the art form in question possess many similarities.

Renee Glass and Kate Stevens’ research project sought to investigate the level of insight that audiences were able to gain from contemporary dance performances and how they could help audiences to understand the works better. Their methodology provided some participants with different types of pre-performance information and some with no information. Participants were then asked to record their responses using what they term the Audience Response Tool (ART) that records psychological responses such as interpretation and enjoyment, as well as affective reactions including emotional response and visceral sensation. This ART is in-fact a questionnaire containing qualitative questions and quantitative seven-point Likert-type scales (Glass and Stevens 2005). 8[8. A Likert scale is a very common scale used in questionnaires in which respondees are asked to evaluate something in terms of their positive or negative reaction. For example: 1. Strongly Agree; 2. Agree; 3. Neither agree or disagree; 4. Disagree; 5. Strongly Disagree. This is an example of a five-point Likert Scale.]

Qualitative and quantitative questions, such as those contained within the ART, have advantages and disadvantages. The advantage of qualitative questions is that so long as they are constructed correctly they will allow the participant to record open and undirected responses. This is especially important for research investigating such a subjective topic as audience response to an artwork. However, a disadvantage of qualitative questions is that such results might not be analysed so easily due to the large amount of data and varied content of the responses. As a result it may be difficult to draw trends and conclusions from the data (Glass et al 2007).

Quantitative questions are very quick for the participants to complete and because the responses are limited to a simple range they are also very easily comparable. It is very simple to obtain statistical data from raw quantitative results. However the highly directed nature of such questions means that there is no depth of responses as with qualitative questions. The most significant ramification of this in terms of audience research is that quantitative questions may discover that participants like or dislike a work, but no contextualization around this information is provided. We can’t know the reasons why. Such questions may also direct audiences to think about a work in a particular way.

Utilizing a mixture of these two question types, such as within the ART, could be seen as an ideal solution but each of the questions in the ART deal with a different topic, the same thing is not asked in both a qualitative and quantitative way. Further to this some of the questions within the quantitative section of the ART could more than likely elicit more useful information if the question were phrased in a qualitative way. The clearest example of this is to evaluate the “Enjoyment Rating Scale”. This scale seeks to discern the factors that audience members found most enjoyable about the dance performance. Fourteen items are listed and audience members are asked to rate each of these using a seven-point Likert scale. There are a couple of significant issues with this. Firstly, any enjoyable aspects of the performance which fall outside of these 14 categories will be ignored. Secondly, the mere presence of suggested enjoyment factors listed before the audience member is likely to direct them to conform their responses to the scale itself, as opposed to allowing the audience member to record a free and open interpretation.

The Relationship Between Musical and Visual Components in Film.

Scott Lipscomb and Roger Kendall’s project investigated audience responses to twenty-five examples that were created from extracts out of a Hollywood film. Five film extracts were taken out of the film and the sound replaced with recordings containing only the orchestral score (thus all diegetic sonic material was removed). 9[9. The term “diegetic sound” refers to audio emanating from the film scene (for example music coming from a radio within the scene) while “non-diegetic” refers to audio that does not emanate from the scene itself and that is generally employed to add to the mood of the film (for example, up-tempo music in a fast-paced action sequence).] Students of film composition were then enlisted to swap the non-diegetic audio between the five extracts, thus creating twenty-five examples. The film composition students were asked to align the non-diegetic audio in the new clips so as to synchronise the sound and image as best as possible. Audiences were then asked to judge which example clip contained the most appropriate orchestral score, and to rate the clips in order of effectiveness. The researchers’ goal was to investigate if what they termed the “composer intended audio” was consistently recognised as the best fit.

Two significant problems become apparent quite rapidly when application of aspects from this methodology is considered for the current study. The first is that all diegetic sonic material is removed from the clip. This creates an artificial listening situation focussing audience attention fully on the orchestral score, which could have occupied a background position behind diegetic sound in the original film clip. It is also possible that the orchestral composer took cues from the diegetic audio materials that informed choices for the non-diegetic audio without which certain gestures or motifs may lack or lose significance. For the current project there is no such distinction between diegetic and non-diegetic audio but it important to be aware that any alterations made to test examples can dilute the original intentions of the composer, for example asking an audience group to rate a stereo rendering of an eight-channel piece will likely not provide the same responses as if the same audience group were presented with the full eight-channel version (Hill 2010).

The second issue with this test is that audience responses took the form of a rating scale and provide no contextual information or reasoning as to why a specific test example would be rated as higher than another. The researchers suggest possible reasons for the trends that become apparent in the results but while their theories may seem reasonable they are speculative without this more detailed compositional information.

The Intention/Reception Project

Leigh Landy and Rob Weale’s Intention/Reception (I/R) project was designed to gather audience response data for electroacoustic music (Landy 2006, Weale 2006). They investigated audience responses to various electroacoustic music works varying in their level of abstraction. Some works were soundscape compositions, others contained recognisable materials combined with more abstract sounds and the final type of work contained a higher level of abstract materials.

The intention/reception methodology asked audience members to complete a qualitative questionnaire detailing their interpretation of the piece and whether they would like to hear similar compositions in the future. 10[10. Detailed information on the content of the Intention/Reception questionnaire can be found in Weale 2006 and Landy 2006.] The entirely qualitative nature of the questionnaires was able to solicit an enormous amount of subjective responses for analysis. A number of questions asked were later evaluated to be ineffective as the response to these questions was too heavily reliant on outside influences as opposed to the test works (these questions were related to purchasing CDs or going to concerts and are unable to provide useful data due to audience members having different personal habits related to these) [Weale 2006].

One of the challenges for the I/R project that I became concerned about when thinking of applying a similar methodology to the current project was a balance between retaining audience engagement and repeat attendance. Audience response test sessions were evaluated to be most effective at 45 minutes in duration. This time period was enough to record the responses to a single piece without the group members becoming distracted or disinterested. This allowed for in-depth responses to all questions. However this also meant that multiple sessions were required with the same participants in order to collect a full data set. Unfortunately it was not always possible for participants to make all three sessions and so some data sets were incomplete. All of the collected data was perfectly usable in the context of the I/R project, but the responses of audience members to different works could not be compared where they had not seen all three.

The results of the project provided a rich set of data that was highly useful in gauging audience response to electroacoustic music.

Methodology of the Current Research Project

Evaluation of the methodologies in previous research projects has provided invaluable insight into the positive and negative aspects of certain research methodologies. The main stimulus and foundation for the current project has come from Landy and Weale’s Intention/Reception project (Weale 2006, Landy 2006). Combining what we have learned from the previous projects it is possible to distil an appropriate strategy for collecting audience response data for electroacoustic audio-visual music.

The Research Sessions

The most significant development for the current study was the decision to project each of the three test works within one research session. This would make each data collection session an hour and a half in duration (plus appropriate breaks in the session to retain participant focus and engagement) longer than the 45 minutes evaluated to be most effective, but would ensure that each participant provided data for all three test works. In order to combat the lack of focus or engagement over the extended group session short breaks in which refreshments were provided were introduced in order to encourage continued focus.

The use of a mixed quantitative/qualitative questionnaire was considered as a possible solution to cut down the duration of the research sessions, but it was decided that the trade off would not have been beneficial as this would have greatly reduced the depth and quality of the collected data.

The test questionnaires have been designed to solicit three main topics of data.

1. The material properties of the work.
2. The perceived meaning of the work and the audiences’ emotional response.
3. The audiences’ desire to see more/keep listening to similar compositions.

The questionnaires are not divided into three separate sections addressing each topic in turn, rather the questionnaire contains an assortment of questions each of which are applicable to one of the three topics (for example Questions 2 and 3 deal with the work materials, Questions 1, 4, 5 and 6 deal with the perceived meaning and audiences’ emotional response, while questions 9 and 11 deal with a desire to see more or keep listening). The questionnaire was designed in this way so as to avoid audience members passing over an entire section that they felt they might not like to answer.

Test Compositions

The compositions selected as test works within this research project were chosen from 37 submissions received when an open call was sent out via the <cec-conference> list in January 2009. 11[11. International mailing list for electroacoustics and related art forms <cec-conference>.] The relevant excerpts from the call are shown below:

This is a call for fixed media electroacoustic audio-visual music works exploring the interaction between sound and image. Works should be cohesive audio-visual entities and not just video/film with a soundtrack or music with a video/film track. The audio within these works should be electroacoustic in nature.

• In this phase of the project, submissions must be fixed media works (as opposed to ‘live’ performance based works).
• Works must have stereo audio and be for single screen.
• The duration of submitted works should be no longer than 8 minutes due to testing practicalities or, alternatively, a well-defined self-contained section of the work lasting no longer than 8 minutes can be identified within the entire work.
• The composer should own all rights to the submitted works and provide the researcher with the right to include selected works as part of future scholarly publications of the research.

Despite the requirements of this call a fifth of the received submissions were collaborative works and were discounted from the selection process. These works were eliminated because of the complications associated with assessing compositional intention. Where two people work collaboratively it is possible that they may not have exactly the same concept or vision for the final work. When audiences are attempting to infer any intentions or meaning from such works this greatly complicates matters as the intentions or meanings from each collaborator might be different.

A number of works were also received that were either audio compositions to an existing film composition, or film compositions made to an existing audio composition. Due to the project’s desire to research cohesive audio-visual compositions (as defined above) these works had to be discounted. These works also possess similar complications to that of collaborative work as there are multiple compositional intentions.

The requirement of fixed media compositions is so that multiple sessions can be run in which the compositions can be projected in a relevantly consistent manner. Testing audience reception to live performances, where performative interpretation will not always be consistent between each performance, means that different audience groups would be judging different realisations of the same works (any investigation in this field would also have to deal with the complication of both compositional and interpretative intentions). Stereo, single screen works were called for as the requirements for projection are a lot less intensive than for multichannel or multi-screen works and so it was envisaged that it would be much easier to set up a test situation in various locations.

Once works conforming to the stipulated requirements had been isolated a final selection had to be made. Three works were required, each representing a different compositional style. The decision to test three works was informed by requirements (duration vs. engagement) of the test procedure and a desire to test a range of compositional styles.

As part of the selection process I constructed a “language cube” based upon Simon Emmerson’s “language grid” (detailed information can be found in Hill 2010). Each of the three axes of this cube represents a different aspect of the composition, the nature of the audio material (ranging from real world to abstract), the nature of the visual material (ranging from real world to abstract) and the nature of the associations within the piece between the two (ranging from simple to complex).

Composition A: Elsa Justel — Destellos (2002 / 0:00)
This work contains audio and visual material that while in some instances is processed it generally retains its source references. The sound and image materials have been captured from similar sources (glass, water and metal among others) and the sound and image events are also largely synchronous resulting in a sound and image association that is less complex in nature. (Video 1)

Composition B: Thierry Gauthier — Portrait d’une femme (2008 / 8:49)
This work contains image materials that are largely unprocessed, and therefore relatively comparable to real world sources, but contains audio material that is processed, abstracted or synthasized. The association between sound and image within this composition is also much less direct than for the other two compositions, with the association operating on a long-term scale (based upon a construction of atmosphere) as opposed to synchronicity of shorter-term audio and visual events. (Video 2)

Composition C: Bret Battey — Sinus Aestum (2009 / 8:30)
Sinus Aestum (Bay of Billows) is an audio-visual piece that was entirely synthesised using a computer (as opposed to containing captured materials as in works A and B). The sound and image materials are largely abstract in nature but do contain motion and form that can be likened to patterns within nature (waves, flocking birds etc.) This composition contains sound and image associations that work to create audio-visual gestures and so operate on a level of complexity between that of the other two works. (Video 3)

Using this range of compositions, each different from one another, it is hoped that audience information will be obtained relating not only reception of audio-visual compositions in general but also audience reception of certain styles of work.

Outcomes

The methodology for this research project has been designed to discover how inexperienced audiences receive and interpret varying styles of electroacoustic audio-visual music. Questions dealing with the material properties of the work will provide an understanding of how audiences respond to abstract and recognisable “real world” materials, as well as their combination, in the context of an electroacoustic audio-visual work.

How these materials are combined and interpreted by the audiences will elicit certain emotional responses as audience members seek to piece together any narrative or meaning within the work. By collecting response data in these areas it is possible to see how well specific ideas or concepts are translated from composer to audience. Knowledge of this could allow composers to reflect on their compositional style and to adopt those methods that are proven to be more effective if they wish to communicate anything more explicitly or if they wish to avoid audiences making a specific interpretation.

Reception data combined with data from the second half of this test phase (dealing with contextual information, see Hill 2010 for more information) make it possible to evaluate the level of access that inexperienced audiences have to these works. Access information of this kind can provide invaluable insights into how it might be possible to help a wider audience to understand and appreciate electroacoustic audio-visual music.

Conclusion

Artistic works that combine sound and image have been conceived of for hundreds of years and with digital technology it has never been easier for an artist to compose both sound and image in the creation of a cohesive audio-visual work. Audience reception studies for electroacoustic audio-visual music have never been undertaken before but research projects in other artistic disciplines, even those investigating other art forms, can provide valuable insight for the development of effective methodologies. It is hoped that this paper might prove useful to anyone else wishing to construct an audience response methodology.

Results are currently being analysed from the first stage of testing and it is very exciting to be able to observe trends emerging from the data. The full results of this study will be available next year in the publication of my PhD thesis.

Bibliography

Brougher, Kerry (Ed.). Visual Music: Synaesthesia in art and music since 1900. London: Thames & Hudson, 2005.

Chion, Michel. Audio Vision: Sound on screen. New York: Columbia University Press, 1994.

Collopy, Fred. “Colour, Form and Motion.” Leonardo 33/5 (2000), pp. 355–360.

Glass, Renee and Kate Stevens. “Making sense of Contemporary Dance”. Fuel4arts Report (February 2005).

Glass, Renee, Kate Stevens and Stephen Malloch. “Audience Response Tool — Information.” MARCS Auditory Labs, UWS, 2007.

Hill, Andrew. "Desarrollo de un lenguaje para la música audiovisual electroacústica: investigación sobre su comunicación y clasificación". En el Límite — Escritos Sobre Sonido, Música, Imagen y Tecnología, pp. 144–165. Editado por Universidad Nacional de Lanús, 2010; compilado por Raúl Minsburg.

Landy, Leigh. “The Intention/Reception Project”. Analytical Methods of Electroacoustic Music. Edited by Mary Simoni. New York: Routledge, 2006, pp. 29–53.

_____. Understanding the Art of Sound Organization. Cambridge MA: MIT Press, 2007.

Lipscomb, Scott and Roger Kendall. “Perceptual Judgement of the Relationship Between Musical and Visual Components in Film.” Psychomusicology 13 (1994), pp. 60–98.

Ox, Jack and Cindy Keefer. “On Curating Recent Digital Abstract Visual Music.” The Abstract Visual Music Catalog. The New York Digital Salon, 2006. Available online at http://www.centerforvisualmusic.org/Library.html (Last accessed 8 July 2010)

Weale, Rob. “Discovering how Accessible Electroacoustic Music can be: The Intention/Reception Project.” Organised Sound 11/2 (August 2006) “Identity and Analysis,” pp. 189–200.