Lumia and Visual Music
Using Thomas Wilfred’s Lumia factors to inform audiovisual composition
The Clavilux, “a name derived from the Latin, meaning light played by key” (Stein 1971, 4), was a device invented by Thomas Wilfred that permitted performances with coloured light. Compared to colour organs, which commonly mapped musical pitch to the rainbow hue scale, the Clavilux had additional capabilities, allowing the form and motion of the colours to be manipulated. The three primary factors of form, colour and motion were used to create a new art form Wilfred called “lumia” (Wilfred 1947, 247). Wilfred designed and made several lumia-generating Clavilux instruments, some of which were self-operating and internally programmed, and others that could be performed in real time. The visual output of the instruments was a light projection and it was capable of creating complex visual arrangements:
A typical composition contains one principal motif with one or more subordinate themes. Once chosen, they vary infinitely in shape, color, texture, and intensity. The principles evident in plastic and graphic compositions — unity, harmony, and balance — function kinetically in lumia. When movement is temporarily suspended in a lumia composition, the result is a balanced picture. However, the static picture’s ultimate meaning can only be seen in relation to what follows it. (Stein 1971, 3)
Wilfred described lumia in detail. Form, colour and motion also had four subfactors each, and he grouped these into a “graphic equation” (Fig. 1). The combination of light’s lumia factors to the left of the equation produced the artistic potential on the right.
Place this inert potential in a creative artist’s hand, supply him with a physical basis — screen, instrument and keyboard — and when his finished composition is performed, the last link in the chain has been forged and you have the eighth fine art, lumia, the art of light, which will open up a new æsthetic realm, as rich in promise as the seven older ones. (Wilfred 1947, 254)
Wilfred believed he had laid the foundations for a new art form that would continue to be studied and practised long into the future, but there appears to be limited work subsequently using lumia or the Clavilux instruments. The qualities of lumia can, however, be used as inspiration when creating fixed media audiovisual compositions and some of the relationships between lumia and video and sound parameters were used in the process of composing my 2010 work, Space Movement Sound.
Form in Visual Music
In many artistic disciplines the subjects and forms of the artwork are an important focus, rather than the colours from which they are created. In classical sculpture, a bust may be carved, or otherwise formed, from a single material of uniform or near-uniform colour. The sketching artist may use a graphite pencil to quickly draw the shapes of people and other objects. In photography, black and white images are still used, and occasionally films are made either completely or in part without colour. The subjects or objects in films are often the important focus, possibly because they are something with which the audience can relate or identify. Colour can often be subjugated by form and motion — or even omitted altogether — without modifying the visual narrative. Visual music compositions also frequently contain objects or forms, however abstract, whose behaviour is central to the composition. Such objects are rarely static, and their movement is often used to inform the modulation and tempo of the music. This point is highlighted by William Moritz’s quote of Oskar Fischinger’s work plan for constructing R-1 Formspiel (Form Play):
Points of Staffs start to dance slowly, rhythmically, and arise gradually up to the middle of the picture. The tempo at first is practically non-existent, and then begins only slowly to become perceptible. Now single pieces grow out above the general line and take the lead in a particular way. Then they destroy the uniform line and attempt to lead an individual, independent life. (Fischinger cited in Moritz 2004, 176)
These notes describe an abstract piece that concentrates on the evolution and movement of variously formed objects. Although very descriptive in terms of movement and pace, the abstract forms that the objects possess appear to play an important role in the design of this piece. The father and son team James S. Hegarty and James H. Hegarty take a more a more minimalist approach to form in their 2004 work Aerial, through the use of a rendered silk cloth that continuously moves and reshapes. Accompanied by a treated vocal harmony, the changing form of the square silk cloth becomes the focus of the audiovisual experience. It is difficult to separate the form from the movement to which the cloth is being subjected but, as the cloth shifts and reforms, it is reflected by changes in the vocal soundtrack.
Colour in Visual Music
Can colour exist without form and motion? Possibly the closest we can get to this state visually is a colour field, an area of colour that engulfs the visual domain. As Patricia Sloane describes it:
The color field is an area of any single homogeneous color extending as far as the observing eye can see at that moment. This translates, in experiential terms, to the experience of seeing nothing but, say, light green. (Sloane 1989, 168)
In practical terms, this visual sensation could be achieved by holding a smooth-surfaced piece of monochromatic coloured card in close proximity to the eyes. Under uniform lighting, a constant colour would be perceived. If the card were to be moved away from the eyes, the colour would eventually take on the rectangular form of the card and the pure experience of colour would be lost. This is similarly the case for colours that are rendered onto a computer or other screen. Although the screen may be filled with a single colour, it is constrained by the physical shape of the display. In painting, a canvas filled with a single hue would be very minimal, but some artists have created works that approach this. One example is Vir Heroicus Sublimis (Man, Heroic and Sublime) by Barnett Newman (1950):
Newman was one of the several Abstract Expressionists who eliminated signs of the action of the painter’s hand, preferring to work with broad, even expanses of deep color. Vir Heroicus Sublimis is large enough so that when the viewer stands close to it, as Newman intended, it creates an engulfing environment — a vast red field, broken by five thin vertical stripes. (Museum of Modern Art 2004, 166)
The painting is two metres high and five metres wide. It is filled primarily with a vast expanse of red colour with five narrow, vertical and differently coloured “zips”. As suggested by the Museum of Modern Art in their 2006 gallery notes for the work, this painting is to be experienced closely; the shape and texture of the canvas are not as important as the colour and its depth and the observer’s interaction with it.
In contemporary visual music there are examples where colour has been used in relatively pure ways with abstract forms, such as Andrew Hill’s Phase (2011). Although the composer describes the work as a study exploring the transformation between states, Phase uses slowly evolving colours and abstract cloud-like formations to convey this transformation. 1[1. Andrew Hill and Jim Hobbs discuss their expanded audiovisual performances in “(I)mage Sound(s): Expanded audiovisual practice,” also in this issue of eContact!] The result is an organic, evolving colour space. A similar use of abstract colour spaces can be found in the middle of “Colour Blends”, the first part of my own Space Movement Sound (Video 4, below, from 1:10–2:00).
Motion in Visual Music
Filmed recordings implicitly contain motion. Even if the camera recording a scene was in a fixed position, there will be some movement of light and shadow over time. If a series of still images were presented as a film or slideshow there would be movement, or light changes, as the images were transitioned. These are quite minimal examples of motion; it is more common for objects to move in an obvious, and often gestural, manner in visual music. Viking Eggeling’s 1924 animated film exploring temporal design and motion, Symphonie Diagonale (Diagonal Symphony), could also be interpreted as a study in geometric form, but the diagonal movement of abstract objects dictates the pace of the animation.
A less overt approach, but nonetheless an effective use of motion, is encountered in Heliocentric (2010) by Semiconductor (Ruth Jarman and Joe Gerhardt). This piece used time-lapse recordings of the sun passing across various landscapes. The motion of the sun is tracked and accentuated by the music, which is predominantly constructed from a high-pitch ringing tone. As the name Heliocentric suggests, the Sun is always at the centre of the screen. The ringing tone that depicts it is therefore always centrally panned in the stereo image (Video 2). As objects in the landscape obscure the sun, the ringing tone becomes quieter. When the screen is brightened, as clouds diffuse the sunlight, the sound becomes louder. Although seemingly simple in concept, the quality of the video recordings and the attention to detail in the soundtrack combine to make the motion of the sun very apparent, although, in reality, it is the earth and landscape which is moving.
The entire “Motion Shifts” section of Space Movement Sound is designed to suggest movement in a more exaggerated way. The captured footage already contains movement, but the way it is treated and the nature of the edits between shots were designed to enhance this. Changes in lighting also suggest motion, even when the objects are relatively static.
Combinations of Form, Colour and Motion
Another way of considering the lumia factors is to examine the results when two of the three factors are paired. Three paired combinations are possible (Fig. 2), and Wilfred assigns them varying degrees of importance:
Form, color and motion are the three basic factors in lumia — as in all visual experience — and form and motion are the two most important. A lumia artist may compose and perform in black and white only, never using color. The use of form and color alone — static composition with projected light — constitutes a less important, but still practical field in lumia. The only two-factor combination that cannot meet the requirements is motion and color, without form. This is because it violates a basic principle in vision. The human eye must have an anchorage, a point to focus on. If a spectator is facing an absolutely formless and unbroken area of color, his two eyes are unable to perform an orientational triangulation and he will quickly seek a visual anchorage elsewhere, an apex for the distance-measuring triangle that has its base between the pupils of his eyes. (Wilfred 1947, 252–253)
Form and motion also pair naturally in film and animation, and moving objects are commonly seen in visual music compositions. Colour is not essential — monochromatic animations can still hold visual interest, as evidenced in Symphonie Diagonale (Eggeling 1924) and in the “Motion Shifts” passage in Space Movement Sound. Colour paired with form is the next combination and this can be achieved by the projection of coloured light. This combination would be experienced at music concert light shows and colour organs would have produced a similar effect. In visual music composition, this pairing would equate to a stationary coloured form. Without motion, the shape and light would remain static on the screen. This can be used in compositions, but with movement, additional compositional opportunities are available. The lumia pair that does not naturally relate intuitively to real-world experience is that of colour and motion. Wilfred suggests this combination is not possible, as colour cannot move without holding a form. If it were possible to see a colour move then it must at least have a boundary line between itself and another colour. The colour would then be holding a basic form. Norman McClaren and Grant Munro later demonstrated, however, that in animation, motion can relate to change (McClaren and Munro 1978). Changes in the lighting, texture and colour of static objects are movements between states and are therefore a subtle type of motion.
Sonic Equivalents of Lumia
Although lumia is intended to be a silent art form, it is possible to compare its factors with their sonic equivalents. Wilfred’s further division of form, colour and motion into their subfactors can be used in this process, as described below.
Sound Form
According to Wilfred’s schema, four subfactors can be attributable to form:
LOCATION — Where is it?
VOLUME — How big?
SHAPE — What is it?
CHARACTER — What is there about it? (Wilfred 1947, 253)
In music, form is commonly used to describe the compositional structure. Wilfred’s definitions of its subfactors refer, however, to individual lumia, so the aim here is to determine qualities of individual sounds relating to these. The first of his four subfactors is location, which, in audio terms can be related to the spatial positioning of a sound. Across a multi-channel loudspeaker array, a sound can be panned anywhere in the sound field. This pan position could also be changed dynamically and will be dealt with below in the description of motion and its subfactors. Between 4:13 and 4:25 in Space Movement Sound, video footage of flowing water appears first to the left and then to the right of a split screen. The audio recording of water naturally pans left and then right to match its visual position.
Another possible relationship between location and sound is the one between physical height and pitch. This concept was addressed by Roger Shepard (2001) when examining methods of representing pitch, height and something he labels “chroma”, to refer to the pitch of a note, independent of its octave. 2[2. Note that Shepard is using chroma here to refer to musical pitch, whereas Wilfred’s chroma is a subfactor of colour.] The note “C” is a chromatic value, but there is a middle C, C an octave above this and so on. Shepard went on to say that
in the world, height and pitch are almost always linked, and things that ascend in height usually ascend in pitch. (Shepard 2001, 160)
Although a commonly used concept, objects that move in the vertical plane are not necessarily represented by a changing pitch value, but it could be a consideration when creating visual music. Volume, form’s second subfactor, can be associated with a sound’s amplitude. This has a basis in physical phenomena, as summarized by Smalley (1986):
In the environment, when a sound approaches the listener its spectral and dynamic intensity increase at a rate proportional to perceived velocity. Moreover, the increase in spectral intensity permits the revelation of internal spectral detail as a function of spatial proximity. (Smalley 1986, 68)
An approaching sound could be created by a car, for example. As it approaches, it is perceived visually as becoming physically larger. The sound of the engine becomes louder to accompany this. The two remaining subfactors, shape and character, are suggestive of timbral qualities. Timbre applies more naturally to colour than form and will be examined below. One other relationship that could apply, however, is to the sound’s envelope. Percussive and gentler sounds, for example, have very different characters and envelope shapes.
Sound Colour
Wilfred also defines four subfactors for sound colour:
HUE — What color is it — red, green, blue?
CHROMA — How much gray has been mixed with the pure hue?
VALUE — How much white in that gray?
INTENSITY — How strong is the light it sheds? (Wilfred 1947, 253)
Colour has several relationships to sound and Wilfred’s subfactors of hue, chroma and value can be equated to the hue, saturation and lightness (value) parameters of the HSV colour system. This system was used to develop a sound selection system by Stephen Barrass (1997, 127–137) and is one possible method of mapping between colour and sound in a more precise way. In Barrass’ experiment, hues were mapped to different timbres, saturation to their pitch, and value (lightness) to how much high-frequency content they contained.
The final subfactor, intensity, could once again refer to the sound’s amplitude. This appears to be a duplication of the volume subfactor of form, but does not confuse the issue as a large intensely coloured object could also be represented with a loud sound.
Aside from Wilfred’s definition, colours can be described as possessing spatial qualities. They fill the space between painted lines and object forms and could therefore be portrayed in music with sounds that naturally fill the aural space. This can be accomplished with sounds of longer duration that would fill the temporal space. Extended sounds of this nature could be more textural, with gradual envelope characteristics and a focus on timbral transformation.
Sound Motion
The third lumia component, motion, also has four subfactors:
ORBIT — Where is it going?
TEMPO — How fast? Speeding up? Slowing down?
RHYTHM — Does it repeat anything?
FIELD — Is it constantly visible, or does any part of its orbit carry it beyond the range of vision? (Wilfred 1947, 253)
Orbit and tempo can be associated with the spatial positioning of sounds in a similar way to the location subfactor of form, but in this case motion is applied. If an object moved across the screen from left to right, this could be accompanied by a sound that also panned left to right. The faster the object moves, the more rapid this audio pan would be. Composers often use this effect and several examples can be found in Joseph Hyde’s 2009 work, End Transmission (Video 3).
Rhythmic movement of on-screen objects could also be represented by a similar panning of the sound, or by a percussive or melodic musical rhythm. The “field” subfactor is less intuitively linked to sounds, but could be said to apply to one that becomes silent. In Wilfred’s discussion, an object carried “beyond the range of vision” (Wilfred 1947, 253) would not be seen. Similarly, a sound object carried beyond the range of hearing would not be heard. Alternatively, if a sound was tracking an object that moved off-screen it could actually appear to also move off screen. This is a phenomenon Michel Chion calls “spatial magnetization” and is perceived even if there is only a single speaker producing all the sound.
If the sound that comes from the fixed speaker is attributed to an onscreen character, and if we see him or her move to the right, we are going to hear the sound move to the right; if the character exists offscreen, we hear the sound as outside the screen too. The phenomenon of spatial magnetization, whereby our attribution of a sound’s location depends on what we see of the real or supposed source, can be observed on countless occasions every day. (Chion 2009, 248)
“Space Movement Sound”
The preceding analysis of visual music in relation to lumia factors and their sonic equivalents led to the development of Space Movement Sound. Not all the audiovisual analogies discussed above were implemented in a literal sense, but they did, as a theoretical basis, inform some of the compositional details and concept for each of the two études that form the work. Since form and motion grouped together most naturally, they were the main compositional tools use to create the second passage, “Motion Shifts”. Colour therefore remained to be explored, and constitutes the focus of the opening section of the work, “Colour Blends”.
Motion Shifts
Form and motion combined naturally in “Motion Shifts”. Motion is evident throughout the piece and various forms are transformed to produce a constantly changing visual experience. Changes in light intensity, often in a strobing manner, also suggested movement. Although colour was not compositionally important, there are subtle colour effects evident on closer inspection. Primarily, however, low colour saturation video footage was used, and in several instances the images are simply greyscale. This created a contrast between the more deliberate uses of colour exhibited in “Colour Blends”. The dynamic movement and transitions in the video footage encouraged tight synchronization between sound and image. This was not without certain problems, as it was difficult for the sound to follow a straight video cut from one scene to another. Translating this directly would have meant an abrupt switch between two different recordings. Some of the transitions were therefore softened by use of volume and crossfades, or through the addition of reverb and delay effects that extended the edits for a short time. Although the audio did not change as immediately as the image, the illusion of a rapid audiovisual switch was maintained.
Colour Blends
Colour was the remaining lumia component to be investigated. Isolating colour in this way is difficult since, following Wilfred’s discussion, colour cannot exist completely without a form. To alleviate this, some compromise was made. Subtle suggestions of form and motion were added to create abstract colour textures in a slowly evolving organic animation. The focus was shifted towards the textural qualities and away from gesture. Many different textures were produced with different colour gradients and blending operations carried out between them. This created many different colour shades throughout the étude.
A consequence of minimizing form and motion in this way was that the visual composition had limited punctuation and transients, which translated to reduced dramatic musical gestures when compared to “Motion Shifts”. Instead, the spatial nature of the colours was emphasized and the soundtrack composed with sounds that filled the aural space. From a musical standpoint, sounds with greater spatial characteristics are washes of sound, sonic airbrushes, or what in synthesis terms are commonly referred to as “pad sounds”. Using this type of sound, its timbre, pervasiveness and gradual transformation were more important than its pitch articulation and phrasing. Apart from synthesizer pads, transformation and time stretching of found-sound recordings resulted in timbres having a textural quality. Outdoor field recordings also inherently contained their own spatial qualities acquired from the environment in which they were recorded.
Summary
The composition of Space Movement Sound demonstrated how lumia factors could influence the production of audiovisual fixed media. Slowly changing colours were combined with evenly paced textural sound, and form and motion used to create a more dynamic and synchronous audiovisual experience. Lumia’s form and motion factors applied most naturally to fixed-media composition. The subfactors relating to position, movement, shape and volume could be portrayed easily in both sound and image. The gestural behaviours of moving objects and edits in video also suggested similar qualities in music. Composing to the dynamically changing video was therefore a relatively simple task. Conversely, composing music to the textured colour animations in “Colour Blends” proved more of a challenge. With limited visual change and movement, the sounds were sustained for longer periods. This resulted in little visual or auditory punctuation, creating stasis and repetition in the audiovisual experience. The nature of this étude, however, highlighted the relationships between reduced visual gesture and textural sound that are described in the sixth chapter of my PhD thesis (Payling 2014). Even though gesture was minimized, it presented opportunities to work more intently on gradual sound transformations.
Bibliography
Barrass, Stephen. “Auditory Information Design.” Unpublished doctoral dissertation, The Australian National University, 1997.
Chion, Michel. Film: A Sound Art. New York: Columbia University Press. 2009.
Eggeling, Viking (Dir.). Symphonie Diagonale. Animated film. 1924.
Fischinger, Oskar. “A Document Related to R-2.” In William Moritz, Optical Poetry: The life and work of Oskar Fischinger, pp. 176–178. Eastleigh, UK: John Libbey Publishing, 2004.
Hegarty, James H. and James S. Hegarty. Aerial (2004).
Hill, Andrew. Phase (2011). Vimeo video “Phase” (5:28) posted by “Andrew Hill” on 2 March 2011. http://vimeo.com/20562897 [Last accessed 22 December 2016]
Hyde, Joseph. End Transmission (2009). Vimeo video “End Transmission” (10:06) posted by “Joseph Hyde” on 16 February 2009. http://vimeo.com/3241660 [Last accessed 22 December 2016]
Jarman, Ruth and Joe Gerhardt. Heliocentric. (2010). Vimeo video “Heliocentric” (3:35) posted by “Semiconductor” on 11 December 2009. http://vimeo.com/8129736 [Last accessed 22 December 2016]
McLaren, Norman and Grant Munro (Dirs.). Animated Motion: Part 5. National Film Board of Canada 1978. http://www.nfb.ca/film/animated_motion_part_5 [Last accessed 22 December 2016]
Moritz, William. Optical Poetry: The life and work of Oskar Fischinger, pp. 176–178. Eastleigh, UK: John Libbey Publishing, 2004
Museum of Modern Art. “Barnett Newman: Vir Heroicus Sublimis (1950–51).” 2006. http://www.moma.org/collection/works/79250 [Last accessed 22 December 2016]
Museum of Modern Art. MoMA Highlights: 350 Works from The Museum of Modern Art. 3rd Revised Edition. New York: The Museum of Modern Art, 2013.
Newman, Barnett. Vir Heroicus Sublimis. 1950–51. Oil on canvas. New York, USA: Museum of Modern Art. http://www.moma.org/collection/works/79250
Payling, David. Space Movement Sound (2010). Vimeo video “Space Movement Sound” (6:35) posted by “Dave Payling” on 8 October 2010. http://vimeo.com/15660433 [Last accessed 22 December 2016]
_____. “Visual Music Composition with Electronic Sound and Video.” Unpublished doctoral dissertation, Staffordshire University, 2014.
Shepard, Roger. “Pitch Perception and Measurement.” In Music, Cognition and Computerized Sound: An introduction to psychoacoustics, pp. 149–166. Edited by Perry R. Cook. Cambridge MA: The MIT Press, 2001.
Sloane, Patricia. The Visual Nature of Color. New York: Design Press, 1989.
Smalley, Dennis. “Spectro-Morphology and Structuring Processes.” In The Language of Electroacoustic Music, pp. 61–93. Edited by Simon Emmerson. London: MacMillan Press, 1986.
Stein, Donna. “Thomas Wilfred: Lumia.” Press Preview. New York: Museum of Modern Art. 9 August 1971. http://www.moma.org/d/c/press_releases/W1siZiIsIjMyNjc2MCJdXQ.pdf?sha=e38815bab435c4e0 [Last accessed 14 September 2017]
Wilfred, Thomas. “Light and the Artist.” The Journal of Æsthetics and Art Criticism 5/4 (June 1947), pp. 247–255.
Social top