Design and Appropriation in Open Source Computer Musical Instruments

A Case study of the Silent Drum

by Jaime Oliver La Rosa

Background

Computer musical instruments function today as codes rather than as closed material objects. They can be widely exchanged and thus regarded as “open”. Because of their accessibility through detailed tutorials and programming languages, computer musical instruments are open to appropriation and modification — anywhere and by anyone with an Internet connection. In order to inform our understanding of how computer-based musical instruments are constructed and exchanged today as a consequence of a few historical developments, I will use as a case study some of the replications of the Silent Drum controller (Oliver and Jenkins 2008).

It is not my goal to do an in-depth analysis and classification of computer musical instruments, though one is certainly lacking. And while we can surely conceive of instruments not intended for live human performance, here I will talk about those that do have an interface meant for human interaction. These are often called gestural controllers or instruments, but I will refer to them as “computer-based musical instruments”, or simply as “computer instruments”. It is also not my intention to draw a distinction between instrument and controller. Suffice it to say that I strongly believe that instruments can be stable, unstable, ephemeral and unrepeatable configurations of bodies, objects, digital devices, analogue equipment, code and/or elements of the environment. In this sense, any controller has the potential to become a part of an instrument.

Theremin as an Early Precursor for Open Source DIY Culture

Introduced to the world by Leon Theremin during a 1927 European and US tour, the theremin immediately captivated the global public. Giving concerts and demonstrations to audiences that often surpassed theatre capacity and drawing newspaper headlines from Mexico to India, the instrument excited people’s imagination about “the music of the future.” Paradoxically, the theremin’s fame was largely due to its expressive potential for playing traditional Western music repertoire and its perceived association with the human voice and the bowed string; that is, what validated the new electric medium was not the new and exciting music produced with it, but rather the ability of performers to make the instrument resemble voices and violins playing 19th-century tonal repertoire.

Encouraged by the instrument’s media frenzy and the perception that it was an instrument anybody could play, Radio Corporation of America (RCA) decided to commercialize it. After disappointing sales, a legal patent suit and several problems regarding technical support for malfunctioning units, RCA decided to release the schematics along with a DIY manual to repair theremins, and subsequently discontinued production. As theremins weren’t available commercially anymore, a used theremin market emerged through newspaper classifieds, and a growing community of “tinkerers” set out to build their own instruments by following, and often writing, schematics and tutorials on how to build them.

As a consequence of its early fame, a failed commercial enterprise, and Leon Theremin’s sudden return to the USSR in 1938 (and subsequent imprisonment), the theremin may be considered the first open source electric musical instrument, and the schematics the code for its exchange (Oliver 2012). One of the most significant cases of code exchange in DIY theremin culture is that of Bob Moog, who started his synthesizer-building career as a theremin maker. After reading an article by Ernest J. Schultz published in Radio and Television News magazine in 1949, Moog built his first theremin and in 1954 published a tutorial and schematic for his own version of the instrument in the same magazine (Moog 1954). Moog became a small-scale theremin builder, constructing each unit by hand and making available to the public the first commercial model in 15 years (Pinch and Trocco 2009).

Interface and Code

One salient distinction between computer-based musical instruments and their electronic predecessors and contemporary counterparts is between physical interface and code. While in the case of electronic devices the schematic is a code for the reconstruction of a physical device, in computers and digital devices, digital code replaces most dedicated electronic circuits and continuous voltages with digital representations on which computer code operates. Furthermore, electronic devices like the theremin perform a single function, while computers can be a musical instrument at one moment and an email service at the next — or even at the same time.

As a consequence, computer instruments with gestural controllers have a dual nature as material objects and as computer code. To a great extent then, computer instruments can be considered new media objects — in the sense developed by Manovich — and as such obey the principles of numerical representation, modularity, automation, variability and transcoding (Manovich 2001, 27–48). In this way, any part of the instrument’s code can be appropriated and repurposed, transformed, recombined or inserted into a larger design. As we will see below, the logic behind the principles of new media often extend into the instrument’s material interface, and many of the changes that happen at the level of code have a material correspondence.

To replicate a computer instrument then, one needs to build (or buy) a hardware device equipped with at least one sensor and connect this device to a computer to operate on this input with computer code. In order to share a computer DIY device with the public, or to build one, one needs to write or follow a tutorial that gives detailed instructions, parts lists and access to computer code. More importantly, tutorials and code provide a point of departure for new designs that appropriate, transform and repurpose a design.

The Silent Drum Controller: DIY and Open Source

DIY Instrument Design as a Compositional Practice

I have been working on designing and composing with my own instruments since 2008. Two of my most representative devices are the Silent Drum (Oliver and Jenkins 2008) and MANO (Oliver 2010) controllers, both of which use computer vision techniques to continuously track and classify hand gestures. The Silent Drum (Fig. 1) consists of a black elastic surface that is depressed to create shapes that contrast with a white background. It is not the aim of this paper to explain how the Silent Drum works in detail, as this is well documented elsewhere. 1[1. See Oliver and Jenkins 2008, as well as the author’s website for texts about, pieces for and notation of the instrument.]

The impulse to build this drum was an æsthetic one. For a long time I had been searching for a way to interact with a computer that would allow for “visceral” (Moore 1988) and “intimate” (Wessel and Wright 2002) control of computers and sound. Dissatisfied with the controllers readily available at the time, I set out to build controllers through an “enactive” (Wessel 2006) approach to computer music performance. That is, I was searching to build a controller that attempted to move beyond the “trigger” paradigm of dominant devices featuring keys and buttons, and which allowed for the continuous, multidimensional control of sound.

I have since developed several compositions with the Silent Drum, such as the 2008–09 work Silent Construction 1 (Video 1) and 3 Environments (2010–11). 2[2. More information about these and other works can be found in the “Works” section of the author’s website.] The title Silent Construction intends to capture the process of composing while constructing an instrument. In this process, design decisions about hardware and software are modulated by æsthetic needs, while at the same time, the potential for sound-shaping actions are constrained by the instrument and the body of the performer. This feedback loop is at the centre of my compositional (and instrument design) practice and is yet another instance of the way contemporary musical practices break the barriers between instrument designer or luthier, composer, and performer, a role which I often inhabit myself.

Repurposing Video Devices

The first version of the Silent Drum used an analogue security camera with a PCI capture card (Fig. 2). That version relied on the security industry’s need for a low-latency system and therefore both hardware and drivers were readily available.

It is commonplace today to see computer music applications that make use of game controllers as human-computer interfaces for live performance. Traditional game controllers with buttons and joysticks, Wii controllers based on accelerometers and, somewhat more recently, cameras such as the PlayStation Eye (PS Eye), and particularly the Kinect have gained the most attention.

The PS Eye is a USB camera that can capture video at 120 frames per second (fps) at a resolution of 320x240 pixels, or 60 fps at 640x480 pixels. Introduced in 2007 at a retail price of US$40, it is now sold online for less than $5. Compared to high-end industrial computer-vision cameras selling for hundreds and often thousands, the PS Eye is certainly more limited in terms of functions and resolution, but it is much faster and better quality than consumer webcams designed for use with computers.

The second and current version of the Silent Drum (Fig. 1, above) uses a PS Eye camera as a sensor. With interest in multi-touch surfaces at a peak, the late 2000s saw a swarm of coders and users around the world hacking and sharing drivers to repurpose a variety of cameras away from the manufacturer’s original intended application. The first Linux drivers for the PS Eye were provided by hacker “Kaswy” and are still available on the web. Today, it is possible to use this camera with the Linux UVC driver now installed by default in Ubuntu.

Open Source

The Silent Drum and MANO controllers were developed by using and hacking open source software. Aside from using Linux and the hacked camera drivers mentioned above, the tracking and sound software were built within the Pure Data (Puckette 1996) and GEM (Danks 1997) open source programming environments. Furthermore, the computer vision algorithm of the Silent Drum was built by learning from and modifying GEM and Pd objects; the chance to see the code of these environments’ objects as well as their GPL license terms was a determinant factor in my ability to learn to code in these environments and ultimately to create new work.

Consequent with open source philosophy, the code for the Silent Drum patches and external objects are open source too and were released under a GPL license. Furthermore, pix_drum, the computer vision object that tracks performers gestures, is now included in the GEM distribution. After several requests to make and to teach how to make a Silent Drum, I published a tutorial on how to build it and included the patches for my own compositions. 3[3. Available in the “Software/Tutorials” section of the author’s website.] These compositions, then, are also open source and can be performed in the state I left them or transformed into new works.

Replication and Appropriation of the Silent Drum

The Silent Drum instrument consists of a physical interface, tracking software and sound software. Any of these elements can be modified or replaced. Since the publication of the tutorial and the release of the code, there have been multiple replications and transformations of all of the aforementioned parts. While some people seek to replicate the device to be able to perform an existing piece, some take the drum only as a point of departure for different æsthetic projects altogether.

Replication

Students searching to learn by example, musicians searching for new works to perform and artists in search for new media for their works often find inspiration in existing works, and find information on the web. In 2009, percussionist Brian Archinal was the first person besides myself to build the Silent Drum and performed Silent Construction 1 in an international percussion competition. 4[4. The jury did not, however, consider the Silent Drum a percussion instrument and disqualified the performance.]

In 2010, composers Alec Hall and Bryan Jacobs, then at Columbia University’s Computer Music Center and founders of Qubit (a contemporary music and performance art initiative in New York), built a Silent Drum and wrote and commissioned works for guitar, saxophone and Silent Drum. Along with composers Aaron Einbond and Paul Clift, they created new work and developed new sound software for the drum to control, as well as new notational practices (Fig. 3).

In 2012, University of Lethbridge student Mathew Hellawell and Texas Christian University student Rodrigo Díaz both built, composed and performed with the drum. I have encountered other examples (with unfortunately very little documentation) on the Internet showing attempts at building a Silent Drum, and there might be more attempts at replication that I am not aware of.

In the cases described here, the Silent Drum’s physical interface and the tracking software as I developed them were integrally replicated, and the sound software was modified or re-written to create new works. The use of the Silent Drum in such cases sits closer to the traditional use of sound controllers, where users do not modify the controller at all, but rather use its output to produce their own creative sonic work.

Appropriation

As the code for the instrument and a tutorial for its construction are available to anyone who wants to use them, there exists also the possibility of individual users modifying the code or its physical construction and, hence, the instrument to suit their own needs. In fact, several people have already done so, appropriating the code or the idea of the instrument in order to explore the potential of the instrument “as instrument” for music or dance performance, or for cross-cultural integration, for example.

Peeravich Panlertkitsakul — The Box

Originally from Bangkok, Thailand, Peeravich Panlertkitsakul is pursuing a double degree in sound and electrical engineering at the University of Michigan. Panlertkitsakul recently built a sort of double-headed Silent Drum with elastic heads at two opposite faces of a cube-shaped box (Fig. 4). It is played from both sides in an accordion-like manner, and it is (for the moment) called “The Box”. Peeravich transformed the physical interface of the drum in a modular fashion and then sent the tracking data to Ableton software to control his musical material. More importantly, the “box” is treated as a module and connected to other elements of a larger setup that he uses to perform techno music.

Ricardo Brazileiro — “Iluminado”

Since 2009, Brazilian media artist Ricardo Brazileiro has been working on a project that could perhaps be best understood as a Silent Drum “fork” 5[5. In software engineering, a project “fork” happens when developers take a copy of source code from one software package and start independent development on it, creating a distinct and separate piece of software.] from an Afro-Brazilian perspective. He has modified an Ilú, a traditional Afro-Brazilian drum, by attaching the tracking hardware and software of the Silent Drum to it. With Iluminado 6[6. The name of his project combines the name of the drum with the idea of being “illuminated” and its associated religious connotations, while at the same time playing with the lighting used for video tracking purposes.], Brazileiro’s intention is to integrate Afro-Brazilian instrument builders and hackers, and to mesh traditional knowledge with digital approaches to music. The project was a collaboration with the Associação Recreativa Carnavalesca Afoxé Alafin Oyó. 7[7. The Afoxé Alafin Oyó Recreational Carnival Association.] He often performs with ensembles combining traditional Afro-Brazilian afoxé 8[8. A secular manifestation of candomblé.] using sounds that are generated and controlled by the computer (Fig. 5), and also presents it as an installation. Brazileiro also uses Iluminado’s openly gestural qualities to teach computer music in a more intuitive and attractive way in classroom, concert and installation settings, and he has taught several people to perform with the new instrument.

Brazileiro’s transformation of the Silent Drum speaks to the variability of computer music instruments. In Iluminado, the physical interface and sound-producing software are modified, but not the tracking software. By replacing a used tom shell with an Afro-Brazilian drum, Iluminado makes an artistic and cultural statement.

Menno Van der Woude — “Veerkracht”

Menno Van der Woude is a dancer and audiovisual programmer in the Netherlands. He recalls that when he first saw the Silent Drum, he thought about the possibility of making a larger version in which “the drummer would be in the drum.” Van der Woude decided to act on this initial impulse and built a large construction entirely covered with a membrane of elastic material that the dancers — performing inside the construction — could push outwards (Video 2). This was one of the central premises for his Veerkracht (Resilience) project in 2010.

In Veerkracht, the physical interface of the Silent Drum has been powerfully transformed both in terms of the scale of the material device and of bodily interaction. Rather than the fine motor articulation of hands and fingers, Van der Woude engages the larger movements of the whole human body. Furthermore, besides programming his own sound software, Van der Woude also ported the algorithm to openFrameworks (oF), a C++ framework for creative coding, and sent data to Pure Data via Open Sound Control (OSC) to control sound synthesis processes.

Daniel Gómez — “Yemas”

Finally, in Cali, Colombia, computer music researcher Daniel Gómez, currently a PhD candidate at Barcelona’s Universitat Pompeu Fabra, developed Yemas while at the University of the Colombian Institute of Advanced Studies (ICESI). Yemas adopts the Silent Drum’s basic principles, but transforms the interface and completely reformulates the tracking strategy using a 3D algorithm in OpenFrameworks, developed together with fellow ICESI researcher Jose Moncada, using a Kinect camera. For the first version of Yemas (Fig. 6), ICESI students Daniela Sanchez and Mauricio Ossa designed a physical interface of their own creation. Yemas has a larger diameter than the Silent Drum and provides three-dimensional parameters. Together with Rafael Vega and Carlos Arce-Lopera, Gómez developed new and interesting sound mappings for Yemas using its surface to navigate “timbre spaces” described using a variety of spectral features (2013).

A new version of Yemas discards the drum body altogether and simply stretches the fabric between two tables (Fig. 7), thus reducing the physical interface to its very essence.

Conclusion

Since the first theremin schematic, tutorials and code provide a mechanism for the replication of musical devices. More importantly, they serve as a point of departure for new designs. In the Internet era, the dissemination of tutorials and code have activated open source mechanisms throughout the world, enabling appropriation practices to act as cultural and artistic statements.

Computer-based music instruments share a dual nature as material objects and computer code. As such, in the process of appropriation and repurposing, many of the features of new media objects and code, such as variability and modularity, materialize into their physical interface and guide their transformation.

When one grounds and draws ideas, devices and code from open source culture, these shape our creative practice. The Silent Drum owes its existence to and is framed by software like Pure Data and GEM. Several of the works reviewed here, such as Yemas or Veerkracht, retain some of the basic principles of the Silent Drum, yet they feel like, and in fact are, distinct works and devices. In these new designs, the Silent Drum has been transformed to such degree and repurposed in so many unpredictable directions, that questions about the æsthetic implications of this exchange can begin to crystalize.

Although this set of devices point to different æsthetic directions, they share some basic sound-shaping properties. Are these shared properties audible? While the impulse to produce devices is an æsthetic one, what is the æsthetic consequence of sharing, appropriating and repurposing such material? As we move towards more fluid and participatory mechanisms of exchange, computer music instruments and, by default, music point to the consolidation of a global culture of appropriation and transformation that transcends the constructed boundaries of Western music and its instruments.

Bibliography

Danks, Mark. “Real-Time Image and Video Processing in GEM.” ICMC 1997. Proceedings of the 23rd International Computer Music Conference (Aristotle University, Thessaloniki, Greece, 1997).

Gómez, Daniel, Rafael Vega and Carlos Arce-Lopera. “Design of a Customizable Timbre Space Synthesizer.” CMMR 2013: “Sound, Music and Motion”. Proceedings of the 10th International Symposium on Computer Music Multidisciplinary Research (Marseille, France: 15–18 October 2013). http://www.cmmr2013.cnrs-mrs.fr

Manovich, Lev. The Language of New Media. Cambridge MA: The MIT Press, 2001.

Moog, Robert. “The Theremin.” Radio and Television News (January 1954), pp. 37–39.

Moore, F. Richard. “The Dysfunctions of MIDI.” Computer Music Journal 12/1 (Spring 1988), pp. 19–28.

Oliver, Jaime. “The MANO Controller: A Video-based hand tracking system.” ICMC 2010. Proceedings of the 36th International Computer Music Conference (New York: Stony Brook University, 1–5 June 2010).

_____. “Código Abierto | Obra Abierta: El Efecto de las Prácticas de Intercambio de Código en la Composición de Música.” In Canto Electroacústico: Aves Latinoamericanas en una Creación Colaborativa. Edited by Luis Germán Rodríguez Leal and Adina Izarra. Caracas, Venezuela: Centro Fundación Telefónica, 2012, pp. 53–64.

Oliver, Jaime and Matthew Jenkins. “The Silent Drum Controller: A New percussive gestural interface.” ICMC 2008: “Roots/Routes”. Proceedings of the 34th International Computer Music Conference (Belfast, Northern Ireland: SARC — Sonic Arts Research Centre, Queen’s University Belfast, 24–29 August 2008). Available on the author’s website at http://www.jaimeoliver.pe/pdf/oliver-silent.pdf

Pinch, Trevor J. and Frank Trocco. Analog Days: The invention and impact of the Moog synthesizer. Cambridge MA: Harvard University Press, 2009.

Puckette, Miller. “Pure Data: Another integrated computer music environment.” Proceedings of the Second Intercollege Computer Music Concerts (Tachikawa, Japan: Kunitachi College of Music, 1996), pp. 37–41.

Schultz, Ernest J. “A Simple Electronic Musical Instrument: The Theremin.” Radio and Television News (October 1949), pp. 66–67. Available online at http://www.theremin.info/-/viewpub/tid/10/pid/47

Wessel, David. “An Enactive Approach to Computer Music Performance.” Rencontres Musicales Pluridisciplinaires 2006: “Le Feedback dans la Creation Musicale” (Lyon, France: GRAME, 17–18 March 2006), pp. 93–98.

Wessel, David and Matthew Wright. “Problems and Prospects for Intimate Musical Control of Computers.” Computer Music Journal 26/3 (Fall 2002) “New Music Interfaces,” pp. 11–22.