From UPIC to IanniX

Development of a three-dimensional graphic, open source environment for music, sound and beyond

by Julian Scordato

Xenakis, CEMAMu and the UPIC

In 1966, Iannis Xenakis constituted informally the EMAMu (Équipe de Mathématique et Automatique Musicales), a group whose activities led to the formal establishment in 1972 of the Centre d’Etudes de Mathématique et Automatique Musicales (CEMAMu), an interdisciplinary research centre in Paris for the development of a new pedagogy focused on the UPIC (Unité Polyagogique Informatique du CEMAMu), an idiosyncratic system for computer music composition through drawing (Colyer 1986).

Indeed, the pedagogical issue in computer music was fundamental and urgent at a time when traditional theoretical tools — the Western music theory and notation developed over the course of many centuries — were becoming increasingly inadequate in responding to or representing the unprecedented growth of new practices and interests encountered in the new musical works of the late 1960s and early 1970s. Even today, in computer music notation and in electronic music in general, there is little to no agreement on the choice of a grammatical systematic priority that can be universally codified and incontestably represented by a symbol- or graphic-based notation. Whereas traditional notation can hardly be separated from its instrumental reference, and is the result of centuries-old historical and semiotic stratification, the notation of electronic music and sound raises many issues in the realm of semiographic invention. In this case, the multitude of texts that are produced is therefore also the result of an ex novo creation of linguistic signs which reveal a wide range of semiotic production possibilities (Valle 2002).

The development of UPIC as a compositional and pedagogical tool finds its roots in Xenakis’ concept of designing the musical construction through a generalized notation capable of expressing sound characteristics in mathematical terms, thus using graphical and numerical elements — that are essential for the dialogue with computers (Xenakis 1992). Already adopted starting in 1953 by Xenakis himself for instrumental music composition and translated into traditional notation for performers, in 1976 this notation system was implemented as a graphical interface in the UPIC and in January 1977 the UPIC was completed (IanniX Association 2012). As the users of the system could execute work entirely without performers or assistants, from sketches of creative ideas to their production and representation in electronic sound, UPIC offered the advantage of “allowing the composer to work in his individual way directly — no intervention or interposition of any person between his ideas and the result” (Colyer 1986, 317).

The UPIC system originally included an electromagnetic pen for drawing, a large graphic tablet that represented a page of music, and a computer with specific hardware for processing data and generating electronic sounds (Fig. 1). The peculiarity of this system consisted in its capability to perform graphic score editing as well as sound synthesis features with a real-time operation.

In 1990, a commodity hardware version of UPIC was presented at the International Computer Music Conference and then commercialized in the following year. The new UPIC system was based on an increased human-machine interaction and the integration of the score design features into the software. Practically speaking, the digital drawing tablet and electromagnetic pen were replaced by a computer screen and a mouse. Furthermore, through a redesigned interface, the user could store, edit, play and record sounds and related attributes defined with a poly-agogic notation on different levels, from macro- to micro-form: pages, events, envelopes and waveforms could be drawn on the screen by means of a two-dimensional representation of parameters as a function of time (Marino, Raczinski and Serra 1993).

The UPIC score is formed by a series of notes called “arcs”, that is to say curves defined in terms of frequency that develop in the time domain with continuous variations (glissando) and comprise a set of attributes, as mentioned above. The curves can be drawn either freehand or through the use of default drawing modes in order to obtain quantified values of frequency and duration (Fig. 2). “In this way the notion of arc is an extension of the Classical notion of note” (Ibid., 250). In addition, arcs can be grouped together as needed up to a maximum of 4000 elements per page, which represents the entire sequence of individual sound events.

As the interactive chronology of UPIC unfolds (Centre Iannis Xenakis 2015), various tools and devices participate actively in the same process of technological evolution and democratization that led UPIC to a prototype of software-only version in 2001 — the UPIX — after passing through a continuous update of its design.

After Xenakis’ death in 2001, the development of UPIX stopped (Bourotte 2012). At that time, Thierry Coduys (La Kitchen), with the collaboration of Adrien Levèfre (Adlef) and Gérard Pape (CCMIX), and supported by the French Ministry of Culture, started to develop IanniX. On the one hand, the new software would be based on the UPIC, but on the other hand it would have become an advanced interface devoted to a wider creative field by integrating current technological developments and a strong level of abstraction.

What is IanniX?

IanniX is a “graphical real-time open source sequencer for digital art” (IanniX website). Through various communication protocols, it synchronizes punctual events as well as continuous data to external environments (e.g., Pure data, Processing) and hardware such as MIDI devices and microcontroller boards.

Its graphical user interface shows a representation of a multidimensional and multi-format score that is programmable via GUI, JavaScript and other third-party applications that use a compatible network protocol (OSC, Raw UDP, TCP, HTTP). This flexible configuration has an advantage over most software environments in that it avoids forcing users to adopt a specific method for the creation of a score and the individual user can benefit from the use of multiple design strategies, according to their expertise, in an individual project. The interface is based on three types of abstract objects to be placed in a tridimensional space: triggers, curves and cursors. Triggers and curves represent isolated events and spatial trajectories, respectively. Cursors are time-based elements — playheads — that can move along the curves in order to read a specific sequence of space-limited events. A theoretically unlimited number of cursors can be added to the score by the user, in contrast to the single pre-determined timeline of its predecessor, UPIC. In this sense, IanniX proposes a tridimensional and poly-temporal sequencer, unlike the UPIC that was based on two-dimensional writing and allowed for a single timeline that could only be read from left to right — an emulation of the conventional direction of reading a Western music score. Also, IanniX runs independent of any audio synthesis engine. The characteristics of the design and interface therefore make IanniX quite suitable for a broad variety of applications.

Main Applications

Through the communication with audio environments or MIDI devices, IanniX can be used as a tool for the creation and performance of musical scores designed using two- or three-dimensional graphic notation. Many of the object attributes and the various mapping modes allow the user to associate the characteristics and the behaviour of cursors, curves and triggers with sound and music parameters, as well as several MIDI events. The ability to import external graphics into a IanniX project amplifies the representational possibilities of basic objects. Furthermore, sketches and notes can be integrated into the score as a background image or as textures, providing a tool to assist the user in the creation of the final work; two-dimensional images can also be included in the final version. With JavaScript, users can programme IanniX scores using a different approach; in a sense, scripts actually are the score (the content of the source file), as every IanniX command — which produces or is produced by an action in the graphical user interface — is defined through a custom JavaScript function (named “run”) and every IanniX reaction to external input from network protocols or interfaces is defined by a Javascript method named “onIncomingMessage”. Several examples of completed projects and practical suggestions are included in the IanniX software package, including the score of Récurrences (2011) by Thierry Coduys (Fig. 4) and an excerpt from Xenakis’ Metastaseis (1953–54). 1[1. See the Projects tab on the IanniX website for a gallery of images, videos and scores from projects that were made using IanniX.]

The strong relation between sound and visual content that emerges through the use of IanniX has often been a stimulus to reveal the score as an integral part of the work (IanniX Association 2016). In this sense, IanniX has been used in audiovisual works as a tool for controlling parameters and showing their graphical representation to the public, or even to facilitate the formal intelligibility of a work (Scordato 2014).

In addition to its use for scoring musical and sonic works, specific usages of IanniX include the control of sound spatialization, both for the definition of virtual sound trajectories (Manaris and Stoudenmier 2015) and the routing of audio signals in complex sound projection systems. Moreover, IanniX has been employed in sonification processes (Bellona 2013). 2[2. See, for example, the author’s interactive installation Studio per una Rete (Study for a Network), in which visitors can call up sounds and change the visualization of the Barcelona metro map via a graphical interface.]

Classification of IanniX Scores

In relation to their functionality, and to the interaction mode with third-party software and compatible devices, six types of scores have been recognized (Coduys and Jacquemin 2014):

• Control score. Autonomous, reproducible and determinist (similar to the operation of UPIC), once the control score has been set, the sequencer produces a data output for controlling further processes (e.g., sound synthesis, sampling and spatialization).
• Reactive score. Reacting to external stimuli without generating any output of control data, the primary purposes of the reactive score are visualization and graphical representation of data received from programming environments and devices (e.g., a tridimensional trajectory detected by a motion capture device, or the deformation of a curve expressed by a parametric equation).
• Generative score. Produced by algorithms written in JavaScript, the generative score can either cause the result to evolve over time or generate materials in a predetermined way; therefore, the script generates the score as output.
• Stochastic score. Using JavaScript and third-party libraries, the user can create a stochastic score in which the global process is predictable while single events are aleatoric; even in this instance, the script produces the output.
• Interactive score. Being based on human-machine or machine-machine interaction, the interactive score involves the cooperation between various entities. In this context, IanniX may act as a mapping device but also introduces compositional and temporal dimensions; a bi-directional data stream is involved.
• Retroactive score. IanniX can control itself, that is to say that the output related to an object can be reinserted into the system as an input command to control either the sequencer or another object; in some cases, this may imply recursion or deformation of the score as a function of time.

Graphic Objects

Except for importable graphics, IanniX scores are built by combining three types of abstract objects — curves, cursors and triggers — in a tridimensional representation system. Objects can be arranged in the score through the IanniX GUI, using JavaScript or via commands received from third-party applications. Of course, changes can be made in real-time during a performance.

In the definition of a score, every IanniX object has a series of general attributes that are customizable by the user: ID, group ID, activation status, thickness / size, label, colour, texture, 3D position (only for curves and triggers) and output messages (only for cursors and triggers). Furthermore, every object type holds specific properties, functions and attributes.

Curves

In IanniX vocabulary, a curve is a spatial trajectory defined by a set of points that establish a sequence of sections joined together. Each section can take three forms: a line segment, a cubic Bézier curve or an ellipse. Additionally, curves can be generated by mathematical equations (Fig. 5).

As vectors, IanniX curves have certain properties, such as being interpolable, scalable and resampleable. Operations on curves and their single points can be done from the Inspector menu or with direct manipulation in the visualization window, as well as through action-equivalent scripts.

Curves can assume different functions in a score: when they are linked to one or more cursors, they constitute their support by defining the cursor’s path; in other circumstances, such as in reactive scores, curves can simply represent a graphical artefact. Also, when curves eventually intersect a cursor linked to another curve, they can describe the evolution over time of values read by such cursor. Unlike cursors and triggers, in fact, curves do not send any message.

Cursors

Whereas IanniX acts as a global sequencer through its Transport panel, single cursors instead perform local sequencing functions and can be considered as “read heads” of curve values and triggers. For this reason, they represent the core of the poly-temporal feature.

Implications of poly-temporality involve specific space-time relations; for instance, they can be experienced in the score implementation of Xenakis’ Metastaseis (Fig. 6), which is included in the IanniX software examples.

By moving along curves according to time, cursors are able to output a set of continuous messages related to a tridimensional portion of the score included in their range, which is defined in terms of width and depth. For example, they can report their position over time as well as values related to colliding curves and triggers. An important aspect to consider when cursors send positioning values is the coordinate mapping (Fig. 7). Included in the Inspector menu, this feature allows the user to rescale the virtual space of the score to the desired range of values needed by the receiver for controlling a specific parameter (e.g., sound frequency or amplitude).

Cursor movement is subject to global transport controls. However, cursors have several specific attributes that define their own behaviour in the score: width, depth, speed / length, loop pattern, easing curve and offset; values are measured in seconds according to a space-time reference displayed as a grid. By default, a grid unit corresponds to one second in time sequencing, but this is of course user-configurable.

Triggers

In their default appearance, triggers are spherical elements with the capability to send individual messages when they are “hit” by a cursor (Fig. 8). In a sense, they can be compared to musical notes, as they involve discrete events in time.

For example, triggers can produce MIDI note messages in order to play a sequence of sounds on an external MIDI device. In this case, the trigger-specific duration [s] attribute should be set for each note according to needs.

In addition, the fact that triggers are placed in a tridimensional spatial system inherently encourages the user to explore unconventional and whimsical ways of conceiving a score (Fig. 9).

However, triggers are able to control any sort of event, from operations on data flow to the final media, depending on the software or hardware device linked to IanniX, of course.

Conclusion

As a kind of “higher-order UPIC in hyperspace” (Hoffmann 2011, 5) with a considerable technical advantage of operational openness over a monolithic design, IanniX offers a range of interesting applications that extend across a wide range of digital practices. In fact, its operation is based on the reception of commands for the creation and management of score data, as well as on the simultaneous transmission of score-related messages for controlling a third-party device. Therefore, it implements various communication protocols and technologies (OSC, raw UDP, TCP, WebSocket, HTTP, MIDI, Syphon and serial port communication) in order to support the interfacing with numerous software and hardware.

Even if its current graphical user interface contains symbolic limitations that do not make it as flexible as a graphic design application as some users might desire, in the context of experimental notation and sequencing programmes, IanniX is capable of providing new ways of performing graphic information (Gottfried 2015) and drawing sound events (Bourotte 2012) with an essential notation system suitable for the definition of discrete and continuous values.

With its openness and flexibility, and a strong level of abstraction, the system is able to manage digital information bidirectionally in extremely diverse applications. The use of IanniX is not limited to sound and music but is also applicable in design and mixed technology contexts, and beyond. Indeed, as-yet unexplored applications include the design of human-computer interfaces, the communication with microcontroller boards and the visualization of digital data.

Bibliography

Bellona, Jon. “Sonification Study of San Giovanni Elemosinario.” Availabel on the author’s website at http://jpbellona.com/public/writing/BellonaJon_SanGiovanni.pdf [Last accessed 25 December 2016]

Bourotte, Rodolphe. “The UPIC and Its Descendants: Drawing sound 2012.” Proceedings of the International Symposium “Xenakis. The electroacoustic music” (Paris, France: Paris 8 University, 23–25 May 2012). http://www.cdmc.asso.fr/en/actualites/saison-cdmc/xenakis-electroacoustic-music

Centre Iannis Xenakis. “L’UPIC de Iannis Xenakis: Chronologie interactive.” 2015. http://www.centre-iannis-xenakis.org/exhibits/show/expo-upic/timeline [Last accessed 25 December 2016]

Coduys, Thierry and Guillaume Jacquemin. “Partitions retroactives avec IanniX.” JIM 2014. Proceedings of the 21st Journées informatiques musicales (Bourges, France: École Nationale Supérieure d’Art de Bourges, 21–23 May 2014). http://jim.afim-asso.org/jim2014

Colyer, Cornelia. “Studio Report: Centre d’Études de Mathématique et Automatique Musicales.” ICMC 1986. Proceedings of the 12th International Computer Music Conference (Den Haag, Netherlands: Royal Conservatory of Music, 20–24 October 1986).

Gottfried, Rama. “SVG to OSC Transcoding: Towards a platform for notational praxis and electronic performance.” TENOR 2015. Proceedings of the 1st International Conference on Technologies for Music Notation and Representation (Paris, France: Université Paris-Sorbonne and IRCAM, 28–30 May 2015), pp. . http://tenor2015.tenor-conference.org

Hoffmann, Peter. “Xenakis Alive! Explorations and extensions of Xenakis’ electroacoustic thought by selected artists.” Proceedings of the Xenakis International Symposium (London, UK: Southbank Centre, 1–3 April 2011). http://www.gold.ac.uk/cmru/xenakis-international-symposium

IanniX Association. “Showcase | IanniX.” http://www.iannix.org/en/projects

_____. “From UPIC to IanniX.” Vimeo video (7:20) uploaded by “IanniX” on 3 July 2012. http://vimeo.com/45161598 [Last accessed 25 December 2016]

Manaris, Bill and Seth Stoudenmier. “Specter: Combining music information retrieval with sound spatialization.” ISMIR 2015. Proceedings of the 16th International Society for Music Information Retrieval Conference (Malaga, Spain: University of Málaga, 26–30 October 2015). http://ismir2015.uma.es

Marino, Gerard, Jean-Michel Raczinski and Marie-Hélène Serra. “The New UPIC System.” ICMC 1990. Proceedings of the 16th International Computer Music Conference (Glasgow, Scotland: University of Glasgow, 10–15 September 1990), pp. 249–252.

_____. “The UPIC System: Origins and innovations.” Perspectives of New Music 31/1 (Winter 1993), pp. 258–69.

Scordato, Julian. “Vision II: An audiovisual performance with IanniX.” Living Lab Music 5 + di_stanze. Proceedings of the 5th Living Lab Music conference (Padua, Italy: Conservatorio di Musica di Padova “ Cesare Pollini”, 17–18 December 2014). http://www.living-lab-music.org | https://issuu.com/sampl-lab/docs/living_lab_music_5

Valle, Andrea. La notazione musicale contemporanea: Aspetti semiotici ed estetici. Turin, Italy: EDT, 2002.

Xenakis, Iannis. Formalized Music: Thought and Mathematics in Composition. Second edition. Stuyvesant NY, USA: Pendragon Press, 1992.