The Static Organ

Biofeedback as new interactions for new action potentials

by Kiel Stuart Long

The Static Organ (2010–ongoing) project is a collection of works that build upon the brain activity of the user, represented through a variety of media, sound and new forms of interaction. It applies biofeedback technology in order to display information based on the user’s cognitive state that guides them through a self-reflective experience through the measuring of subtle internal bodily functions (specifically the activity of the brain). This is achieved by using electronic devices placed on a participant’s head to measure the electrical activity of the brain that without technology would be undetectable. These Electroencephalography (commonly referred to as EEG) readings are then used to control either parameters in the production of sonic and digital media, or to control sonic and kinetic installations.

Although these works are concerned with the current advancements in human-computer interaction (HCI), the pivotal core is in the user’s experience, exploration and interaction with the environment and interfaces in which they engage with.

The project consists of two major works (Fig. 1):

1. Mind Drops (2010), consisting of three dynamic audio-visual projections that surround the viewer as an immersive experience attempting to represent the activity of a person through a new perspective by audio-visualising their mental activity through three differing acts.
2. Mind Pool (2010–ongoing), an interactive installation that seeks to create a unique personal experience for the participant through the feedback of their own brain activity via audio and a reactive liquid “pool”.

Whilst generating these experiences these works aim to provoke and enhance an understanding of the functioning of the brain in relationship to the cognitive state of the user and their interactions with the environment that surrounds them.

Interface and Affordance

The initial undertaking of The Static Organ arose from the aspiration to create a new musical interface that was open to a wide range of users regardless of previous musical experience or ability. The need for progression from the “traditional” forms of musical interface has been discussed in the past. John Cage for example noted how “Most inventors of electrical musical instruments have attempted to imitate eighteenth- and nineteenth-century instruments…” (Cox and Warner, 2004, p26).

This statement on musical interface design indicates a preserving of familiar/traditional interfaces with neglect to the need of progress. To further the ways in which musicians compose and perform answers may exist within novel interfaces/interactions that are distant from pre-existing methods, an argument that can be explained through the notion of “affordance” (Norman 1988).

By designing electronic musical interfaces around pre-existing forms (such as pianos for example), a “perceived affordance” is given to the interface. Affordance is a meaning or use presented through attributes (size, shape, location, etc.) of an object or environment as “action possibilities” (a potential of how the object may be used), which are reliant on qualities of the user such as physical abilities, goals, beliefs and past experiences.

This can be understood if looking at a piano and a user with the goal of generating sound, for example. The keys afford being pressed to achieve a goal of generating a sound, as the user is physically able to make the action and from past experience is aware this is needed to achieve the goal. Norman proposes that although objects may hold several action possibilities (theoretically infinite), they have a “suggested affordance” that may be linked to users’ past experiences. This suggested affordance might be dominant to the extent that we fail to reveal the additional action possibilities. Should, for example, the keys of a piano be removed or altered in such a way that they no longer hold this affordance, a new interaction must carried out to achieve the goal of generating sound, revealing new action possibilities.

By removing the suggested affordances held by traditional interfaces, such as keys to push or surfaces to strike, it is hoped that the intention to compose and perform in a traditional manner will also be removed. In creating an interface and interaction that does not hold experience or culturally suggested affordances (such as biofeedback technologies), the user is encouraged to experiment in their interaction with the interface (to learn the action possibilities), affording new forms of creativity that might not be uncovered with interfaces replicating those of traditional musical instruments. This will also produce a requirement of learning that might be considered more demanding than that of traditional interfaces, as pre-existing knowledge gained from observation and interaction with similar interfaces will not be applicable, requiring a new understanding of interaction process from the user.

EEG and the Brain: A Light Introduction

The brain is a structure of approximately 100 billion nerve cells (called neurons) that are in a complex network of connections. In order to communicate with each other (that might be seen as sending information to each other) the neurons use changes in action potentials (electric charge). Although the action potentials emitted by single neurons are too small to be measured by EEG, many neurons synchronized in communication, known as neural oscillations or brainwaves (a rhythmic “firing” of a group of neurons), can produce a large enough field to be measured on the scalp as voltage changes, and the rate of this oscillation can then be defined as a frequency (in Hz). Typically a high brainwave frequency correlates to high neural activity and can be seen when engaging with a complex task (such as complex thought) and a low brainwave frequency correlates to a low neural activity (such as deep sleep).

The Static Organ project uses the ability to “see” (by measuring its frequency) the activity of the brain, produced by the mental processes of the user, to control interactions.

Mind Drops: Feeding, Sleeping and Meditating (2010)

Introduction to the Installation

The installation, entitled Mind Drops, consists of three 30-minute audio-visual projections. The videos are generated from pre-recorded data (through EEG sensors) of brain activity gathered from a subject while eating, sleeping and meditating. Whereas these actions activate a broad variety of mental qualities, Mind Drops attempts to display them with a minimalist æsthetic in order to facilitate a comparison of subtle yet important transformations over time.

The following data collections were used for the audio-visual projections:

• Feeding (Fig. 3, left) was recorded while the artist was eating. Eating is a necessary act to replenish the body and can be seen as maintaining the body’s physical existence within the world.
• The data for Sleeping (Fig. 3, centre) was recorded as the artist began to fall asleep. The sleeping state allows the brain to recuperate and continue to function normally. While sleeping, a dream state may also be entered, where a new reality is created for the dreamer that can be seen as a challenge to the dreamer’s perception of reality.
• The data for Meditating (Fig. 3, right) was recorded during a half-hour meditation. Meditation can be considered to be an act of replenishing the mind or spirit. During the meditation the artist focused on the thought and awareness of reality and existence of the self.

The settled æsthetic and the slow changes of the work anticipate another dimension to our reality that might exist beyond the scale of human observation, but might be made accessible through such æsthetic experiences and experiments.

Information Visualisation

A drop will only be placed, however, if there is sufficient activity within the frequency to give dominance of presence to the more active frequencies (through measuring the amplitude of the brainwave frequencies).

This form of information visualisation and sonification provides three layers of information:

• Sonic representation;
• Momentary activity, through the initial drop placement;
• Activity over a duration of time, indicated by lighter areas showing activity within a frequency range over a period of time.

Mind Pool (2010–ongoing)

Introduction to the Installation

The interactive installation piece Mind Pool is a feedback of the activity of the brain presented to the user in real time. This representation of activity is presented to the user through a liquid that responds in a varying motion of interacting wave patterns. This generates a unique symbiotic sculpture constantly flowing and changing as thought itself. This interaction provides an experience that is an externalization of the functions of the brain, allowing the user to engage as both a performer and observer, exploring the relationship between their mind and body in a dynamic physical and sonic environment.

In addition to audible feedback of the user’s brainwave frequencies, the installation uses a novel liquid feedback system. This is produced through the use of a liquid (ferro-fluid) that when free from magnetic fields has the properties of a liquid and when in the presence of a magnetic field is attracted towards the field and increases viscosity. Due to these properties ferro-fluid can be manipulated through control of magnetic fields, and the feedback system exploits this by inducing then removing magnetic fields to create waves (through the use of electromagnets placed under the pool’s container). The feedback system uses twelve electromagnets positioned in a circle (Fig. 6): the six electromagnets on the left half of the feedback system respond to EEG measurements taken from the left brain hemisphere and the opposite six (on the right half) respond to the right brain hemisphere. Additionally, the magnets closest to the user respond to the lower brainwave frequencies (e.g. 0–8 Hz) and the magnets furthest from the user respond to the higher brainwave frequencies (e.g. 40–48 Hz) (with a division of the frequencies assigned to the magnets between).

The electromagnets are turned on and off to create waves in cycles (time between each wave) that are proportional to the brainwave frequency. This produces a slow repetition of waves for low brainwave frequencies and a faster repetition of waves for higher frequencies. The waves produced develop an interference pattern towards the centre of the pool that can also be read as an overview of the feedback, being more turbulent with higher activity and calmer with lesser brain activity. Audio is also triggered for each wave produced that also changes in pitch in relation to the brainwave frequency.

Aesthetic Affective Feedback System

The work Mind Pool is an interactive installation applying affective computing. Affective computing is the study and development of interactive systems that are influenced by the emotional states of a user (in interaction) and respond in an appropriate manner, giving the system empathy to the emotional state of the user (Picard 2000). Mind Pool explores how applications of affective computing can allow users to reflect more deeply on their interactions with technology, focusing on the pleasure (or displeasure) that arises from interactions without a predetermined goal. Mind Pool draws upon work within interaction design and human-computer interaction that conceptualizes the user of technology in terms of their felt experience and the æsthetics of interaction (Wright and McCarthy 2004). Because of this shift in understanding, the user is required to look beyond functional, quantitative information-feedback models of interacting with technology and instead explore human experience in more detail.

It is understood that interactive experiences focusing on functionality and quantitative information feedback only activate limited capacities of the experience of the user. Subsequently, attempts to develop interactive systems based upon this paradigm often focus on usability (efficiency, user satisfaction, etc.) and performance of the system (information feedback), rather than the subjective experience (how the experience “felt”) of the user (Dillon 2001). Particularly relevant to this is emerging research within the area of “experience-centred design” (McCarthy and Wright 2010). Wright and McCarthy take that whilst development of the functional attributes of systems is still highly relevant, these should be supported by an understanding of the emotional values that people construct through interactions with technology.

These views highlight how previous interaction and biofeedback design is limited in that it focusses on making interactions between people and technology “information efficient” rather than attending to the subtle and ongoing changes in emotions people might feel. Building upon this, Mind Pool applies an æsthetically focused feedback over a more defined quantitative feedback.

A pool of liquid (Fig. 7) is manipulated to create waves that correspond to the user’s mental activity. When the user exhibits high brainwave frequency (indicating intense mental activity) the pool becomes turbulent (with corresponding high pitched audio) and, if the user has a low brainwave activity (indicating calmer or lower mental activity) the pool will produce a calmer wave pattern (with lower pitched corresponding audio). Although this system permits a less detailed feedback of information, it encourages engagement and interpretation through interaction with æsthetics over delivery of quantitative information. This provides the user with the means to make a qualitative interpretation over a quantified measurement of their subjective experience.

Mind Pool and Flow

To provide an experience that is immersive and pleasurable, Mind Pool was developed to exploit the theory of “Flow” (Csikszentmihalyi, 1992). The theory of Flow exposes the necessary factors required to make a task or role immersive and pleasurable (a state of “optimal experience”). When observing artists creating works, Csikszentmihalyi found that there was intense investment into the creation process but not the finished artifact; the pleasure gained was from the immersion into the task. This optimal experience was observed to be so powerful that it could overcome the desire for nourishment and sleep, with some only leaving the state through exhaustion.

For Flow to occur, an activity must facilitate the sensation of some or all of the following properties:

1. Goals and a balance between ability and skill requirements.

The goals within an activity should provide a challenge and require a high degree of skill that is comparative to the user ability and therefore remain achievable. By not presenting a specific challenge or goal to achieve (such as achieve a brainwave frequency of x Hz) Mind Pool allows the user to decide his or her own goals and skill requirements. This may, for example, be to establish control of the installation (such as increasing the pitch of the audio or the turbulence of pool at will), however, as the user selects a particular challenge it may be adjusted to suit their own perceived ability.

2. Concentration.

A high level of concentration on a minimal number of tasks, allowing the user to focus attention on a particular goal, is required to allow for immersion. Because the user is provided with a challenge of control that is directly linked to mental activity, a high degree of concentration is required in order to control the installation.

3. The loss of self-consciousness, the merging of action and awareness, and the focus of awareness solely on the activity.

The process of learning the control of the installation involves experimenting with action and becoming aware of that action’s effect on the feedback system (audio and wave patterns). The user will first learn what effect varying states of cognition, body and environment have on the reponse of the system. The user will then link the effect produced to that process, and a correlation between action and awareness will occur. The user will eventually merge the intention with the outcome (action with the awareness).

This act of merging action and awareness can be related to the Heideggerian concepts of “ready-to-hand” and “present-at-hand” (Heidegger 1962) that describe the use of tools (interaction with interfaces) in terms of experience. When a tool (or interface) is said to be present-at-hand, the user is aware of the tool/interface as a distinct form, for example a pen on a desk. However, when the pen/tool (interface) is in use by the user, its distinct qualities (and separation as an individual object) become transparent to the experience of the user, or ready-to-hand. The pen may be seen as a continuation of the user’s body and used as a merging of action and awareness. If the pen should break (and not function as intended), its distinct qualities will become present and it may again be viewed as present-at-hand (there is no longer the facility for the merging of action and awareness).

In order for Flow to exist, the interaction process must become seamless and transparent to the user, or ready-to-hand. To successfully control the installation, a state of ready-to-hand must be achieved between the user and installation. As the brain rapidly responds to environmental stimulus, control of the installation requires a high degree of focus to prevent these environmental stimuli from affecting the user’s control. This activity of focusing awareness also aids in producing a high degree of immersion.

4. Distorted sense of time.

During immersive activities, a loss of the subjective experience of time occurs. This is not to say that time (actual time) is distorted, however the experience of time (real time) for the user may not exist parallel to it (for example, the user experiences a short amount of time while the actual time passed is greater).

5. Direct and immediate feedback.

For the continuation of Flow, direct and immediate feedback is required to allow the user to immediately adjust to the successes and failures of actions. The feedback system (the liquid pool and audio) is presented to the user in an immediate and direct way, thereby facilitating the ability to merge action and awareness, as well as adjustment to success and failures in trying to achieve goals.

6. Personal control of the activity.

As the installation is connected to the user directly (via the EEG sensors) a high degree of personal control of the activity is given to the user. This personal control may be in the ability to set one’s own goals, decide on the level of interaction or the duration of engagement.

Continuation of Flow

For Flow to continue, a task or interaction must be reactive to the user’s ability.

The user’s challenges are indicated on the vertical axis and the user’s skills are indicated along the horizontal axis. Should a task have a high level of challenge and the user a low level of skill, the user will not be in Flow and will instead experience anxiety. Alternatively, should a user’s skills far exceed the challenges presented in an interaction, the user will experience boredom and again fail to achieve Flow. To facilitate Flow there must be an equal balance between perceived challenges and user skill throughout the interaction process (Fig. 8).

Mind Pool allows the continuation of Flow experience by keeping the interaction at a level that is neither too hard nor too easy by allowing the user to continuously choose a level of challenge (e.g. maintain a low brainwave frequency for a prolonged period of time) that is within their skill ability. As the user can choose his or her own challenge level, it is hoped that once entered into a state of Flow, the desire to maintain this state will encourage the user to find and set new challenges.

Conclusions

It can be seen that biofeedback interfaces facilitate new forms of creative expression in the arts and encourage the potentials of producing highly immersive and pleasurable states through interaction. Unlocking new action potentials in music and new media art, these new interactions allow for novel modes of composition and expression that offer unexplored alternatives to more traditional forms. Through understanding these interactions with technology in terms of felt user experience over usability, richness can be added to the communication that might otherwise be lost in these uniquely personal experiences. Although biofeedback will almost certainly provide much advancement to healthcare in the future (such as enhancing control of prosthetics for example) its application to the arts should also be further encouraged.

Bibliography

Cox, Christoph and Daniel Warner, eds. Audio Culture: Readings in modern music. New York: Continuum, 2004.

Csikszentmihlyi, Mihaly. Flow: The Psychology of Optimal Experience. London: Harper Collins, 1992.

Dillon, Andrew. “Beyond Usability: Process, Outcome and Affect in human computer interactions.” Lazerow Lecture 2001, Faculty of Information Studies, University of Toronto, 2001.

Heidegger, Martin. Being and Time. 1st ed. London: SCM Press, 1962.

McCarthy, John and Peter Wright. Technology as Experience. Cambridge MA: MIT Press, 2004.

Norman, Donald A. The Psychology of Everyday Things. Basic Books, 1988.

Picard, Rosalind W. Affective Computing. Cambridge MA: MIT Press, 2000.