Current Art & Science
My interest in Art & Science explorations is motivated by a need to bring together two complementary ways to interact with reality that have always been similar in my head. In science and in art, I am developing virtual worlds that must be internally consistent and built by leaps of creativity cemented together through meticulous work. However, whereas in science these worlds must contain a linear succession of logical steps, internal coherence, axioms and consequences that mimic the physical world, an art piece can have its own unique internal coherence that follows no linear logic and does not have to parallel the physical world.
In the last few years, I started connecting my work in music and physics into a third approach that combines both perspectives. As my science became increasingly focused on complex systems and my art in electronic music, common themes started to emerge that suggested projects at the interface of both perspectives. The two are naturally related, since complexity research addresses questions beyond traditional physics and electronic music is strongly influenced by science and technology. The resulting combination has been extremely fruitful. It provides me with complementary insights. It has produced projects that are uniquely engaging and generate strong public interest. It allows me to work simultaneously with two communities where I feel equally at home but that rarely interact closely with each other. It is timely, since our increasingly technological world provides new intuitive widespread ways to interface with music, but seems to result in more and more obscurantist views of its underlying science.
I describe below some of the themes and projects that I am developing at the art/science interface.
1. Music as an emergent phenomenon
This is the theme that started bringing together my artistic and scientific perspectives. After giving a series of public lectures on the physics of music (such as my talks at the Chicago Cultural Center shown on these videos: part 1, part 2, part3), I organized with Prof. Thilo Gross a workshop at the ZiF Center for Interdisciplinary Research in Bielefeld, Germany, entitled: Networks and Nonlinearity in the Musical Experience. In this workshop, we brought together electronic musicians, complexity scientists, and entrepreneurs involved in new music technologies to discuss common perspectives (workshop program). A novel connection between music and science soon emerged, which I have presented in several talks (e.g. this plenary talk on networks, complexity and emergence in music at NecSys 2013 in Koblenz, Germany) and in publications (e.g. this chapter on music and flocking dynamics in the book Controls and Art – Inquiries at the Intersection of the Subjective and the Objective).
Up to now, music and science have been related through what I call the physics of acoustics, the mathematics of harmony and rhythm, and the biology of hearing, but a new connection is developing between music and the science of complex systems. In this context, music can be viewed as an emergent phenomenon, i.e. one where complex interactions between parts produce a collective effect that is more than the sum of the individual components, which can often be characterized by a small number of parameters described in physics as Order Parameters.
Music is indeed an emergent phenomenon in at least three ways: 1) at the neurological level, an extremely complex combination of firing neurons produces distinct emergent effects in the organism, such as the physiological effects related to a change in mood or to an urge to dance; 2) at the social level, the definition of new musical styles and their appreciation criteria result from a complex network of social interactions between listeners; and 3) at the composition level, where complexity is engineered by combining multiple melodies, harmonies, textures, and rhythms that are not meant to be heard in isolation but instead must achieve a desired emergent effect in the listener. In all these contexts, new technologies (from brain scans to music recommendation engines to the algorithms behind electronic music tracks) are increasingly showing the connection between music and complex systems.
As an initial project connecting music and complex systems, we implemented with Prof. Rodrigo Cadiz software that maps in different ways flocking dynamics to sound. Its main panel allows us to present in real time a flocking simulation and the corresponding generated sounds using different algorithms. This software was later used by Cadiz in a presentation with the Man and Machine Robot Orchestra in Gent, Belgium.
2. Binary music and criticality
Another natural connection between music and complex systems is the link that we are exploring between music and criticality with Profs. Gross in the UK and Cadiz in Chile. As explained in the Current Research section, I am interested in understanding the link between modular-hierarchical evolution and the apparently critical dynamics observed in many living systems, which seems to extend to art.
Life requires a combination of adaptability and robustness that would seem to place it in a critical regime, at the interface between dynamics that are too stable (too rigid to rapidly adapt to new environmental conditions) and too chaotic (too sensitive to perturbations that could disrupt essential biological functions). Similarly, music should contain a combination of elements of order and disorder, coherence and decoherence, and structure and variation to make it appealing, which also places it in some sort of critical regime. This combinations can appear in different ways and at various levels; in the interplay between expected and surprising note sequences found in engaging melodies, in the superposition of perfectly and imperfectly coinciding harmonics required for a rich chord, or in the mixture of regular and syncopated rhythms that build a driving percussion layer, to name a few.
We are interested in exploring the hypothesis that the same type of signatures of criticality studied in living systems can be found in music. The reason that music is so universally appealing to humans may well be that it is in some fundamental way attuned to our biology, mimicking its dynamics. To develop this project, we needed ways to analyze music mathematically, which is an open and challenging question; although sound is nothing more than a one dimensional function of time, the information we receive from it spans multiple timescales (of timbre, pitch, rhythm, and structure), but we have no mathematical description to capture them all. Our solution was to approach music to mathematics rather than vice versa: we thus created a new musical style that we call binary music.
The binary music style is restricted to use only identical percussive click-like sounds distributed over time with limited maximal temporal resolution. By considering only a series of zeroes and ones that represent on or off instances of this sound, we avoid the unresolved issue of identifying the information content of the multiple factors and scales involved in standard music. Interestingly, the use of atonal clicks also provides a perspective that is orthogonal to most previous studies of the mathematics of music, which focused mainly on harmony.
I have thus developed a research and creative project that analyzes binary pieces that I have composed (such as my binary tracks Opus01, Opus10, and Opus11) to generate and compare with others produced algorithmically. We wrote a computer program that generates binary pieces using dynamical systems with critical dynamics, which are not yet as appealing as the composed ones but have already shown the potential of this approach. This process has allowed me to better understand how I compose and to translate this knowledge to new versions of the algorithm. The artistic challenge of composing engaging binary pieces despite their minimal elements is fascinating and I am starting to share it with other musicians. The possibility of unambiguously translating binary pieces to mathematical language opens multiple potential research avenues, which include using brain scans to measure the biological response to binary pieces with known mathematical properties and using crowdsourced online experiments to explore if listeners prefer indeed music close to criticality.
3. Interactive performances
In a series of visits to Berlin, I started developing a project with the label minus, headed by renowned electronic musician Richie Hawtin (also known as Plastikman). This label has implemented multimedia shows that interact with audiences in through their smartphones. By creating a local area network and asking fans to download a specifically designed App, they can engage concert goers in real-time. The audience can view on their smartphones an onstage camera feed, receive information on the electronic instruments’ activity, display text messages on the giant screens surrounding the stage, play music samples when the performer allows it, and display shapes and colors that complement the current light show.
While the smartphone-based interactions developed by minus are fascinating, they are all between each individual concertgoer and the stage, rather than engaging the crowd as a whole. We are developing a collaboration that will go beyond binary interactions and aims to extract the collective crowd dynamics by processing accelerometer information from all connected smartphones (note that these are dance shows where audiences react to the music by moving) and computing order parameters that can summarize the crowd activity, such as mean motion amplitude, synchronicity, or others.
We plan to first have a programmer develop an App that piggy-backs on the existing minus application, gathering data from crowd motion in some of the massive festivals and shows where Plastikman performs throughout the year. This information will then be correlated with show recordings to extract order parameters that adequately describe the crowd dynamics, an interesting and unexplored scientific question in itself. The resulting parameters will then be computed in real time in future shows, leading to a number of potential applications. One could display them on a vu-meter for DJ’s to help them connect with the crowd (one of the main goals of an electronic music DJ). One could use them as inputs for the algorithmic art that is displayed on giant onstage screens, thus opening the fascinating possibility of creating a massive show that is in itself a “control loop” where the visuals are coupled to the crowd response through an algorithm that attempts to optimize the show experience. Going beyond the artistic realm, one could also search for subtle signatures of the crowd dynamics that may predict safety hazards such as stampedes.