General Information about the Computer Music Project

The Computer Music Project conducts research in the areas of real-time control, acoustic signal processing, music synthesis, music understanding, and performance. In all of these areas, our work is focused on issues of representation. This project, in existence since 1983, is part of the Carnegie Mellon University School of Computer Science.

Belinda, Lorin, Roger, Eli, around 1998 in the Music Lab

Real-time Control has been the subject of much of our work. The CMU MIDI Toolkit was developed here, and served as the platform for pioneering work in computer accompaniment and interactive music systems. Current work is on a system named Aura, which extends the toolkit to support a mix of time-critical music control processing, audio synthesis with low latency hard real-time deadlines, and ``soft'' processing of tasks such as interactive graphics or high-level AI related search tasks. The primary platform for this development work has been Windows-NT, but we are generally moving to Linux, which now offers better real-time performance.

Another real-time development is the Serpent language, which is modeled after Python, but includes a sub-millisecond latency real-time garbage collector and allows multiple instances to run in one address space, allowing true concurrency with preemptive scheduling. Serpent is the "scripting language" for Aura applications.

Work on Signal Processing and Music Synthesis is proceeding along several parallel paths. First, there is research in language design. The very high-level language Nyquist allows composers to compute and manipulate signals using a functional programming style. Nyquist integrates the representation of discrete events such as notes with continuous audio. A spinoff of the Nyquist work is a study of efficient synthesis by software, considering such things as cache effects, buffer sizes, and the organization of inner loops. As a result of this work, Nyquist was by far the fastest synthesis language in a comparison of several popular synthesis packages on standard benchmarks. More recently, we have started an effort to visualize and edit musical information, starting with an extensible audio and event editor, Audacity. Audacity accepts plug-ins, is integrated with Nyquist, and provides a simple way to construct new editing operations for signals and discrete events such as MIDI data. Audacity has taken on a life of its own as one of the most popular audio editors. Finally, we are investigating a synthesis technique developed at Carnegie Mellon called spectral interpolation synthesis, which models the natural spectral evolution of wind instruments. The synthesis algorithm is computationally efficient and lends itself to "synthesis by rule" of sounds for which there is no original model to copy. Our trumpet synthesis is unsurpassed.

Music Understanding is the task of identifying pattern and structure in music. For example, using a combination of techniques borrowed from the fields of signal processing, discrete algorithms, artificial intelligence, and real-time control, we have built systems that reliably ``listen'' to live performers by following written scores and synchronizing pre-determined accompaniments to real-time MIDI input. These systems are robust in that they can speed up or slow down with the performer as well as tolerate a large number of mistakes! The basic technology of the SmartMusic system by Coda Music Technology was developed here and is licensed by CMU to Coda. We have worked on many varients of the score-based performance following task:

Considerable effort has also been spent mapping improvised performance to higher level notions of structure. This is an entirely different problem because there is no score to follow. Assuming the computer is a semi-autonomous extension of the improviser's intent, it must remain tightly coupled to the performer's nuance, all the while providing robust, useful, and interesting additions to the performance. Success requires programs that react in non-linear yet consistent and interesting ways; progress in the music domain is therefore congruent with HCI and iteractive-media research. It also requires formalization and/or learning of meaning in improvisation. We are experimenting with AI techniques (rule based, symbolic, numeric) and real-time systems in the following applications:

Most recently, we have collaborated with researchers at the University of Michigan on a large project, MUSART, related to Music Search and Retrieval. While many researchers have looked at this as a melodic pattern matching and search problem, we are taking a much broader view. We believe that style is important for music perception and classification. We are investigating how to characterize style and use it for music search and retrieval. This raises problems of audio analysis, symbolic analysis, motive and phrase identification, machine learning, indexing, representation, and even music composition and synthesis.

An important outgrowth of this work is a set of new techniques for Music Structure Analysis, in which music signals (audio) is analyzed automatically to determine form and structure, e.g. AABA. In addition, we have developed techniques for aligning audio to symbolic data such as scores and MIDI data. This is essentially a "poor man's music transcription system": given a score and corresponding audio, we can map from symbols (beats, bars, rehearsal numbers, etc.) directly to the audio.

Finally, and perhaps most importantly, the Computer Music Project supports Music Synthesis and Performance, and much of our research is used by visiting composers. For example, the CMU MIDI Toolkit has been used to realize many compositions, Nyquist has been used to synthesize several works, and our recording facilities are used regularly for music production. An excellent integration of all of these efforts is seen in Roger Dannenberg's interactive computer music pieces. These compositions integrate real-time improvisation with intelligent computer graphics and musical accompaniment:

These pieces are truly interactive: the level of passivity and reactivity depends upon what the user is doing and what mode the piece is in; in many places, the program is able to synchronize video events with a particular playing style (eg. frenetic video for frantic playing). These compositions are not only interesting artistically, but also serve to identify future directions in our own research efforts.