Music and Engineering Find Common Ground in CIEMAS' soundSense

Scott Lindroth, others create a space where movement becomes music

Effortlessly moving around, visitors in the studio simultaneously play and enjoy unusual, unrepeatable concerts. Infrared sensors track every movement in the room, their readings magically becoming the sounds of cellos, chimes, Tibetan prayer bowls and an ethereal chorus. This "sonification" of sensor data not only floods the studio with intriguing musical textures, but also give visitors an opportunity to hear patterns and recognize the activities that cause them.

These are the goals for soundSense, a special room in CIEMAS developed by the Fitzpatrick Center for Photonics and Communications, the Department of Music and the Information Science + Information Studies program (ISIS).

"soundSense transforms data into music but also communicates higher-order information," said composer Scott Lindroth, chair of the music department and a founding board member of ISIS. "For example, you can hear a crowd forming."

Lindroth said the numerical sensor readings for different areas of the room travel by wireless network to the control room. Powerful computers transfigure the numbers into sound with a program co-authored by Lindroth and John Bower, a musician and research associate, in a sound-synthesis language called SuperCollider.

Although soundSense is easy to experience, it was difficult to create, said Lindroth. A week before the premiere, Lindroth and Bower were perched on the edge of armchairs in the CIEMAS lobby, squinting at the screens of gleaming white notebook computers.

"This is what computer music looks like," Lindroth said. "No staffs or notes, just cryptic operators, brackets, slashes and the like. It isn't easy to see what's wrong, why we're not hearing the harmonies we programmed."

"The program should be cycling through variations on a harmonic pattern, with richer textures as the activity level in the studio increases," Bower said.

Lindroth and Bower discussed possible causes -- a missing curly bracket in the program, flaws in SuperCollider's sequencing of subprograms, poor sensor data. They decided on a possible fix, typed changes and launched the program again.

"I'm still not hearing harmony," Lindroth said. "Maybe we're not getting enough sensor data."

Lindroth hurried across the CIEMAS lobby to the studio. Steve Feller, John Burchett and other engineers from The Fitzpatrick Center were climbing a scaffold, installing sensors modified to work on the high ceiling.

Only one row of sensors was operational. Lindroth walked back and forth along the row, arms flapping. "Beethoven never had to do this," he said.

Back in the lobby, Lindroth learned the new data failed to produce the desired harmonies. The cycle of trial-and-error diagnostics began again.

On Tuesday, just two days before the CIEMAS opening, a kiosk with an additional sensor hung from the ceiling. Forty LCD monitors were on the walls and four on the kiosk. The movements of engineers generated a jumbled stream of beautiful sounds from speakers on all four walls.

"You're hearing chaos because we've turned off the thresholds between activity levels," Bower said.

The sound changes according to different levels of numbers of people and amounts of activity in the studio.  What they want is a progression of pleasant sounds based on varying activity levels.  Two people standing still in the room should bring relative quiet.  A large group of people being active should bring rich, multi-part harmonies. 

Choosing the thresholds -- the point at which new, richer harmonies kick in -- was tricky because activity levels at the opening were unknowable, said Lindroth.  "That's one of our tasks today. We've come a long way since last week but still have a ways to go."

Other last-minute preparations included readying the poetry of Joseph Donahue, senior lecturing fellow in English, for display on the monitors. Research scientist Rachael Brady led a team creating demonstrations of the many ways in which the same data can be represented in images rather than sounds. ISIS administrative director Casey Alt guided preparations of explanatory materials for a plasma display and posters.

"soundSense embodies what ISIS does - merging science and technology with the arts and explaining the significance and value of the interactions," said Cathy Davidson, vice provost for interdisciplinary studies and an ISIS board member.