Cyborg Drummer and AI Team Create Music That’s Not Humanly Possible

Designing a Robotic Prosthetic for a Professional Drummer

Advancements in technology, from artificial intelligence to 3D printing, have revolutionized prosthetic design. State-of-the-art prosthetics enable their wearers to complete tasks most of us take for granted. And since the goal of prosthetics is to enable their wearers to complete tasks important to their daily lives, it is common to find prosthetics customized for various activities, from running to cycling, and even for dancing.

Prosthetics design requires skills from various disciplines, such as biomedical engineering, robotics, mechanical engineering, computer science—and music. Yes, music. Runners need to run, and drummers need to drum.

Creating a custom robotic prosthetic for drumming proved to be the perfect project for a team from the Georgia Tech Center for Music Technology (GTCMT). And while the advancements in prosthetic design often include muscle- or mind-controlled activation, this project was unique in that part of the design was autonomous: The wearer does not completely control the actions of the prosthetic.

A Drummer’s Drive to Return to Drumming

Jason Barnes is a drummer. He’s played in reggae and progressive metal bands and lists drummers the likes of Matt Garstka, Luke Holland, and Neil Peart among his musical influencers.

"I first got into music when I was 12. My Dad started teaching me to play the guitar and I never really picked it up that well. Then I got a drum kit for Christmas when I was 15. Ever since then, I've been a drummer.”

At 22, he lost his right arm in a workplace accident. The accident had happened just days before he was scheduled to audition for the Atlanta Institute of Music and Media (AIMM). The aspiring drummer was determined to return to his music. Shortly after being released from the hospital, he created his own basic drumming prosthetic. It was an important step in his emotional recovery.

“It was only a couple of weeks after the accident, and I still had bandages on my residual limb,” he says, “so I taped a drumstick to my bandages and proceeded to play. That was the turning point in my life that let me know it was still possible.”

He developed his own spring-loaded drumming prosthetic, which he used when he reapplied to the school. He was accepted to AIMM in 2013, one year after he was originally scheduled to audition.

New Prosthetic, New Music

During his first private lesson with AIMM teacher Eric Sanders, Barnes described his concept for developing a robotic version of his drumming arm. Sanders showed Barnes some online videos of the robotic musicians from Georgia Tech. Through a chain of introductions, Barnes met Professor Gil Weinberg, the founding director of the university’s Center for Music Technology, who was ready to take on a new project.

Professor Weinberg and his students blend the worlds of music and technology, building musical robotic performers. They focus on the science behind how music is perceived by humans and then use machine learning to build algorithms, so the robots can understand music. The robots learn to comprehend musical elements such as beat, tempo, and syncopation. While they understand music the same way humans do, they can perform with unparalleled speed and virtuosity. They perform music that humans cannot.

“That’s what's interesting about this work,” Weinberg says. “We use musical ideas to push our robotic capabilities. Then the new robotic abilities push new ideas for music.”

Barnes emailed Professor Weinberg, asking if the GTCMT team would build a prosthetic arm so he could regain the capabilities of a wrist. He wanted a muscle-controlled prosthetic that would translate the muscle movements in his upper arm to control a mechanized forearm and wrist.

"Wrists are very important for expression of drummers, and we worked on these kinds of actuators for other robots. But we didn't have the experience of building robots that are actually an extension of the human body, like a Cyborg. This made the project very interesting.”

Professor Weinberg agreed to create a robotic prosthetic but was also interested in linking the design to his existing research on musical improvisation that was used with the robotic musicians.

“I said, ‘Sure, we're interested.’” Weinberg says. “Then I asked him, ‘What if the design allows you to have one stick being controlled by your muscles so you'll have your artificial wrist, and we add a second stick that has a mind of its own? It will improvise, and that will push you to new musical domains.’”

While not his original goal, Barnes readily agreed. Weinberg obtained a grant from the National Science Foundation to fund the development.

Professor Weinberg playing the keyboard in the GTCMT lab.

The Design Process

The initial design was completed in a mere six months. The drum strokes were modeled using MATLAB and Simulink. The prosthetic has motors that power two drumsticks, each capable of drumming faster than human drummers, at speeds up to 20 beats per second. The first is controlled both physically by Barnes’s arm and electronically using electromyography (EMG) sensors on his bicep. The second is autonomous, listening to the music and improvising a complimentary beat. It is a wearable robot.

A close-up of the autonomous robotic drumming prosthesis.

"This prosthetic is far superior to anything that’s currently available. It also works fairly similar to the way I would drum before. I just flex the same muscles that I normally would use, and they send a signal to the arm and make it act accordingly. The harder I flex, the tighter the grip is on the stick.”

There are also differences: The wearable robot enables Barnes to create truly unique music. It uses artificial intelligence (AI) to detect the beat, tempo, and density in his drumming and responds with a beat that complements what it “hears”. It also autonomously listens to the chords played by the guitarist and adjusts its speed based on the chords.

“The secondary stick has a mind of its own, improvising its own patterns accordingly. Latency isn’t an issue. It’s also extremely fast and can play polyrhythms that aren't humanly possible. It’s like I'm jamming with a robot. Sometimes it’s cool and creative, and sometimes it’s annoying. So, it’s just like a typical band practice with bandmates.” Barnes does maintain creative control: “It throws things at you that you wouldn't expect, so you have to be on your game. But it’s ultimately up to me to decide which drum I want to play it on, or even if I want to play the robotic drumstick on a drum in the first place. While I can’t control what it plays, I do control if it plays.”

At the start, the team relied heavily on electrical and mechanical engineering skills. With the robotic arm constructed and operating reliably, the team transitioned to a computer science and AI focus. They developed the machine learning and deep learning code that enables the prosthetic and other robots to interact with the musicians.

Zach Kondak (left), Jason Barnes (on drums), Professor Weinberg (right), in the GTCMT lab.

“We use different techniques of varying complexity, from very basic machine learning techniques such as Markov chains to more complex algorithms such as convolutional neural networks,” says Zach Kondak, a recent graduate of the GTCMT who specialized in robotic musicianship.

"For instance, we're able to run the simple algorithms in real time and have the robots interact with us very quickly. The deep learning algorithms, however, are able to actually compose very rich and structured music. That's done offline.”

Redesigning for Portability

The team would like to create a design that can be made available to other musicians who require prosthetics, but challenges remain. The current design is heavy and requires an AC power supply. It is also tethered to two computers for the signal processing. They are addressing these issues with a redesign.

The planned redesign will continue to work with EMG signals from the arm to control the device but will use embedded processors, so all signal processing will be completed on the device itself. Battery power will replace the AC power supply requirements. It will also be lighter weight, since supporting the weight of the device can become tiring during a long performance.

“We will make the next iteration portable, so Jason can use it on his own,” Weinberg says, “and have a career as a traveling musician.”

For Barnes, his ultimate goal is to tour. “Yeah, just playing music,” he says. “It doesn't get any better than that. That's always been my passion.”

Read Other Stories

BIOTECH