Miguel Nicolelis chose a vast stage for the exoskeleton demonstration: the 2014 FIFA World Cup Opening Ceremony in Sao Paolo, Brazil.
Dr. Miguel Nicolelis emerged from the tunnel of the Arena de Sao Paolo into the brilliant Brazilian sunlight. The scene before him was breathtaking: dancers twirled around an enormous planet-like soccer ball pulsing with multicolored lights at midfield, music blared over the stadium’s speakers, and the stands were a cheering, roaring sea of yellow and green, the Brazilian national colors.
Nicolelis was nervous. He and a team of over 150 collaborators had been racing the clock for almost a year and a half in order to get to this moment. If they failed, they would fail in front of nearly 70,000 people in the stadium and hundreds of millions around the world watching the 2014 FIFA World Cup Opening Ceremony on television.
“It was the most intense feeling I’ve ever had as a scientist,” says Nicolelis, a distinguished Duke School of Medicine Professor of Neuroscience and co-director of the Duke Center for Neuroengineering. “I was running an experiment in front of a billion people. We knew there was a significant chance of something going wrong: a hydraulic line malfunctioning, a circuit failing, anything. It’s like launching a rocket. You do everything you can to make sure everything works, but until it happens you don’t know.”
One person in the 25-member team with Nicolelis in the stadium seemed unusually calm and confident. Juliano Pinto, a 29-year old Brazilian, had been confined to a wheelchair, with no movement or feeling in his legs, ever since he had suffered a complete lesion of the upper thoracic spinal cord in an automobile accident six years earlier, leaving him totally paralyzed from the mid-chest down.
Several assistants helped Pinto out of the golf cart that had carried him through the tunnel and into an extraordinary device called a robotic exoskeleton, a full-length frame fitted with hydraulic lines, mechanical joints, and digital circuitry. His legs were strapped to the exoskeleton’s legs, and on his head he wore a special cap equipped with electroencephalogram sensors. If all went as planned, the device would allow Pinto to do something that not long ago would have been called a miracle.
Most of us walk or run or kick a soccer ball just by thinking about it: our brain sends the “kick” signal to our legs, and they kick. In Pinto, or any other paraplegic, the channel that carries that signal has been severed. Nicolelis and his collaborators designed and built the exoskeleton and EEG cap to bypass the broken connection; the sensors should read Pinto’s neuronal signals and transmit them to the exoskeleton, which should respond accordingly: in this case, by making a kicking motion. The person attached to it essentially wills it, and by extension his legs, into motion. Pinto placed his arms on the exoskeleton’s armrests. Sensors there lit a series of blue lights, indicating that the exoskeleton was correctly receiving his brain waves. Nicolelis checked the timer on his phone. Pinto’s kick, the ceremonial kickoff of World Cup 2014, was supposed to take place precisely 18 minutes and 33 seconds into the tightly scripted ceremony. At the appointed moment Nicolelis signaled his team. A former Duke graduate student, Solaiman Shokur, PhD, knelt and placed a soccer ball at Pinto’s feet.
Pinto concentrated. He thought, “Kick.” And, in a remarkable feat of mind over matter, he kicked.
A GLIMPSE OF THE FUTURE
Pinto’s right leg flexed backward about 30 degrees and then swung forward to make contact with the ball, which rolled several feet. By World Cup standards, it was a humble kick indeed—but that gentle tap offered a glimpse of a future in which people might be liberated from the physical limitations of the body, in which patients with paralysis might one day be able to use their minds to power their legs and walk again.
For Nicolelis, the World Cup represented the culmination of 17 months of almost ceaseless work—and, in a larger sense, of the two decades he has spent at Duke conducting the research that ultimately led to Arena de Sao Paolo.
He and his team established the laboratory, built the exoskeleton, and trained the patients for the World Cup demonstration in Brazil because that’s where the World Cup was, and because the Brazilian government funded the project, along with ancillary programs, with a $14 million grant. It didn’t hurt that Nicolelis was born, raised, and earned his medical and doctoral degrees in Sao Paolo.
But the whole thing was built on years of work he and his lab have done at Duke developing Brain-Machine Interface (BMI) systems that use neuronal signals to control and manipulate external devices.
“This entire story was done here,” says Nicolelis. “That’s something I feel
very good about, because it came almost exactly 20 years after I came to Duke. The entire thing, from the early experiments in rats and then monkeys and now this, was done at our brain control lab at Duke. I think it’s something we ought to celebrate as an institution, because the intellectual development of this entire story belongs to this place.”
BRAIN POWER
That story grew out of work Nicolelis originally did with his postdoctoral fellowship advisor, John Chapin, at Philadelphia’s Hahnemann University after he came to the U.S. in 1989, and it expanded after he arrived at Duke as an assistant professor in 1994. In the late 1990s, Chapin and Nicolelis devised tiny sensors that, when implanted in specific neuronal regions of rats’ brains and connected to a BMI, allowed the rats to operate a robot arm without physically touching any controls, using only their brain activity.
In subsequent experiments, Nicolelis and his colleagues moved from rats to primates and expanded the range and complexity of their actions. An owl monkey named Belle used her brain activity via a BMI to move two sets of robotic arms at once, one in the lab where she sat at Duke and the other at the Massachusetts Institute of Technology. A rhesus monkey named Aurora learned to play a video game using only her thoughts. And in 2008, Nicolelis and his colleagues made inter
national news when they had a rhesus monkey at Duke named Idoya propel a 5-foot-tall humanoid robot to walk on a similar treadmill—7,000 miles away, in Kyoto, Japan—just by thinking about it.
Perhaps the most obvious potential clinical application of the concept would be to restore the transmission of signals from brain to limbs via BMI prosthetics in patients with paralysis. Nicolelis began thinking about creating an exoskeleton that could allow such patients to move their legs with brain activity alone.
Shortly after the Idoya experiment, Brazil was awarded the World Cup. To Nicolelis, a lifelong Brazilian soccer fan, the tournament represented an extraordinary opportunity.
“The Brazilian government wanted to showcase the country,” he says. “They wanted to highlight things besides music and soccer, like innovation, science, medicine, and technology, things that could give hope to millions of people around the world. I was able to visit the president and make a presentation to her. I said, ‘I can help a paraplegic Brazilian patient kick off the World Cup using a brain-controlled exoskeleton.’ She loved it, and she decided that we should go for it.”
PHANTOM LIMB
Nicolelis was going out on a limb. He had no brain-controlled exoskeleton, not even a prototype, and even if he could make one, he had no proof that it would work. He needed collaborators and money, lots of both. For the first, he established the Walk Again Consortium, a team of more than 150 scientists and engineers from around the world. For the second, the Brazilian government agreed to provide some $14 million to fund the project.
But all of that took time—the funding wasn’t approved until January of 2013—and the clock was ticking: World Cup 2014 would begin on June 12, 2014, exoskeleton or no exoskeleton.
“We had 17 months, and we had to go from nothing,” Nicolelis says. “We had to build a robot. We had to figure out the interface. We had to train the patients. We knew what we wanted to do, but we had never built such a complex device. How close did we cut it? I tell people we made it by about 30 minutes.”
The team built a lab in Sao Paolo and pitched into the work to design and build the exoskeleton and support systems. They identified eight patients, six men and two women, all of whom were paralyzed below the waist, and began training them on the exoskeleton.There were a thousand hurdles. But the biggest single technological challenge turned out to be a fundamental, though perhaps not immediately apparent, issue: giving the patients not only the power of movement, but also the sense of touch.
“Everybody thinks, ‘Let’s restore the motor component, let’s get the legs to move,’” says Nicolelis. “But the biggest discovery of this whole project, to me, was the importance of the tactile feedback. Without that, the patients felt like they were floating. They felt ungrounded, unsupported by the earth, and it was pretty scary. They found it very unsettling.”
Nicolelis’s team solved that problem by developing a system of sensors attached to the patients’ forearms that vibrated whenever their knees bent or their feet touched the floor. Their brains quickly adapted to interpret the sensation in the arms as the feeling of their legs moving and their feet touching the ground.
“When the foot is planted on the ground, they feel the pressure in the arm,” Nicolelis says. “After you do that for a few minutes, your brain gives you the illusion that you’re feeling your feet as you walk. In a way, we create a phantom limb sensation.”
FALLING INTO PLACE
As the World Cup drew near, everyone—scientists, engineers, and patients—stepped up the pace even more. It was exhausting work, maybe especially for the eight patients. Making their legs move was not a simple matter of thinking, “Walk.” It required enormous concentration, especially at first, and after they mastered the basics Nicolelis and his team began putting them through their workouts in a sort of virtual World Cup atmosphere.
“We would give them a lot of distractions: flashing lights, recordings of the loudest, most obnoxious soccer fans we could find,” Nicolelis said. “We wanted to know they could do it in the real world. But it becomes natural, like learning to ride a bike. At the beginning you had to think very hard about every movement, but then it falls into place. The brain adapts.”
In Sao Paolo, all eight patients learned to walk using their thoughts and the BMI exoskeleton. Pinto became the most adept, so he was chosen to make the World Cup kick. As the Opening Ceremony got underway, deep within the stadium Pinto did one last practice kick. Perfect. Then he, Nicolelis, and the rest of the team headed for the light at the end of the tunnel.
‘NOT JUST A KICK’
In the end, a lot of things did not go as planned, says Nicolelis. The stadium announcer was supposed to direct the crowd’s attention to Pinto. That didn’t happen. The television announcers were also supposed to alert viewers to the kick, and the broadcast was supposed to cut to Pinto for a full 30 seconds to show his approach, the kick, and the celebration. Those things didn’t happen, either. The TV broadcast cut to Pinto too late, and without any introduction. It happened so quickly that many viewers missed it entirely.
Nicolelis hasn’t gotten answers to what went wrong. But the goofs didn’t really matter, he says, because all the important things went right. Pinto kicked the ball, then yelled and thrust his right arm skyward in triumph. For a moment, everyone in the team seemed overwhelmed; for them, too, after so much work and sweat and worry, it was all over so fast.
Nicolelis approached Pinto. The young man was beaming, and he had tears in his eyes. “I felt it,” he told Nicolelis. “I felt the ball.”
“When he said that, that’s when I broke,” Nicolelis said. “Because that’s what we wanted. We wanted him to have a whole experience, not just a kick. I’ll never forget it. We were all crying.”
NEXT STEPS
Back when Nicolelis arrived at Duke two decades ago, he never dreamed that his work would eventually lead him back to Sao Paolo and one of the world’s biggest stages.
“Are you kidding?” he asks. “This Brazilian kid gets hired at Duke University, where my neighbors are all from Caltech and MIT and Harvard. I just wanted to survive.” He still has some mementoes to remind him of those early days. Underneath a window in his office at Duke’s Bryan Research Building stands a cracked black leather couch, and on the floor is a small boxy television set.
“Those were the first things I bought when I came to the U.S.,” he says. “My first encounter with the American Dream was that color TV and this couch to sleep on. I learned English by watching David Letterman’s old show on that TV.”
As for the future, he and his colleagues are planning a second exoskeleton prototype that will incorporate much of what they learned from the first one. The next one should be lighter and more maneuverable, he says.
“The patients told us they don’t need some of the things we thought they would, so we can remove those and bring the weight down,” Nicolelis says. “And we want to make it fully autonomous, so that a person could really walk around and turn and accelerate and slow down.”
THE INDISPENSIBLE MACHINE
The World Cup demonstration, which drew a great deal of publicity, also attracted its share of critics. They point out that other researchers are doing groundbreaking work in BMI, robotic prosthetics, and even exoskeletons too—just not in front of international TV audiences. Some scientists have questioned whether the scientific progress the project represents matches the attention it’s gotten.
Nicolelis says the discoveries and successes the Walk Again Consortium achieved represent significant scientific and technological gains and plant the seeds for future advances. The project will inform and influence the next generation of BMI devices, he says, perhaps most importantly in the area of tactile feedback.
What is clear is that the breakthroughs in BMI in the past 20 years have implications almost beyond calculating.
“I cannot even imagine what all the applications might be, but obviously there’s a huge list of possibilities,” says Nicolelis. “Eventually I’d expect the robotics to become much lighter, almost like a suit. We’ll need new, softer materials and new controls.”
Whatever technological breakthroughs await, the only really indispensible machine is the soft one encased inside the human skull. It is the extraordinary adaptability of the brain that has made possible any of the BMI work Nicolelis and others have spent the last two decades working on. Discoveries about the remarkable plasticity of the human brain have changed the entire field of neuroscience, Nicolelis says.
“When I came to the U.S. in 1989, the brain was still seen as a piece of engineering,” he says. “After childhood, people thought, it was fixed, carved in stone. But it’s not like that at all. It is never stable. It’s constantly in flux. And that has dramatically changed the way we approach the brain, intellectually and medically. That’s the main message of the 20 years of work I’ve done here."