Duke Computer Scientists Clone the Internet for Software Testing

Duke University computer scientists Amin Vahdat and Jeff Chase have developed an improved computer software equivalent of a scale model of the Internet for testing new programs before they "go live" on the real Internet.

According to the scientists such testing is critical because most software designed for the Internet is released without sufficient testing. Vahdat said, "Critical services such as finance, health care, and even the military, now run across the Internet. Unfortunately, essential Internet infrastructure shows only 98-99 percent reliability because of outright failures and inadequate resources to meet real demand. That translates to about three to seven days of downtime each year, with lost business, widespread inconvenience, and worse. There's a pressing need for better pre-release testing."

Vahdat explained why such scale-model environments, such as theirs called "ModelNet," are needed. "Testing Internet software is like trying to perform an experiment on the universe. You can't duplicate the universe to make sure you don't harm the real one or its inhabitants. You can't make a real duplicate of the Internet either. It's too big and chaotic. The result is that software is released for use on the Internet without ever passing realistic tests."

According to Chase "It's impossible to capture the full complexity of the Internet, but we believe ModelNet is much more realistic than previous testing alternatives. We hope that ModelNet will enable software researchers to adopt large-scale network emulation as a basic methodology in developing experimental Internet systems." Such network emulations can process the same data, run the same programs, and produce the same results as the system imitated, said Chase. "ModelNet marks a departure from previous emulations, which have tended to be small and static."

Chase said the need to make Internet services more reliable and "scalable" -- continuing to perform well under higher workloads and with greater numbers of users -- is driving many developers to create more elaborate software.

"There is a desire for software to take advantage of large numbers of systems located in many different places on 'wide-area' networks," he said. "These networks may span the nation. Dividing work dynamically among the computers on the network should make it possible for systems to work more reliably despite changing network conditions and user demand."

A team led by Vahdat and Chase created ModelNet because they needed a better way to test such software as systems to automatically allocate resources such as disk storage and computing power in very large distributed systems. Improving such "resource provisioning" in large networked systems prevents one computer from being overwhelmed while another is almost idle, and one disk from filling up while another is almost empty. However, Vahdat said, "An important problem in developing software for distributed resource provisioning is finding a way to test the software as you develop it -- to determine whether it can really do what you want it to do."

According to Chase, "Realistic pre-release testing seems like a prerequisite to making any claims about improving scalability and reliability."

Vahdat said previous approaches to pre-release testing, while valuable, suffer from serious limitations on the number of central network computers, called servers, that have been used. "The best we have been able to do until now is to test a program on a few dozen servers or write mathematical models with formulas that try to represent the way you think a new program may behave."

Chase said, "Wide-area Internet conditions change quickly in ways beyond anyone's control, including the researcher's. Conditions used for testing software on a couple of dozen servers on the Internet can't be reproduced to validate results. This makes test results unreliable for predicting real performance on the Internet." Chase said that mathematical modeling tools for software are useful, too, but "they run a high risk of failing to take into account all the interactions that can happen. Most critically, they don't run the real programs at all."

ModelNet works by running real application programs to be tested on some computers in a network, while others mimic the transmission characteristics of the Internet -- including data routing, delays and losses. Said Chase "Running real, unmodified programs is important. Rewriting programs to run in test environments raises questions about whether the changes make the testing less realistic."

Chase said ModelNet has already demonstrated the ability to run 10,000 instances of a real program, using a cluster of 350 computers to stand in for the Internet. "Thanks to support from the National Science Foundation and IBM, we have what we think is one of the largest academic 'test rigs' in the country."

Vahdat and Chase have decided to release ModelNet for use by others because of requests from other researchers. ModelNet is undergoing testing at Carnegie-Mellon, the University of California at Berkeley and other sites.