Duke Team Attempts a Real-Life Version of CSI 'Zoom and Enhance'

The idea of an “enhance” button has been a staple of Hollywood crime dramas for years. You know the trope: Armed with nothing but a tiny, pixelated photo or grainy security camera footage, a team of detectives zoom in again and again until they are able to decipher a blurry license plate, or identify the killer in a reflection in the victim’s eyes -- essentially creating pixels out of thin air.

It’s long been the stuff of science fiction. But today, researchers worldwide are using artificial intelligence to build a more realistic “enhance” button, and now one Duke team is among them.

The Duke Data Science Team has developed a set of algorithms that are able to turn small, fuzzy images into more detailed ones in a matter of seconds. Their system of neural networks quickly makes a best guess at the missing details and patterns in a scene, without magnifying the image’s flaws.

“This is the closest you can get to ‘zoom and enhance’ while still being in reality,” said team member Sachit Menon ’20.

The technology is called super-resolution imaging. After working on their methods for just over two months last spring, the Duke Data Science Team was one of the top-ranked teams in the NTIRE 2018 Super-Resolution Challenge, held last June in conjunction with the Conference on Computer Vision and Pattern Recognition in Salt Lake City, Utah.

To show what their system can do, Menon and team member Nikhil Ravi ’20 pulled up a grainy low-resolution image of a mountaineer sinking his picks into the ice as he scales what looks like a frozen waterfall.

“You can see artifacts and noise everywhere, like the screen static you might see on an old TV,” Ravi said.

“And this is the image that our model gave us back,” he added, pulling up a version with four times the resolution of the one they started with.

Using their approach, the “upscaled” version doesn’t just have more pixels. It also has sharper edges, realistic textures, fewer artifacts. The tiny speckles that gave it the “TV static” effect are gone from the shadows.

“Their algorithm sharpened details in these high-resolution images that were only barely visible in the original low-resolution images,” said the team’s coach Cynthia Rudin, associate professor of computer science, electrical and computer engineering, and statistics at Duke.

The team trained their system using two sets of 800 images, one of high-resolution images and one of their low-resolution counterparts scaled down to fewer pixels per inch.

Based on what it “learned” by analyzing these pairs of photos, the system then takes a new, noisy, low-res image, works out what a cleaner, sharper version should look like, and mathematically fills in the missing pixels -- essentially creating new information that wasn’t there before.

To show how accurate their reconstruction is, Ravi opened up a high-quality original of the mountaineer for comparison. “There are some fine details that our system isn’t able to figure out because of the noise, like some patterns in the helmet. And it over-smooths some of the snow,” Ravi said. “But it does quite a good job.”

The team said their approach won’t help police identify and recognize a person from their face like on CSI. “You can’t stick an image from a crime scene through this and say, ‘oh it looks like this guy’s face,’” Menon says. “It’s extrapolating based on what it thinks people in general look like,” not specific individuals.

But it can help make out blurry text on license plates and elsewhere.

And of course, surveillance isn’t the only application. Many big-screen TVs already use similar technology to help convert standard definition content to fit their high-resolution displays.

Super-resolution imaging is also used in medical image analysis, for example to zoom in on suspicious regions in MRI or PET scans or mammograms; and in remote sensing, such as recognizing a military target in a satellite image.

For the NTIRE 2018 Super-Resolution Challenge, each team was given 100 low-resolution images of people, plants, animals, urban and natural landscapes and other scenes taken from the internet, some also with noise and motion blur, and asked to produce high-resolution versions from them.

When their results were compared with the high-quality originals, the Duke team’s methods ranked among the top, out of the hundreds of participants and more than 30 teams that competed.

Menon and Ravi worked alongside team members Yijie (Webster) Bei ‘20, Alex Damian ‘20 and Shijia (McCourt) Hu ’20.

“The students’ competition entry was a feat of fast engineering,” Rudin said. “Some of the teams they were competing against had been working in this area for decades.”

“Their images were crisp and clean, and they worked tirelessly to get them that way,” she added.

The Duke Data Science Team got its start as a computer science class first offered in spring 2018, taught by Cynthia Rudin, with teaching assistant Rachel Ballantyne-Draelos. Within its first six months the team has entered and won two competitions.

CITATION: "New Techniques for Preserving Global Structure and Denoising with Low Information Loss in Single-Image Super-Resolution," Yijie Bei, Alex Damian, Shijia Hu, Sachit Menon, Nikhil Ravi, Cynthia Rudin. https://arxiv.org/abs/1805.03383