Mysteries of the Unregulated Internet

This image is a visualization study of inbound traffic measured in billions of bytes on the NSFNET T1 backbone for the month of September 1991. The traffic volume range is depicted from purple (zero bytes) to white (100 billion bytes). It represents data collected by Merit Network, Inc. CREDIT: Donna Cox and Robert Patterson, courtesy of the National Center for Supercomputing Applications (NCSA) and the Board of Trustees of the University of Illinois

This Behind the Scenes article was provided to LiveScience in partnership with the National Science Foundation.

One Sunday afternoon last February, the YouTube website disappeared from the Internet. YouTube didn't take it down.

The problem came from Pakistan, when a telecommunications company suddenly began rerouting traffic to and from the website into an Internet black hole.

Incidents like this fascinate University of New Mexico (UNM) graduate student Josh Karlin. With an NSF grant he built and posted an Internet Alert Registry that automatically sends an email to registered Internet service providers when there is trouble with traffic in their section of the World Wide Web. The registry is free and any Internet service provider can sign up for the warnings at http://iar.cs.unm.edu/.

What happened

Karlin says anomalies in the way Internet traffic flows show up nearly 200 times a day. Most of the problems are small and disappear in a few hours. A few, like the hijacked YouTube traffic, are big and require Internet Service Providers to work together to solve the problem.

"What happened is the ISPs that were close to Pakistan Telecom, that were in fact forwarding Pakistan's data, said 'Oh, this is obviously wrong. We're not going to propagate it.' And then they shut it off," said Karlin. "They filtered it out and then suddenly the problem disappeared and YouTube was getting data again."

Karlin points out most Internet users don't think much about how it works. He says we assume someone, somewhere is in charge, taking care of problems, settling disputes, and punishing troublemakers.

But that's not true. The Internet works because thousands of independent ISPs work cooperatively together to keep traffic running smoothly.

How it all works

Every computer in the world that is connected to the Internet has an address. Those addresses come from the Internet Assigned Number Authority (IANA). That entity assigns the numbers, but it doesn't police them.

"The IANA has been giving out these addresses for a very long time, and people have lost track of where they've gone," said Karlin. "So some companies that were given Internet Protocol addresses have folded or sold it to other companies or broken them down into small blocks and given them out to other people, so nobody really knows what's where."

For instance, the University of New Mexico has thousands of Internet addresses assigned to it. But there is no agency that monitors whether UNM only uses the addresses it has been assigned. So how does any ISP sort of what is legitimate and what is not?

There are dozens of companies that sell services to help Internet Service Providers sort out suspicious activity from normal traffic. The IAR will alert providers as well. But researchers are only now trying to figure out how to handle suspicious traffic when it suddenly appears.

Karlin is one of them. He and his advisor, UNM computer science professor Stephanie Forrest and Princeton University computer sciences Professor Jennifer Rexford are working on an improvement to the Border Gateway Protocol. The modification changes preference to allow ISPs to automatically route traffic around a source that makes an unexpected change in routing.

Getting around problems

Their protocol emphasizes the status quo. If traffic is flowing along like it is normally, it means everything is ok. If traffic suddenly begins to flow in a different way, the yellow flags go up and their protocol automatically selects a more stable and trusted route.

That buys time for the ISPs to figure out whether this is a traffic hijacking, as occurred in the YouTube case, or not. If an alarm is raised, as it was by YouTube, the ISP can avoid using the anomalous route.

The Internet Alert System and the new protocol will eventually work together so that routers can automatically avoid suspicious routes while the pertinent ISPs are informed of the problem. This way, potential problems unfold slowly rather than instantly.

Karlin has noted that the Internet began as a messaging system between researchers who trusted each other, and so far the system still basically works on the idea that routes being advertised around the world are correct. But as more and more networks join the Internet, it becomes increasingly likely that mistakes will be made that cause problems. Karlin's new protocol treats the mistakes as mistakes rather than attacks and allows for a positive rather than a punitive solution.

• Internet Full of 'Black Holes'
• Innovation: Ideas and Technologies of the Future
• Why Global Hackers Are Nearly Impossible to Catch
  

Editor's Note: This research was supported by the National Science Foundation (NSF), the federal agency charged with funding basic research and education across all fields of science and engineering. See the Behind the Scenes Archive.