Theory of Everything: Holy Grail or Fruitless Pursuit?

Astronomers using data from ESO's Very Large Telescope created this composite photo of the nebula Messier 17, also known as the Omega Nebula or the Swan Nebula. The image shows vast clouds of gas and dust illuminated by the intense radiation from young stars. CREDIT: ESO/R. Chini

NEW YORK – Einstein died before completing his dream of creating a unified theory of everything. Since then, physicists have carried on his torch, continuing the quest for one theory to rule them all.

But will they ever get there? That was the topic of debate when seven leading physicists gathered here at the American Museum of Natural History for the 11th annual Isaac Asimov Memorial Debate.

The quest for a theory of everything arises because two of the most celebrated, successful theories in physics are contradictory.

The theory that describes very big things – general relativity – and the theory that describes very small things – quantum mechanics – each work amazingly well in their own realms, but when combined, break down. They can't both be right.

And we can't just sweep that fact under the rug and continue to use them each as they are, because there are some cases in which both theories apply – such as a black hole.

"Its size is small in terms of length; its size is large in terms of mass. So you need both," explained Brian Greene, professor of physics and mathematics at Columbia University.

Scientists hope that a unified theory would resolve this incompatibility, and describe anything and everything in the universe in one fell swoop.

Vibrating strings

Many physicists say our best hope for a theory of everything is superstring theory, based on the idea that subatomic particles are actually teensy tiny loops of vibrating string. When filtered through the lens of string theory, general relativity and quantum mechanics can be made to get along.

For that reason, string theory has inspired many physicists to devote their careers to developing it since the idea was first proposed in the 1980s.

"There's been an enormous amount of progress in string theory," said Greene, a proponent of string theory whose 2000 book "The Elegant Universe" described the theory in layman's terms. "There have been issues developed and resolved that I never thought, frankly, we would be able to resolve. The progress over the last 10 years has only solidified my confidence that this is a worthwhile direction to pursue."

But other experts are getting weary of string theory, which has yet to produce concrete, testable predictions. Perhaps string theory, and the whole idea that a single theory can explain the universe, is misguided, they say.

Neil deGrasse Tyson, director of the museum's Hayden Planetarium, suggested that string theory seems to have stalled, and contrasted the lack of progress of "legions" of string theorists with the seemingly short 10 years it took one man – Einstein – to transition from special relativity to general relativity.

"Are you chasing a ghost or is the collection of you just too stupid to figure this out?" deGrasse Tyson teased, beginning a friendly banter that would continue throughout the night.

Greene admitted that string theorists have not produced testable predictions that experiments can confirm, but said it wasn't time to give up.

"As long as progress is carrying forward, you keep going," he said. "To say there's no progress, come on man, that's just not right!"

The theory is so complex, he charged, and deals with such fantastically small scales that are inaccessible to experimental data, that no wonder it's taking a while to crack.

"Nowhere is it written that we "have to solve problems in one human lifetime," agreed Janna Levin, a physicist at Barnard College in New York. I don't see why we should be shocked that solving incredibly challenging problems may take more than one human life span."

Hidden dimensions

One aspect of string theory that riles many is that many versions of it require the universe to contain more than the three dimensions of space and one of time that we are familiar with.

The most popular version of string theory, in fact, calls for 11 total dimensions.

"Why don't we see them?" Levin said. "It might be that they're very, very small. Or it might be that we are somehow confined to a three-dimensional kind of membrane. Or it might be that they're not there. But these are very interesting ideas that have some very compelling consequences."

Yet such a bizarre notion is disquieting to many.

"I'm a higher dimensional refusnik," said physicist Jim Gates of the University of Maryland-College Park, who argued that sometimes it seems like physicists invoke higher dimensions when they can't make their theory work as it is.

"It is not at all that we can't solve a problem so we pull extra dimensions out of a hat," Greene said.

"I'm just saying it looks that way," deGrasse Tyson said, carrying on the friendly debate.

Testing string theory

Luckily, the question of higher dimensions isn't entirely restricted to the theoretical domain. There is some hope that experiments such as the Large Hadron Collider – the world's most powerful particle accelerator in Geneva, Switzerland – will be able to provide experimental evidence of hidden dimensions in the universe.

The evidence may be in the absence of certain particles, or missing energy, that might result when a particle leaves our normal dimensions and enters one of the hidden ones.

"What we have to do is go to the highest energies at accelerators and send something off into the extra dimensions," said Katherine Freese, a physicist at the University of Michigan.

Another possible test for string theory will be analyzing the detailed observations of the light left over from the Big Bang, called the cosmic microwave background radiation, which permeates space. This radiation is thought to preserve an imprint of the tiny fluctuations in density that would have been present in the early universe, and might reveal evidence for some of string theory's predictions.

"If we're lucky we can actually use this to test some of the ideas of string theory by looking at imprints in the cosmic microwave background," Freese said.

Should we even be searching?

Ultimately, some physicists say the search for a theory of everything will be a fruitless chase.

"To me the problem of a notion of a theory of everything is that it implies we will eventually know everything there is to know," said Marcelo Gleiser, a physicist at Dartmouth College in New Hampshire. "For me physics is a work in progress."

As our knowledge of physics grows like an island, he said, so too will the "shores of ignorance increase." Thus there will always be more to know, bigger questions, greater areas of uncertainty.

"I have a disquiet with the dream of a search for the final theory," said Lee Smolin, a theoretical physicist at Perimeter Institute for Theoretical Physics in Ontario, Canada. He said the quest was incompatible with the modern way of physics, which has outpaced the scientific methods of Newton, in which scientists do experiments over and over, varying the initial conditions, to isolate the generalities, or laws, that apply.

Now, Smolin said, "we no longer can do experiments over and over again. There's one experiment, which is the universe as a whole."

We can't run other universes in test scenarios to understand cosmology, he said.

"No longer can we separate out the laws from the initial conditions. We are left with the question not just what are the laws, but why these laws? Why these initial conditions rather than other initial conditions? The method that Newton gave us no longer tells us how to go ahead. We have to change the methodology by which we try to understand the universe."

You can follow LiveScience senior writer Clara Moskowitz on Twitter @ClaraMoskowitz.