Fight or Flight: How the Nose Knows What to Do

This image illustrates the cellular diversity underlying sensory detection in the mouse vomeronasal organ. A spectrum of colors highlights sensory neurons harboring different receptors, each recognizing a different set of social or predator cues. CREDIT: Yoh Isogai and Catherine Dulac View full size image

Cats arch their backs at the smell of a rival, and mice scurry at the scent of a fox. But how does the nose know who or what is lurking? Now scientists have identified several special receptors in the noses of animals that react to specific scents given off by others.

It's these receptors that signal to the brain whether the animal needs to flee, make itself large and scary, or perhaps even woo a mate.

"Animals in the wild need to be able to recognize other animals, whether they are predators, potential mates or rivals," study researcher Catherine Dulac of Harvard University told LiveScience. "Many animals rely on the sense of smell; they can distinguish one type of encounter from another one based on chemicals."

Experimenting on mice, Dulac and her fellow researchers discovered that more of the animal's receptors seem to be dedicated to sniffing out predators than to detecting potential mates.

Animal senses

When a cat or mouse senses the chemical compounds secreted by other animals, it activates a special sensor in the nose called the vomeronasal organ. This organ, which is found in many animals and consists of a set of receptors, sends a signal to the brain, which interprets the signal and takes action. (Though humans have lost this organ, research has suggested humans do react in various ways to chemical cues.)

The first receptor genes were discovered more than 15 years ago by Dulac herself, but researchers are just now figuring out which receptors respond to what kinds of chemical signals. "From a given receptor gene, there is no information about what the receptor protein detects," Dulac said. "You have a bunch of receptors and you didn't know what they do."

The researchers set out to determine what chemicals these animals might be recognizing and how specialized these receptors are.

The researchers ran their experiments on mice, using a special method to identify the receptors in the nose that are associated with different vomeronasal responses and consequent behaviors. They marked the rodents' vomeronasal neurons with a compound that would make the neurons glow when activated. They also looked at which neurons expressed which receptors.

The researchers watched which brain cells lit up in response to a particular chemical signal, then determined which receptors the cells had in common, pinpointing which could be reacting with the signal.

Predator or potential mate?

Animals rely on these chemicals secreted by others to identify both individuals of their own species (in this case, they are often called pheromones) and those of other species. Dulac said that to the researchers' surprise, most of the receptors they found were cued to respond to scents from other species, especially those from potential predators.

"The vomeronasal organ has always been advertised as the organ that determines mating behaviors," Dulac said. "We expected we could find male-specific cues and female-specific cues, but actually most are responsive to other animals."

The mice have behavioral responses to these different chemicals. If they sense the chemicals of a predator, like a fox, they will quiver in the corner of their cages. "If the animal needs to recognize every possible predator ... then you probably need a lot of receptors just to be safe," Dulac said. "Some are specialized in detecting predators; some, even the type of predator — reptilian, mammalian, ferret or fox."

A human's vomeronasal organ seems to disappear during fetal development. The presence of human pheromones is still hotly debated, though. Some studies argue that pheromone-like chemical-behavior interactions could come from normal scent-based communications in humans.

The study was published today (Sept 21) in the journal Nature.

You can follow LiveScience staff writer Jennifer Welsh on Twitter @microbelover. Follow LiveScience for the latest in science news and discoveries on Twitter @livescience and on Facebook.