To help solve a mystery, scientists enlisted unlikely detectives — mice.

The question involved baboons. Knowing who's considered family is critical for these primates, not only to avoid inbreeding, but to also help relatives along with nepotism. The problem is that in some baboon species, the females are promiscuous, potentially making it highly uncertain who is related to whom.

The answer seemed linked to scents, with related baboons perhaps carrying similar body odors. However, vision is supposed to be the most important sense in baboons as well as humans, with smell being of lesser importance, so a common view was that odors would not be used for kin recognition.

Human noses are often quite weak compared with the rest of the animal kingdom, making it hard for us to find out if baboons can be told apart by smell. Researchers therefore decided to draft much better noses — those of mice.

The researchers swabbed the armpits and groins of wild chacma baboons (Papio ursinus) from two different troops of the primates in Namibia. They next tested 24 adult male Swiss mice to scents from 14 adult female baboons. They chose female baboons partly because "some male mice were peeing on male baboon odors as if they were in competition," said researcher Aurélie Célérier, a behavioral biologist at the CNRS and the University of Montpellier II, France.

First, the scientists put a mouse in a box containing just one baboon's odor, long enough for it to get used to the scent. Next, the scientists replaced the first odor with two new odors, one from a baboon related to the animal used in the first box, the other not. Each new odor was placed in a hole at a different corner of the box.

They found the mice preferred exploring the hole with the scent from the unrelated baboon, presumably because it smelled unfamiliar.

"It's a well-known innate behavior of rodents to explore new things more than familiar ones," Célérier said.

These findings are evidence that odor might play a stronger role in primates than widely thought, including perhaps humans.

"Humans asked questions to mice on baboon odors and relatedness, and mice answered us," Célérier said.

In addition, "we have now a cost-effective biological tool to potentially ask a lot of questions to mice on odors of a lot of species," Célérier said. Besides primates, the researchers are now using mice to investigate bird odors.

Célérier and her colleagues detailed their findings online April 16 in the Journal of Experimental Biology.