Live Science Plus
Websites Engage in Rivalry To Catch Google's Eye
When Google assumed a near-monopoly on Internet searching and became the main tool for navigating the World Wide Web, the results of its searches became a modern-day Mother Lode. Webmasters competed against one another in a gold rush to be mentioned most prominently in Google's responses to a user's query. And, as in the actual Wild West, "outlaws" emerged to try to beat rival websites to the top of the list.
These rebels flaunt Google's own system for determining which websites are most relevant to the user's query, and they have stirred a controversy that has given them an unflattering name straight from Wild West imagery: "black hats."
To Google (and to the other search engines, which have their own systems for listing websites) , black hatters are rascals at best and criminals at worst, people who degrade confidence in their product and potentially direct users toward malware.
Black hatters counter that Google’s rules are arbitrary, with no legal bearing, and are unjust barriers to profiting in a marketplace in which hundreds of millions of dollars are up for grabs.
The exploiters
The first three sites that appear in a Google search result will draw orders of magnitude more traffic than the websites listed farther down. They also stand to make orders of magnitude more money, which has led to an entire industry called "search engine optimization." SEO revolves around engineering websites so that they attract the attention of Google’s computers.
Everyone from small bloggers to entrepreneurs to big media and global retail brands engage in SEO, in an effort to drive traffic to their own web sites through search engines.
Get the world’s most fascinating discoveries delivered straight to your inbox.
Named after what the good guys wore in Hollywood's old western films, SEO engineering that follows Google’s guidelines is called "white hat" SEO. Webpage alterations that violate Google’s terms to inflate a website’s search position past where it would naturally appear is "black hat" SEO.
“Generally, economics is what a lot of it comes down to," said Aaron Wall, founder of SEO Book, a search engine optimization consulting service. "Anything that is predictable, reliable and profitable goes from white to gray to black. It’s always shifting, and people design a business model to fit the algorithm.”
Although black hat SEO can encompass anything that violates a search engine’s rules, most black hat strategies exploit two areas of Google’s website-ranking mechanism: links to a page, and the page’s written content.
How black hat SEO works
Google’s computers read every webpage, then use a special equation called an algorithm to determine the content of the page, its relevance versus other websites that contain similar content, and its editorial merit. The more external websites that link to a page, and the more written phrases that Google’s algorithm associates with particular searches, the higher a site will appear in the search results.
“As a user, you care about this pollution of the search results. As a user, you’re hoping one of these search engines will provide the most relevant result with the fewest clicks as possible,” said Mark Risher, former Yahoo! Mail spam czar. “And the more that black hat SEO criminals lead to you finding less-relevant links close to the top, the worse your search engine experience will be.”
One black hat trick, called cloaking, involves crafting a ghost webpage visible only to the Google computers, and a separate page with different content that the human visitors see. The invisible page results show up in Google searches and can tricks users, who click on the link expecting to get a site about one topic, and then get a site about something else.
Some black hatters don’t even bother to hide the keywords in a ghost page; they simply populate a normal page with text that makes a website filled with advertising or malware appear legitimate to the Google algorithm, said Hillary Mason, lead scientist for the company Bit.ly.
“When you as a naive consumer make a query, you don’t know if one of the results you get back is one of these sites or not,” Mason said. “And they may steal your credit card info, or just make a lot of money sending you to particular vendors.”
Another common trick involves creating bogus websites that all link back to the relevant website. Google sees all these links, assumes the relevant site is important, and bumps it up in the rankings. Sometimes black hatters will buy links from sites. And there are black hatters who have resorted to illegal hacking, breaking into respectable websites and adding links to their site.
Hang 'em high
Some SEO engineers argue that Google is no better than the hanging judges and vigilante posses of the Old West. Google's rules are self-serving, they claim, and they charge that the company metes out punishment differently between large, profitable sites and small, powerless ones.
“I don’t sign anything for Google that says I’ll be a good boy and won’t be naughty,” said David Naylor, managing director of UK SEO. “I’m just creating a website to make money.”
In a high-profile case in 2006, Google caught car company BMW cloaking its website.
The BMW site consists primarily of car photographs, not the text that Google’s algorithm reads. As a result, the car company’s site ranked low in the search results.
To goose the ranking, BMW’s webmaster created ghost pages with BMW information. Even though all the information in the ghost pages was relevant, Google still considered it cloaking, since the Google computers and human visitors saw different things.
As a punishment, Google banned BMW from its search results.
These days, a banning from Google amounts to exile from the Internet as a whole, and could spell death for a company dependent on online revenue.
But unlike other websites, which often get banned for years, BMW’s site was reinstated by Google within two days, after it had removed some of the cloaking material.
“A lot of websites, like the BMW website, Google needs to have in the index. It’s not an option to exclude the website, because people are looking for the information. They would punish their users by excluding it,” said Mikkel Demid, director of Demid.com and a self-described black hatter.
“So there are certain websites that can get away with anything because Google needs them to be in there, and the users expect them to be in there. If Whitehouse.gov did cloaking, Google couldn’t do anything.”
Another instance that black hatters point to involves the blogging site Wordpress. In 2005, the company Hot Nacho paid Wordpress for hidden links on the Wordpress.org homepage to Hot Nacho advertising pages. This move allowed Hot Nacho to increase its search ranking, and thus ad revenue, by piggybacking on Wordpress’ high traffic, Wall said.
Google de-listed both Wordpress and Hot Nacho. After the hidden links were removed, Google reinstated Wordpress.org within a week. Meanwhile HotNacho.com remains banned from the search results.
Gray’s anatomy
In between black hat techniques and Google’s white hat guidelines lies the ambiguous no man’s land of gray hat SEO. Gray hat SEO encompasses any violation of Google’s rules done without malice, often as a second- or third-order side effect of seemingly white hat techniques.
“Most people are in the gray area, although I like to call it the colorful area, and are moving a little bit outside what you’re supposed to do,” Demid said.
One gray hat technique that is officially banned but widely employed involves using computer programs to generate articles, blog posts or social networking messages that appear written by humans. The artificially generated content emphasizes Google-attracting keywords. Wall said it is so pervasive and useful that Google turns a blind eye.
Google employs humans by the thousand to make subjective decisions about content's relevancy. A team of around 50,000 remote quality rankers works for Google evaluating search results, and in the process re-evaluates the boundary between white and black hate SEO, Wall said.
Through those evaluators, Google and the black hat community engage in an indirect dialogue about the morality of different search techniques. By showing the utility of a particular outlawed trick, or discovering a nefarious use for a seemingly benign practice, black hatters work with Google, albeit antagonistically, to advance the science of Internet search.
“Google to me is just a medium like any other media," Demid said. "There’s a market here, and I’m just trying to exploit that media the best way I can. Sometimes the best way to exploit the media is to go a bit outside conventions.
“I’m reminded of a quote by T.S. Eliot: ‘Only those who will risk going too far can find out how far one can go.’ I think he would have been perfect for black hat SEO.”