In This Realm, What Happens When a Search Engine Giant Doubles in Size?

By Caryl Felicetta, Single Throw COO

Well, the Google Monster* has done it again. Apparently an index of over 4 billion pages wasn't enough to capture the World Wide Web, because back in November 2004, the Googlebot (Google's spider that goes out and captures web pages for the index) doubled it's appetite and grabbed up another 4 billion pages, bringing their published index size (as of this writing) to a whopping 8,168,684,336! For those of you that don't want to count the number of digits and commas, that's a tad short of 8.2 BILLION pages!

So what does that mean to you and me? It means that there's a much larger pool of pages for Google's algorithm to determine what's most relevant to a searcher's request.

Essentially, it's much like the old, pre-video era arcade game where you try to persuade the big claw down into the pile of trinkets. Imagine that you are trying very hard to grab up that certain trinket, then someone dumps a whole 'nother batch of trinkets right on top of the one you were trying for. Now there's a whole bunch more to choose from so you may have to dig a little deeper. Sometimes the one you want comes right to the top, other times you dip the claw down and come up with something totally unexpected.

That's pretty much the type of scenario we are seeing in Google's results these days. This is reminiscent of the time in November 2003 that those of us "in the business" recall as the "Florida Update." About once a year, Google's engineers release new algorithms and apply them to a greatly expanded database. This causes havoc for both searchers and businesses alike as results tend to be in a flux for a few months before things start to make sense again.

So what's all the flux about?
Google's algorithm defines the results we see when we search on a specific term. In the background, this complex formula runs through an index, determining what's most relevant to your request. Besides analyzing the text content on your pages, these algorithms look at several other different factors to ensure that your content is REALLY relevant to the searcher's request. It's these factors, and the "weight" with which they are applied, that change during a major update. Google will test new weighting structures, as well as various methods to detect sites that are spamming,** all in an effort to increase the validity of their results.

With this live testing, we will see strange and varied results, lasting anywhere from several weeks to a few months. This may even include the dropping in rank of some pages. Once again, Google's goal is to provide the searcher with the most relevant information. That may mean that Google has found a more relevant page in their newly doubled index or we are still seeing the results of their testing process.

We also see cases where one searcher may see one list of results in their browser, while another executing the same search on a different machine may see a totally different list of results. This happens because results are delivered through more than one index server. The index on each server may not be the same or Google may be testing new algorithms on that particular server.

A small step for Google, one giant leap for business!
Google's expansion and algorithm update presents a great opportunity for us all. It opens up the world's most popular search tool to even more content. And it holds us even more responsible to build sites that are relevant to the viewer. Why the viewer? Because Google, Yahoo!, MSN and others are trying to deliver the most valuable and relevant content to their viewers. Their algorithms are not based on trickery or excess but genuine, relevant and valid content.

There's no taming this Monster.and I don't know that we want to. As long as their goals stay focused on delivering relevant content to the viewer, it's a win-win situation!

*The "Monster" designation is a term of endearment representing the shear giant, mondo-monster size stature that Google has obtained over the time of its existence. There is no actual "Google Monster" that we are aware of.

** "Spamming" is general term used to define those sites that are "pretending" to be relevant to a search term by filling the page senseless, repetitive text or delivering one set of content to the search engine and another to the viewer. Essentially, like the "spam" you receive in email, it's not really what you want or expect it to be. Google will actually ban or blacklist sites that they find practicing deceptive techniques.