How fast google can work

Just goes to show the power of a blog based site, google seems to work quicker on blogs, my theory (only a theory) is that because blogs have auto link backs, generally content rich, and written by the more informative, google see's more relevance, this is also why it is key to stay on top of it, keep your posts coming that are relevant to the content, much like Mike is doing at SMART HSS, they have shown that being proactive is rewarding! Keep at it bro, its all gravy!

Having large database driven sites, I'm obsessed about indexing and like to see exactly what big G is up. So heres a sneak look at our indexing stats for MLS UK and USA for the last few days.

My Local Services UK
Date Pages Indexed
2010-03-25 28927
2010-03-24 20047
2010-03-23 24554
2010-03-22 31261
2010-03-21 16456
2010-03-20 19454

My Local Services USA
2010-03-25 7795
2010-03-24 6307
2010-03-23 6448
2010-03-22 6224
2010-03-21 6509
2010-03-20 5749

Now what I've found, (at least on our sites) any content that is brand new gets indexed normally within 12 hours. Any content that is updated, seems to get updated within 12 to 48 hours. The remaining static content is then periodically refreshed every couple of months.

The UK site currently has 20 times the traffic of the US site, and a lot more activity, sign ups, editing etc which is one of the reason more content is indexed, but also the US site currently doesnt have an xml site map.

The UK site has an xml sitemap which updates every night, and Google picks it up every day.

The US site doesn't have a sitemap, as we are running a 12 month experiment to compare the indexing. Having 16 million odd pages, to generate a daily site map will require another server, as test runs slow the US server to a crawl, so for the time being, we are breaking all the rules.

So hope it was of interest, may shine a little light on indexing habits, or reinforce your own opinions

All we need to wait for now is for Google to drink it's Caffeine (or may be it has already nobody knows!) and see what effect it has on it's algorithms.

For those that have not heard about Google Caffeine, it's an update to Google's search engine, which is going to focus more on a lot of what we've regarded as good practice but for what it's worth here is my view to make sure you're best placed for it.....

From the original testing we know that greater keyword density, quicker site speed, social media links, video listings, fresher content and keywords in the domain name are going to be important.

So, make sure your site is up to speed, create some videos (optimising for SEO), create some link baits, have real-time content and then maybe your site will be indexed even more frequently than ever