Google Indexing Problems

This article contains useful tips, help and advice to solve common Google indexing and crawling problems including partial website indexing. We offer a separate page on solving Google Supplemental Results.
Firstly, it's common for a low ranking website to be included in the Google index without actually being visible in the search engine results (SERP).
You can check whether your website is in fact indexed and cached by Google, using a simple query command. To accomplish this, type site:www.mydomain.com in a Google search window, replacing mydomain with your registered domain name.
If Google returns the message: "sorry no information is available for the URL www.mydomain.com" then none of your website pages are Google indexed and you may have a Google indexing or crawling problem.
Sometimes Google indexing problems can be the result of little more than Robots.txt file errors. This is a text file which sits in the root directory of your web server informing search engine robots what they should exclude when indexing a website. A Robots.txt error can sometimes prevent Googlebot (Google's search spider) from crawling your website altogether. For help with creating or formatting a Robots.txt file click here.
Check Google Cache

The Google site:www.mydomain.com query returns a list of all web pages in your domain which are indexed and cached in the Google index. If no webpages are indexed, this is often due the web domain being new or recently launched with not enough quality backlinks to make it into the index. To solve a Google indexing problem (including partial indexing) first check for website navigation problems which will prevent Google crawling your website.
If no website navigation problem problem is found, we recommend getting more quality links to your website from other WWW websites and that you consider submitting a Google Sitemap to inform Google about your website hierarchy and how often your content is updated. This will help influence how often Googlebot crawls your website with a view to helping Google keep a fresh cache of your recently updated page URL's.
Tips for Increasing Google Crawl Rate

Even a few additional links from other websites pointed to your domain can help increase the crawl rate and frequency of Googlebot visits, ensuring that your website is deep crawled more regularly.
It's worth mentioning that Google operates a smart-crawling system so it will notice when extensive page updates are made to a site as it interrogates and utilises the web server responses. Matt Cutts did an interesting video on how Google crawls sites and we'd recommend taking 5 minutes to watch it: Matt Cutts Googlebot crawl method video.
For more help and advice on acquiring additional inbound links for your website, read our informative link building and Google SEO strategies articles.
If, on the other hand, Googlebot is visiting your site too often then the crawl rate can be manually reduced from the Google Webmster Console (Webmaster Tools). Unfortunately, Webmaster Tools does not allow upward adjustment where a site gets infrequently crawled - you really need to get more inbound links (backlinks) and to update your website content frequently to encourage that.
More backlinks will significantly help to get more of your page URL's Google cached and fully indexed.

Google Big Daddy Update

Following the "Big Daddy" Google infrastructure update in Spring 2006, the crawling rate of websites is now heavily influenced by the number and quality of backlinks the site has acquired. For this reason, it is not unusual for a website with few inbound links to receive one or less Googlebot deep crawls a month.
Since Google's Big Daddy update, many websites have developed website indexing and Googlebot crawling rate problems. After Big Daddy, Google seems to be indexing fewer web pages, particularly on recently launched website domains and low quality sites. This has affected the backlink count for many sites, which previously relied on low quality directory links amongst other sources.
Partial Google indexing is now common for websites with few inbound links. This frequently results in only the top level domain (homepage) being Google crawled and included in the Google index, with other internal pages being partially indexed or not indexed at all.
The well documented Big Daddy update problems have now been resolved, but some new less new domains and older, trusted sites are still left with significant numbers of partially indexed pages in the Google index. These pages would have shown up as Google Supplemental Results until the labelling of such pages was removed in the summer of 2007.