SEO Challenge

+Tony McCreath  : exactly! That was what I was asking for: what the Disallow directive asks, not if search engines honor it. 🙂 It was intended just as a technical question, not as a way to talk about search engines correctness (I’m more interested into the engineering aspect of what a software does).

I think I can answer to your question about noindex and follow, from a technical point of view, because I studied some information retrieval and search engine database design. Please consider what follows just as a (common) way to develop a simple search engine database, I’m not referring to any actual implementation by a specific search engine.

The Main Index

First, some words about the main index. The index is not an archive containing the raw documents collected by the search engine but it’s a special database structure that makes it possible to quickly find what documents are related to some words (usually the words typed by the user during a search).

When a search engine downloads an HTML resource and parses it, the search engine basically splits all the text into single words, then it inserts those single words into the index, which is a list of rows like this:

the word “dog” appears in documents number 56, 356, 676, 803 and 4762.
the word “ball” appears in documents number 94, 257, 356 and 700.
etc…

These rows are one of the most useful structure of the database, but the important thing to understand is that after the indexing phase, there isn’t such a thing as the entire original document, because the document has been split into its component words.

Many search engines also keep a copy of the original raw resource (for example the cache copies shown to users) but these resources are not used during a search: only the index is queried.

The Links

What happens when the search engine finds a link on a document? The search engine does something very similar to what it did with the word index: it populates a second list, containing only information about links:

the document 356 links to the document 700 using the text “xyz”
the document 4762 links to the document 56 using the text “online casino”
etc…

Again, the original document doesn’t exist anymore, here. When the search engine needs to know something about what documents link to other documents, it queries this second list.

Two Different Structures

What I rougly described above it’s what actually happens to text documents when they are indexed. You have two different structures: the first one is a list of relations between words and documents, the second one is a list of the links found.

Let’s call it “word index” and “link index” (or “link graph”, if you prefere).

Noindex and Follow

How is it possible to “noindex but follow”? It is possible because text and links are stored in two completely different database structures.

“index” means: “please, add the words in this document to the word index”.
“noindex” means: “please, don’t add the words in this document to the word index”.

“follow” means: “please, add the links in this document to the link index.”
“nofollow” means: “please, don’t add the links in this document to the link index.”

Real Noindexing and Cosmetic Noindexing

When faced with a noindex or nofollow directive some search engines could decide to index words and links but to ignore those rows during a search. From an user point of view the result is the same, but in this way the noindex and nofollow directives lose their original meaning, because the search engine still has somewhere that information, even if it is ignored.

I hope that the explanation clarifies a bit what actually happens in the search engine software and database and how it is possible to “noindex but follow” when two completely different structures are used to save text and links. 🙂

For more information about the indexing, this Wikipedia article is a good reading: http://en.wikipedia.org/wiki/Search_engine_indexing

Now, I’ll try to reply to your other questions. You asked about Google but, again, these concepts apply to any search engine.

> Does Google still crawl and establish the link graph regardless of robots.txt directives and meta robots tags?

The robots.txt file talks to the spider while the meta robots tag talks to the indexer. This distinction is important in order to answer correctly to your question.

Any search engine which honors the robots.txt file will not ask for a disallowed resource. If a disallowed resource contains links, those links will not be perceived by the search engine. So the robots.txt file is a good way to actually influence the link graph.

About the robots meta tag and its nofollow directive, the links will be perceived but there is no way to understand if a nofollowable link is actually not inserted into the link graph or inserted but ignored. For sure, the search engine needs to save somewhere into the database the total count of links in the page.

> Why can Google still find links on a page that is blocked by robots.txt but can’t read the robots meta tag on the same page?

A resource disallowed in the robots.txt file is not requested by any popular search engine and as a consequence its contents (html code, inside text, links, HTTP headers) will not be perceived.

Can a search engine perceive a link in other ways, without downloading the resource? Theoretically there are some ways to (badly) achieve this goal, but I don’t know if any search engine actually uses them.

> Bonus: I think Google recently stated that they will still crawl blocked pages if they contain a Google+ badge on them!?

Yep!