Indexing & Crawling Vintage

Before Google can rank a page, it must first discover, crawl, and index it. Bill Slawski analyzed numerous patents that reveal the mechanics of how search engines manage the enormous task of keeping track of the web.

The Indexing Pipeline

Google's Inverted Index

Bill's 2021 analysis explained how Google's inverted index works — the fundamental data structure that makes search possible.

How an Inverted Index Works

What Gets Stored in the Index

Data Type	What Gets Indexed	Why It Matters
Terms	Individual words and phrases	Basic retrieval
Position data	Where terms appear in the document	Phrase matching, proximity
Structural data	Which HTML element contains the term (title, heading, body)	Weight by position
Entity data	Recognized entities and their attributes	Entity-based retrieval
Link data	Anchor text associated with links	Link graph construction
Metadata	Page title, description, last crawl date	Result display, freshness

Source: Google's Inverted Index of the Web (2021)

Crawl Rate Management

Patents describe how Google decides how often to crawl each page on the web.

Crawl Priority Factors

Crawl Budget Factors

Factor	Effect on Crawl Budget
Site size	Larger sites get more crawl budget
Update frequency	Frequently updated sites get crawled more often
Server response time	Fast servers can serve more pages to the crawler
Crawl errors	Many errors reduce crawl budget
Page importance	High-value pages (by link metrics) get priority
Robots.txt directives	Blocking pages preserves crawl budget for others
Sitemap presence	Sitemaps help Google discover and prioritize pages

Duplicate Content Filtering

Google's indexing pipeline includes multiple stages of duplicate detection:

Filtering Pipeline

Bounce Pad Site Detection

A specific type of duplicate filtering targets "bounce pad" sites — sites that exist primarily to redirect users through chains of pages:

• Redirect score — Measures the proportion of pages that redirect
• Spam score — Measures other spam indicators on the site
• Combined high scores trigger bounce pad classification and removal

Source: How Google Might Filter Out Duplicate Pages from Bounce Pad Sites (2011)

Web Decay Detection

Bill documented patents describing how Google handles "web decay" — when pages become stale, broken, or abandoned.

Web Decay Indicators

Indicator	Detection Method
404 errors	Direct crawl response
Content staleness	No changes detected over many crawls
Broken outbound links	Links on the page lead to dead pages
Certificate expiration	SSL certificates not renewed
Domain expiration	Domain registration lapsed
Hosting issues	Frequent downtime or slow responses

Impact of Web Decay on Ranking

• Pages showing decay signals may be crawled less frequently
• Severe decay (persistent 404s, expired domains) leads to removal from the index
• Stale content on time-sensitive topics gets demoted
• Broken outbound links may reduce page quality signals

Anchor Text Indexing

Anchor text from inbound links is indexed as an additional content signal for the target page:

How Anchor Text Augments the Index

This means a page can rank for terms it does not contain in its own content, based on what other pages say about it through anchor text.

URL Handling and Canonicalization

Patents describe sophisticated systems for handling URL variations:

URL Normalization

URL Variant	Canonical Resolution
http:// vs https://	Prefer HTTPS
www. vs non-www	Follow redirect/canonical
Trailing slash vs no slash	Treat as same page
Query parameters	Filter tracking params, keep content params
Fragment identifiers	Generally ignored for indexing
URL encoding	Normalize encoded characters
Mixed case	Convert to lowercase

Page Segmentation for Indexing

Before indexing content, Google segments pages to identify the most important content blocks:

Segmentation for Index Weighting

Source: Web Page Segmentation, and Identifying the Most Important Block (2008)

Key Takeaways

1. The inverted index is the foundation — Understanding how terms, positions, and metadata are indexed explains why on-page SEO works.
2. Crawl budget is finite — Optimize server speed, fix errors, and use sitemaps to ensure important pages get crawled.
3. Duplicate content is filtered at multiple stages — URL, content, and near-duplicate detection all run during indexing.
4. Anchor text augments your index entry — Inbound links add terms to your page's index that may not appear in your own content.
5. Web decay is detected — Maintaining live, updated, functioning pages is necessary to maintain index presence.
6. Page segmentation affects indexing weight — Main content carries more index weight than boilerplate elements.