Crawling and indexing
I review which pages search engines can fetch and which belong in search results. Robots.txt controls crawling, but does not reliably prevent indexing. To exclude a page, I check measures such as a noindex setting Google can read.
- Robots.txt and robots settings such as noindex
- Preferred URLs (canonicals) and duplicate content
- Indexing status and the XML sitemap