Robots.txt Guide: Syntax, Examples, Noindex, Sitemaps and Safe Deployment

First choose the outcome: crawl control, noindex, canonicalization, removal or privacy

A robots.txt rule tells compliant crawlers whether they may request a URL path. It does not require a search engine to forget a URL it already knows, and it does not stop a person from opening that URL in a browser. That distinction explains many robots mistakes.

If the page should remain public but not appear in supported search results, a crawlable noindex directive is usually the relevant mechanism. If the content moved, use a redirect. If the content is gone, return the appropriate not-found/gone response. If it is private, use authentication/authorization. robots.txt belongs only to the crawl-access part of that decision.

robots.txt must live at the root of the exact host it controls

For https://www.example.com/, crawlers look for https://www.example.com/robots.txt. A file at /folder/robots.txt does not control the whole site. A different subdomain or protocol is a different origin and can have its own behavior.

That means production, staging and preview hosts need deliberate behavior. A production site might allow crawling while a public preview host deliberately returns a fail-closed Disallow: / file. Test both hostnames instead of assuming one source file produces the same public behavior everywhere.

Understand the four lines most sites actually need

A user-agent line begins a crawler group. Disallow identifies paths the group should not fetch; Allow can make a more-specific exception where supported; Sitemap points to an XML sitemap URL. Comments begin with #. The safest robots file is often short because every additional wildcard or crawler-specific exception becomes another rule to maintain.

Do not assume line order means ‘last rule wins’. Matching behavior depends on crawler rules and specificity. For important cases, use the search engine's current robots documentation and test representative URLs.

# Production example
User-agent: *
Disallow: /api/
Disallow: /internal-preview/
Allow: /

Sitemap: https://www.example.com/sitemap.xml

User-agent groups should express real crawler differences, not copied folklore

Use a wildcard group when all supported crawlers can follow the same policy. Add a crawler-specific group only when you have a documented operational reason, such as a known high-cost bot or a provider-specific rule you actively maintain.

Copying a large robots file from another company can block resources your rendering stack needs, include paths that do not exist on your site, or create false confidence that sensitive areas are protected.

Test path matching against real URLs from your own application

Framework applications create routes for assets, APIs, search/filter states, previews, user-generated pages and static content. Decide which of those are useful public crawl targets. Do not block CSS/JavaScript simply because the directory looks technical; search engines may need resources to render public pages.

Also separate crawl-state URLs such as /tools?q=pdf from canonical content. A filtered state can be noindex/follow while /tools remains indexable. robots.txt is not a substitute for canonical/noindex architecture.

Why Disallow can conflict with a noindex plan

A crawler generally needs to fetch a page to see its robots meta noindex or X-Robots-Tag header. If robots.txt blocks the fetch first, the crawler may know the URL from links without being able to read the noindex directive. That is why ‘Disallow this page so Google removes it’ is often the wrong mental model.

For a public page that should be excluded from indexing, keep it crawlable while the noindex directive is present, then monitor the search-engine status. For content that should not be public at all, use access control instead.

The sitemap and robots policy should describe the same public site

A production sitemap should contain canonical public URLs you actually want crawled/indexed. Retired tools, old guide aliases, internal filters, redirects and 404 pages should not remain merely because their source code still exists.

The Sitemap line in robots.txt should use the real production hostname. A local or preview hostname in the production robots file is a release defect even if every page itself works.

1. 1
   Generate from the public catalog

   Only canonical published tools/guides/static pages.

2. 2
   Open representative sitemap URLs

   Expect HTTP 200 and self-consistent canonicals.

3. 3
   Check redirects/404s

   They should be absent from the sitemap.

4. 4
   Compare with robots rules

   Do not advertise sitemap URLs that your robots policy unintentionally blocks.

Three sample robots files—and why they differ

Templates are useful only when the site model matches. Use these as reasoning examples, not blind copy/paste snippets.

# Preview/staging host
User-agent: *
Disallow: /

# Do not publish the production sitemap here

Deploy, fetch and verify the actual production response

A generator preview is not the release. Routing layers, CDN/Worker logic, environment hostname checks and framework metadata routes can change the response. Request /robots.txt through the same public path a crawler sees.

Then monitor crawl/index reports after deployment. robots rules can reduce crawling of known low-value areas, but they do not guarantee immediate recrawl, deindexing or ranking changes.

1. 1
   Fetch production /robots.txt

   Confirm HTTP 200/plain text and exact directives.

2. 2
   Fetch preview /robots.txt

   Confirm it remains fail-closed if preview indexing is unwanted.

3. 3
   Open sitemap URL

   Production hostname and public canonical URLs only.

4. 4
   Test representative paths

   Public page, API route, filter state, retired route and asset.

5. 5
   Monitor search tooling/logs

   Validate actual crawler/index behavior instead of assuming the text file solved everything.

Common robots.txt mistakes

A good robots file is boring, explainable and covered by a release test. Complexity should be earned by a real crawling problem, not by the desire to look technically sophisticated.

• Using robots.txt as access control for confidential files.
• Blocking a page and expecting a hidden noindex tag on that page to be read.
• Disallowing rendering assets without understanding the impact.
• Publishing a sitemap full of redirects or retired pages.
• Copying wildcard syntax from another site without testing real paths.
• Serving production-friendly robots rules on preview/staging hosts.
• Assuming a robots change produces immediate removal or ranking changes.