Robots.txt controls crawling, not secrecy
A disallowed URL can still be discovered through links or other sources. Do not use robots.txt to protect confidential information.
Group rules intentionally
Each user-agent group should contain the allow/disallow patterns intended for that crawler. Broad wildcards can block more of a site than expected.
Sitemap declarations are optional pointers
A Sitemap line tells crawlers where a sitemap lives; it does not validate the sitemap or guarantee indexing.
Test production behavior
Crawler interpretation and supported extensions can differ. Verify important rules with the documentation and testing tools of the search engine you care about.