Question 1

Where should robots.txt live?

Accepted Answer

At the root of your site: https://example.com/robots.txt.

Question 2

Does robots.txt hide pages?

Accepted Answer

No—it’s crawl guidance, not security.

Question 3

Should I block /admin?

Accepted Answer

If it should not be crawled, yes—but secure it properly too.

Question 4

How do I verify rules?

Accepted Answer

Use Robots.txt Tester to test important paths for your user-agent.

Question 5

Should I include a sitemap URL?

Accepted Answer

Yes, when you have a sitemap.

Robots.txt Generator – Generate Allow/Disallow Rules + Sitemap