Question 1

Does robots.txt remove pages from Google?

Accepted Answer

No. It controls crawling, not guaranteed indexing.

Question 2

Is robots.txt security?

Accepted Answer

No. Use authentication/authorization for sensitive content.

Question 3

Why is my page not indexed?

Accepted Answer

Robots may be one factor—also check noindex, canonical, and discovery.

Question 4

Which user-agent should I test?

Accepted Answer

Test the crawler you care about (e.g., Googlebot) and also “*”.

Question 5

Should I add a sitemap line?

Accepted Answer

Yes when available; it improves discovery.

Robots.txt Tester – Validate Allow/Disallow Rules for URLs