How to Write a robots.txt File (With Examples)
The robots.txt file is a tiny text file with outsized power: it tells search engine crawlers which parts of your site they may and may not access. Used well, it keeps crawlers focused on what matters. Used carelessly, a single line can accidentally hide your entire site from Google. Here's how to get it right.
What robots.txt does (and doesn't do)
It lives at your domain root (yoursite.com/robots.txt) and gives crawl instructions. Crucially, it controls crawling, not indexing — a page blocked in robots.txt can still appear in results if other sites link to it. To truly keep a page out of search, use a noindex meta tag instead. Google's robots.txt documentation spells this out clearly.
The basic syntax
Two main directives: User-agent (which crawler the rule applies to) and Disallow (what not to crawl). A simple file that allows everything:
User-agent: *
Disallow:
And one that blocks a private folder for all crawlers:
User-agent: *
Disallow: /private/
The dangerous mistake
This single line blocks your entire site from every crawler:
User-agent: *
Disallow: /
It's a common accident on sites moved from a staging environment, where blocking everything was intentional. Always double-check this line before and after launch — it's the number one reason a new site never appears on Google.
Add your sitemap
Best practice is to point crawlers to your sitemap from robots.txt:
Sitemap: https://yoursite.com/sitemap.xml
Generate it without the risk
Rather than hand-editing and hoping, our free robots.txt generator builds a correct file from simple options, so you don't accidentally block the wrong thing. Pair it with an XML sitemap for complete crawl control.
Bottom line
robots.txt guides crawlers to the right parts of your site — but remember it governs crawling, not indexing, and that Disallow: / hides everything. Generate it carefully, link your sitemap, and verify it after every launch. It's one line of text that can make or break your visibility.
