Robots.txt Generator

A robots.txt generator builds the small text file that tells search-engine crawlers which parts of your site they may visit. The file lives at your domain root (example.com/robots.txt) and uses a simple directive syntax. The standard example most sites need:

User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml

That says: all crawlers (*) may crawl everything except /admin/, and here is the sitemap. Get the syntax slightly wrong, though — a stray Disallow: / is the classic disaster — and you can de-index an entire site. Generating the file beats hand-typing it.

Pick your default policy (allow all or disallow all), list any paths to block — one per line, like /admin/ or /cart/ — add your sitemap URL, and the correct robots.txt appears ready to copy.

Common robots.txt examples this covers: the standard site (allow everything, block admin and checkout paths, declare the sitemap); the staging site (Disallow: / to keep test versions out of Google — just remember to remove it at launch); and WordPress-style setups (block /wp-admin/ while leaving content crawlable). Upload the result to your site root as robots.txt — it must be at the top level, not in a subfolder.

One boundary worth knowing: robots.txt controls crawling, not indexing — a blocked URL can still appear in results if other sites link to it. To keep a page out of Google entirely, use a noindex meta tag on a crawlable page. And robots.txt is public (anyone can read yours), so never list secret paths in it.