A short robots.txt file is easier to review when each directive has a clear purpose. The Robots.txt Generator creates one user-agent group, a list of disallowed paths, and one sitemap reference. It prepares text; it does not inspect your site’s current crawl rules or change your hosting configuration.
Enter the intended scope
Set User agent to the crawler name you intend to address, or use the default * for the general group. Put one path per line in Disallowed paths (one per line). Every nonempty path must begin with /. Enter the complete sitemap address in Sitemap URL, then choose Generate result.
The user-agent value must fit on one line. The generator removes surrounding whitespace from each path and skips empty lines. It validates the sitemap as an HTTP or HTTPS URL without embedded credentials. It does not check whether that address serves a sitemap.
Inspect a concrete result
With user agent *, paths /admin/ and /drafts/, and sitemap URL https://example.com/sitemap.xml, the output is:
User-agent: *
Disallow: /admin/
Disallow: /drafts/
Sitemap: https://example.com/sitemap.xmlRead every path before downloading robots.txt. A short typo can change the area addressed by a directive. The tool accepts text beginning with /; that basic validation does not establish that a path exists, that a rule is useful, or that the finished file expresses the policy you intended.
Review deployment as a separate step
Add the reviewed file through your site’s normal source and deployment process. After deployment, inspect the publicly served file at the site’s root and compare it with the intended output. Generating a file in this browser does not upload it to a server. If the site already has robots rules, review the complete existing file before replacing anything.
Robots.txt is public crawl guidance, not access control. Do not list sensitive locations expecting the file to conceal them. Protect private content with appropriate authentication rather than relying on these directives. This generator also does not implement a noindex field, combine multiple user-agent groups, or evaluate crawler-specific behavior.
The input is processed locally and no site is fetched, so you can draft a public policy without connecting a hosting account. Prepare the referenced file with the XML Sitemap Generator when needed. For page titles and summaries, the Meta Preview covers a different part of the publishing review and should be checked independently of crawl directives.