Skip to main content
NeatUtility

robots.txt Builder & Validator

Web & API100% Client-Side
Ready
TEXT
0 lines0 chars
0 B
About this tool, examples & FAQ

Generate, format, and audit robots.txt crawler directives locally with zero server transmissions. Validate Disallow and Allow paths against RFC 9309 specifications, test sitemap references, configure crawl delays, and export structured JSON crawler models.

Key facts

Verified capabilities and limits of this utility
Standards SpecificationRFC 9309 (Robots Exclusion Protocol - IETF Standard)
Supported DirectivesUser-agent, Allow, Disallow, Sitemap, Crawl-delay
Maximum Input50 KB
Execution Deadline2 s

How to use

  1. Paste your existing robots.txt directives or list of paths to allow or disallow.
  2. Select your target operation: Build robots.txt, Validate & Audit, or JSON Representation.
  3. Optionally customize default user agents, crawl delays, or append your XML sitemap URL.
  4. Click Run (or press Ctrl/Cmd + Enter) to generate normalized directives or view RFC 9309 audit warnings.
  5. Copy the verified robots.txt content directly into your website root directory.

Example

Build standard robots.txt with disallow paths and XML sitemap

Input

User-agent: *
Allow: /public/
Disallow: /admin/
Disallow: /private/
Sitemap: https://example.com/sitemap.xml

Example output

User-agent: *
Allow: /public/
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Edge cases

Empty Disallow vs Disallow Slash

In RFC 9309, `Disallow: ` with an empty value grants crawlers full access to the entire site, whereas `Disallow: /` completely blocks crawlers from indexing any path.

Crawl-delay Support Differences

While Bingbot, Yandex, and Baidu respect the `Crawl-delay` directive, Googlebot ignores it. Google recommends configuring crawl rates through Search Console instead.

Case Sensitivity and Path Prefixes

Path matching in robots.txt is case-sensitive. Disallowing `/Admin/` does not block crawlers from requesting `/admin/`.

How it works

  1. Parses directives into crawler user-agent groups following RFC 9309 line syntax.
  2. Validates URL path prefixes for leading slashes and syntax correctness.
  3. Verifies that Sitemap declarations reference valid, absolute HTTP/HTTPS URLs.
  4. Executes completely in-memory inside client Web Workers with zero external server calls.

Related workflows

Frequently asked questions

Where must the robots.txt file be located?

Search engine crawlers expect robots.txt exclusively at the root of the domain: https://example.com/robots.txt. Subdirectory locations such as /assets/robots.txt are ignored.

Does robots.txt hide sensitive pages from public access?

No. robots.txt is a public advisory file visible to anyone. It does not enforce password protection or access control. Never use robots.txt to secure confidential data.

Can disallowed pages still appear in Google search results?

Yes. If external websites link to a disallowed URL with anchor text, search engines may index the URL based on external signals without crawling the page content.

References