Robots.txt is a plain text file at the root of your domain that tells search engine crawlers which URLs they may and may not request. Getting robots.txt for SEO right matters because one careless line can block your entire site from Google, and one misunderstood rule can leave private URLs sitting in search results. It controls crawling, not indexing, and confusing those two things is the most expensive mistake in this whole topic.
Most robots.txt files I see in audits are either dangerously permissive or quietly broken. The file looks trivial, so nobody reviews it, and then a staging Disallow: / survives a migration or a wildcard blocks half the product catalogue. This guide covers exactly how the file works, the syntax that matters, and a repeatable audit you can run in ten minutes.
What Robots.txt Actually Controls (and What It Doesn’t)
Robots.txt implements the Robots Exclusion Protocol, which Google formalised as an internet standard in 2022. When a compliant crawler like Googlebot arrives at your domain, the first thing it fetches is /robots.txt. The rules it finds there decide which URLs it is permitted to crawl.
The critical distinction is crawling versus indexing. A Disallow rule tells a crawler not to request a URL. It does not tell Google to keep that URL out of the index. If other pages link to a disallowed URL, Google can still list it in results, showing the bare URL with the message “No information is available for this page.” I have seen client login pages and internal search results ranking this way for years because someone assumed Disallow meant “hide this.”
So robots.txt is a crawl-control tool, full stop. If your goal is to keep a page out of search results, robots.txt is the wrong instrument. You need a noindex robots meta tag or X-Robots-Tag header on a page that Google is allowed to crawl. This is the difference that separates people who understand technical SEO from people who have merely read about it.
There is one more thing robots.txt does not do: it is not a security boundary. The file is public, anyone can read it, and disallowing /admin/ just advertises where your admin panel lives. Never rely on it to protect sensitive content. Use authentication for that.
Robots.txt Syntax: The Rules That Matter
The syntax is small. A robots.txt file is a series of groups, each starting with one or more User-agent lines followed by Allow and Disallow rules. A blank line separates groups.
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help
User-agent: Googlebot
Disallow: /internal/
Sitemap: https://example.com/sitemap.xmlUser-agent names the crawler a group applies to. * matches every crawler that does not have its own dedicated group. Here is the trap that catches people: a crawler obeys only the most specific group that matches it. If you have both a * group and a Googlebot group, Googlebot ignores the * group entirely and follows only its own. It does not merge the two.
Disallow specifies a path prefix the crawler must not request. Disallow: /cart/ blocks everything under /cart/. An empty Disallow: means “nothing is disallowed,” which is how you grant full access. Allow carves exceptions out of a broader Disallow, and it is essential for patterns like blocking a folder but permitting one file inside it.
Two wildcard characters give you pattern matching. The asterisk * matches any sequence of characters, and the dollar sign $ anchors a rule to the end of a URL. So Disallow: /*.pdf$ blocks every URL ending in .pdf, while Disallow: /*?sort= blocks any URL containing that query parameter. When a URL matches both an Allow and a Disallow, Google follows the most specific rule, measured by path length; on an exact tie, Allow wins.
The Sitemap directive is independent of any user-agent group and can appear anywhere in the file. It should list the full absolute URL of your XML sitemap. This is worth doing even though you also submit sitemaps in Search Console, because other crawlers discover them here. If you want the full picture on sitemap files, I wrote a dedicated XML sitemaps guide.
Robots.txt for SEO: The Directives Worth Knowing
Two once-common directives are now dead, and you should remove them. Google stopped supporting noindex inside robots.txt on 1 September 2019, when it dropped support for unpublished and unsupported rules in the Robots Exclusion Protocol. If your file still contains Noindex: lines, they do nothing. Google also ignores Crawl-delay; to manage Googlebot’s crawl rate you use crawl-rate settings and, more effectively, fix the underlying server and site issues that make aggressive crawling a problem.
What you should use robots.txt for is disciplined. Block crawl traps: faceted navigation parameter combinations, internal search results, session IDs, and calendar pages that generate infinite URLs. These waste crawl budget and add nothing to the index. Managing this well overlaps heavily with crawl budget optimisation, which matters most on large sites.
Do not use it to hide thin or duplicate content you actually want removed from the index, because, again, a disallowed page can never be crawled to read a canonical or noindex signal. For duplication, canonical tags and noindex are the right tools. Robots.txt is for cutting off crawl paths you never want touched at all.
A minimal, correct robots.txt for most sites looks like this:
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?*add-to-cart=
Allow: /
Sitemap: https://example.com/sitemap.xmlThat is it. If your file is much longer than this, question every line.
The 7-Step Robots.txt Audit
In my experience auditing sites, the robots.txt review takes ten minutes and catches problems worth thousands in lost traffic. Here is the exact sequence I run.
-
Fetch the live file. Load
https://yourdomain.com/robots.txtin a browser. Confirm it returns HTTP 200 and renders as plain text, not HTML. A robots.txt that returns a 404 or a soft-404 HTML page is treated as “allow all,” which may or may not be what you want. -
Check for the catastrophic line. Search the file for
Disallow: /with nothing after the slash. On a live site this blocks everything. It is the number-one robots.txt disaster, usually left over from a staging environment. If it is there and the site is live, it is a fire. -
Verify rendering resources are allowed. Make sure you are not blocking
/wp-content/,/assets/, CSS, JS, or image directories that Google needs to render pages. Blocking these degrades how Google understands your content and mobile usability. -
Confirm the sitemap line. Check that a
Sitemap:directive points to your current, correct, absolute sitemap URL. Stale or missing sitemap references are common. -
Test high-value URLs. Take your money pages, key category pages, and top blog posts and test each against the file using Google Search Console’s robots.txt report or a parser. None of them should be blocked.
-
Look for redundant and conflicting rules. Overlapping
AllowandDisallowrules, duplicated groups, and thousands of hyper-specific lines usually mean the file has grown by accretion. Consolidate with wildcards. -
Check every subdomain and protocol. Remember each host needs its own file. Verify robots.txt on
wwwand non-www, on any subdomains, and confirm https and http agree. A migration to https that leaves an old restrictive http robots.txt causes strange, hard-to-diagnose crawling.
Run this whenever you launch, migrate, or replatform a site. A robots.txt for SEO review belongs in every technical SEO audit as a standard first check, not an afterthought.
Common Robots.txt Mistakes That Tank SEO
The single worst mistake is the leftover Disallow: / from staging. It deindexes the entire site over the following weeks, and because nothing breaks visually, teams often miss it until traffic craters. Every launch checklist should verify this line is gone.
The second is blocking a URL to remove it from the index. As covered, this backfires: the page stays indexed as a naked URL, and you have destroyed your ability to serve a proper noindex. If a page is already indexed and you want it gone, keep it crawlable, add noindex, wait for a recrawl, and only then consider disallowing it.
Third is over-blocking with a careless wildcard. A rule like Disallow: /*? intended to catch tracking parameters can also block paginated pages, filtered categories, and legitimate query-string content. Test every wildcard against real URLs before shipping it. According to Ahrefs research on large-scale crawl data, misconfigured crawl directives are among the most common technical issues found across audited sites, precisely because they are easy to write and rarely tested.
Fourth is treating robots.txt as private. It is a public file, so never document sensitive paths in it. Fifth is forgetting subdomains and letting a permissive or missing subdomain robots.txt undo careful work on the main domain.
Testing and Monitoring Your Robots.txt
Never edit robots.txt live and hope. Use Google Search Console’s robots.txt report, which shows the file Google last fetched, flags parsing warnings, and lets you test whether a specific URL is allowed for Googlebot. This is the source of truth for how Google actually reads your file, which can differ from how you think it reads.
Set up monitoring so you know if the file changes unexpectedly. A robots.txt that suddenly returns a 503, a 404, or an unexpected Disallow should page someone. Google’s documentation notes that if robots.txt returns a server error (5xx) for an extended period, Google may stop crawling the site entirely, treating the unreachable file as a signal to back off. That is a quiet, severe failure mode on sites that do not watch for it.
After any change, test the high-value URLs from step 5 of the audit again, then confirm in the Search Console robots.txt report that Google has fetched the new version. Crawl signals take time to propagate, so verify rather than assume. If crawling and indexing feel off after a change, robots.txt is the first place I look, and a broader technical SEO consulting engagement almost always starts here because it is cheap to check and expensive to get wrong.
Robots.txt for SEO is a small file with an outsized blast radius. Treat it as production infrastructure: keep it minimal, understand that it governs crawling and never indexing, test every rule against real URLs, and monitor it like you would any other critical endpoint. Do that, and it will quietly do its job for years. Ignore it, and it is one deploy away from taking your rankings down with it.
Frequently Asked Questions
Does robots.txt stop a page from being indexed?
No. A Disallow rule blocks crawling, not indexing. If a blocked URL has inbound links, Google can still index it and show it with no snippet, usually labelled “No information is available for this page.” To keep a page out of the index, let Google crawl it and serve a noindex robots meta tag or X-Robots-Tag header. Blocking the page in robots.txt actively prevents Google from seeing that noindex.
Where does the robots.txt file have to be located?
At the root of the host: https://example.com/robots.txt. It only governs the protocol, host, and port it is served from, so https and http, and www and non-www, are separate origins that each need their own file. A robots.txt in a subfolder like /blog/robots.txt is ignored completely, and subdomains do not inherit the main domain’s rules.
Should I block CSS and JavaScript in robots.txt?
No. Google renders pages like a browser and needs CSS and JS to understand layout, content, and mobile-friendliness. Blocking these resources is a technical SEO mistake that can hurt rankings, and Google’s documentation explicitly asks you to allow crawling of the resources needed to render the page.
What is the difference between Disallow and a noindex tag?
Disallow controls whether a crawler may request a URL; noindex controls whether a crawlable URL may enter the index. Use Disallow to save crawl budget and keep bots out of infinite or low-value URL spaces. Use noindex to remove a specific crawlable page from results. Never apply both to the same URL, because a disallowed page can never be crawled to read its noindex.
How big can a robots.txt file be?
Google enforces a maximum parsed size of 500 kibibytes; content beyond that is ignored, which can silently change how rules are interpreted. In practice a correct robots.txt is short. If yours approaches the limit, you almost certainly have redundant rules that should be consolidated with wildcards.
