How do you fix a robots.txt blocking your site?
Open yourdomain.com/robots.txt and look for Disallow: / — one slash blocks everything. Correct the file, confirm pages report as crawlable in Search Console's URL inspection, resubmit the sitemap, and add a post-deployment check so a staging configuration cannot go live again.
Check it right now
Open https://yourdomain.com/robots.txt in a browser. The line that removes your whole site is:
User-agent: *
Disallow: /
That single slash blocks everything. It is the default configuration on many staging environments and it goes live with alarming regularity.
Also check for narrower blocks that are almost as damaging:
Disallow: /services/
Disallow: /*?
Disallow: /wp-content/
The second blocks every URL with a query string. The third can block the CSS and JavaScript needed to render your pages, which prevents a search engine seeing the site as a visitor does.
Understand what it does and does not do
robots.txt controls crawling, not indexing. Two consequences people get wrong:
Blocking a page does not remove it from the index. A blocked URL can still appear in results, usually with no description, if other sites link to it. To remove a page, use a noindex tag — and the page must be crawlable for that tag to be read.
Blocking a page prevents its noindex being seen. If you disallow a page and also noindex it, the noindex is never read. This combination is a common and confusing mistake.
Fix it and verify
Correct the file, then confirm:
- Load the live robots.txt and read it
- Use the URL inspection tool on several pages and check they report as crawlable
- Resubmit your sitemap
- Watch coverage reporting over the following weeks
Recovery is usually reasonably quick once crawling resumes — days to weeks depending on the site — but it is not instant, and the traffic lost while it was blocked does not come back.
A sensible file for a local business
Most small business sites need very little:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml
That is close to sufficient. Blocking admin paths, allowing everything else, and pointing at the sitemap with an absolute URL.
Resist the temptation to add rules you found in an article. Every additional line is a chance to block something important.
Prevent it happening again
This is entirely preventable with a small amount of process:
- Make robots.txt part of the deployment configuration, differing by environment, rather than a file that gets copied around
- Add a post-deployment check that fetches the live robots.txt and fails if it contains a blanket disallow
- Check it manually after any migration, replatform or host change
- Set up an alert for a sharp drop in impressions, which is how you find this if the check fails
Why it matters especially for a local business
A Katy business relying on local search does not have a large backlog of brand recognition to fall back on. A fortnight of invisibility during a busy season is real revenue, and the cost of the five-minute check is nothing by comparison.
Frequently asked questions
What line blocks an entire site?
User-agent: * followed by Disallow: / — a single slash. It is the default on many staging environments and it goes live with alarming regularity.
Does blocking a page remove it from search results?
No. robots.txt controls crawling, not indexing. A blocked URL can still appear, usually without a description, if other sites link to it. Use a noindex tag to remove a page.
Can I use robots.txt and noindex together?
Not on the same page. If the page is disallowed the crawler never reads the noindex, so the page stays in the index. It is a common and confusing mistake.
What should a small business robots.txt contain?
Very little — block admin paths, allow everything else, and point at the sitemap with an absolute URL. Every extra line copied from an article is a chance to block something important.
How do I prevent this recurring?
Make robots.txt part of environment configuration rather than a copied file, add a post-deployment check that fails on a blanket disallow, verify manually after any migration, and alert on sharp impression drops.