Search engines cannot rank pages they cannot find. Two small files help you guide them: an XML sitemap, which lists the pages you want found, and a robots.txt file, which tells crawlers where they may and may not go. Used well, they make crawling cleaner. Used badly, they can hide your whole website.
This guide explains both files in plain language. They are part of the foundation covered in our guide to technical SEO.
Key Takeaways
- An XML sitemap lists the pages you want search engines to find. Robots.txt tells crawlers where they may go.
- Include only indexable, canonical, working pages in the sitemap.
- Robots.txt controls crawling, not indexing, so use noindex to keep a page out of results.
- Submit your sitemap in Search Console and check for errors regularly.
What Is an XML Sitemap?
An XML sitemap is a file that lists the important URLs on your website, often with their last update date. It does not guarantee that pages will be indexed, but it helps search engines discover them, especially new pages or pages with few internal links. Learn how pages get stored in our post on website indexing.
XML Sitemap Best Practices
- Include only pages you want indexed, which return a normal working status
- Leave out redirected, broken, duplicate and blocked pages
- Keep it updated automatically when you add or remove pages
- Use canonical URLs, as explained in our guide to canonical tags
- Split very large sites into several sitemaps, since a single sitemap is limited to 50,000 URLs
- Submit it in Google Search Console and reference it in robots.txt
What Is Robots.txt?
Robots.txt is a plain text file at the root of your site, for example yourdomain.com/robots.txt. It gives crawlers instructions about which areas they may visit. It controls crawling, not indexing, which is a key difference.
Robots.txt Best Practices
- Block only areas that genuinely do not need to be crawled, such as admin or internal search pages
- Never block the CSS and JavaScript files that pages need to display properly
- Add a line pointing to your XML sitemap
- Test changes before publishing them
- Remember that to keep a page out of search results you should use a noindex tag, not robots.txt alone
What a Good XML Sitemap Looks Like
- It lists only pages that return a normal status and can be indexed
- It uses the preferred canonical URL for each page
- It updates automatically when pages are added or removed
- It stays within the size limits, using several files if needed
- It sits at a predictable address and is referenced in robots.txt
- It is submitted in Google Search Console
Common Sitemap Errors and Fixes
| Problem | Cause | Fix |
|---|---|---|
| Sitemap could not be read | Wrong address, blocked file or invalid format | Check the address, access and format |
| Submitted URL blocked by robots.txt | The page is both listed and blocked | Remove the block or remove the URL from the sitemap |
| Submitted URL marked noindex | The page asks not to be indexed | Remove noindex or remove the URL from the sitemap |
| Submitted URL redirects | The listed URL is old | List the final destination URL |
| Submitted URL not found | The page was deleted | Remove it from the sitemap |
How to Create and Submit a Sitemap
- Generate the sitemap with your CMS, an SEO plugin or a tool
- Check that it lists only the pages you want indexed
- Open it in a browser to make sure it loads
- Add the sitemap address to your robots.txt file
- Submit it in the Sitemaps report of Google Search Console
- Check back regularly for errors
How to Read a Robots.txt File
A robots.txt file is made of simple lines. A User-agent line says which crawler the rules apply to. A Disallow line lists a path that crawler should not visit. An Allow line permits access to something inside a blocked area. A Sitemap line gives the address of your sitemap. A single Disallow line that blocks the whole site is a common and costly mistake, so always check this file after redesigns and launches.
Sitemap and Robots.txt Mistakes
- Listing every URL regardless of quality
- Forgetting to update the sitemap after big changes
- Blocking CSS or JavaScript files
- Using robots.txt to hide sensitive content
- Leaving staging blocks in place at launch
Not sure if search engines can crawl your site properly?
Get a free SEO audit and we will check your sitemap, robots.txt and other crawl issues.
Sitemap vs Robots.txt
| XML sitemap | Robots.txt | |
|---|---|---|
| Purpose | Lists pages you want found | Sets crawling rules |
| Works as | A helpful suggestion | A set of instructions crawlers usually follow |
| Location | Often /sitemap.xml | Always /robots.txt at the root |
| Biggest risk | Listing broken or blocked pages | Blocking important pages by mistake |
Common Mistakes
- Blocking the entire site after a redesign and forgetting to remove the block
- Listing non-indexable pages in the sitemap
- Using robots.txt to hide private content, since it is publicly visible
- Forgetting to update the sitemap after major changes
To go deeper into how search engines spend their time on your site, read about crawl budget. For expert help with crawling and indexing, see our SEO services.
Frequently Asked Questions
What is an XML sitemap?
An XML sitemap is a file that lists the important pages on your website so search engines can discover them more easily.
What does robots.txt do?
Robots.txt tells search engine crawlers which parts of your site they may or may not crawl.
Does robots.txt stop a page from being indexed?
Not reliably. It controls crawling, so use a noindex tag if you need a page kept out of search results.
How do I submit a sitemap to Google?
Add the sitemap URL in the Sitemaps section of Google Search Console.
Why does Search Console say my sitemap could not be read?
The address may be wrong, the file may be blocked or the format may be invalid. Open the sitemap in a browser and check it loads and follows the standard format.
Why are pages in my sitemap not indexed?
Submission does not guarantee indexing. Pages may be low quality, duplicates, blocked or simply not yet crawled. Check each page in the URL Inspection tool.
Do I need an XML sitemap for a small site?
It is not essential for very small, well-linked sites, but it is easy to add and helps search engines discover pages and updates faster.
Want your site crawled and indexed the right way?
Our team can review and fix your sitemap, robots.txt and crawl setup.
