Robots.txt and XML sitemap best practices for Indian websites
Two files control a huge amount of how Google crawls your website. Most Indian websites have them configured incorrectly. Here is what they should look like.
I once audited a website for an Indian educational institute that had accidentally blocked Googlebot from crawling their entire site. They had been wondering why their rankings dropped for four months. The robots.txt file had one incorrect line. Four months of ranking loss from one mistake in one file.
Robots.txt and XML sitemaps are the two most basic technical SEO elements, and they get broken more often than you would expect.
What robots.txt does and how to write it correctly
Robots.txt is a plain text file that lives at yourwebsite.com/robots.txt. It tells search engine bots which parts of your site they are allowed to crawl. You are not legally blocking access to those areas, just giving instructions that well-behaved bots follow.
A basic robots.txt looks like this:
User-agent: *
Allow: /
Sitemap: https://yourwebsite.com/sitemap.xml
This allows all bots to crawl everything and points them to your sitemap. Simple and usually correct for most business websites.
Where mistakes happen is when developers add Disallow rules to block staging environments or admin areas and then accidentally copy those rules to the live site. Always double-check your live robots.txt after any deployment.
What to disallow and what to leave alone
Disallow your admin pages: /wp-admin/ for WordPress sites. Disallow your checkout and cart pages on e-commerce sites since these have no SEO value and waste crawl budget. Disallow search result pages if your site has internal search (typically ?s= parameters in WordPress).
Do not disallow pages you want to rank. This sounds obvious but I have seen Indian businesses disallow their blog folder by mistake, then wonder why their content marketing is not working.
XML sitemaps explained
An XML sitemap is a file that lists every important URL on your website. It helps Google find your pages faster, especially on larger sites or new sites with few inbound links. Your sitemap lives at yourwebsite.com/sitemap.xml (or sometimes /sitemap_index.xml for larger sites).
In WordPress, Yoast SEO, Rank Math, and the default WordPress SEO tools all generate sitemaps automatically. Submit your sitemap URL in Google Search Console under Index, then Sitemaps.
What should and should not be in your sitemap
Include: all pages you want indexed and ranked. Product pages, service pages, blog posts, category pages.
Do not include: pages that are set to noindex, pages blocked in robots.txt, pagination pages beyond page 1, duplicate content URLs, and thank-you or confirmation pages.
Submitting noindex pages in your sitemap sends Google a conflicting signal: "Here is this URL" and "do not index this URL" at the same time. Google handles this gracefully, but it is sloppy practice. Keep your sitemap clean.
Checking your sitemap health in Google Search Console
After submitting your sitemap, Search Console will show you how many URLs were submitted versus how many were indexed. A large gap between submitted and indexed is a signal worth investigating. It could mean Google is finding quality issues, crawl budget problems, or simply that some pages are very new.
For Indian e-commerce stores, I recommend splitting sitemaps by content type: one sitemap for products, one for categories, one for blog posts. This makes it easier to diagnose indexation problems by content type.
Frequently asked questions
How do I check if my robots.txt is blocking Google from crawling my site?
Go to Google Search Console, then Settings, then Robots.txt Tester. You can test any URL against your current robots.txt to see whether it is blocked. You can also simply visit yourwebsite.com/robots.txt in your browser to see its contents.
How often does Google re-read my robots.txt file?
Google typically re-reads robots.txt every 24 hours. If you make a change, Search Console usually reflects the update within a day. For urgent changes (such as accidentally blocking the whole site), you can request a re-fetch through Search Console.
Does having more URLs in my sitemap improve SEO?
No. Quality over quantity. A sitemap with 200 well-optimised pages is better than one with 2,000 pages including thin content, duplicates, and redirect destinations. Clean up your sitemap periodically to remove pages that have been deleted or are performing poorly.