# Robots, Sitemaps & Crawling

Crawlers are programs that visit pages to index them for search engines and AI services. A robots.txt file asks them to skip certain paths, with separate groups of rules for different user agents, while an XML sitemap lists the pages you want found, up to 50,000 URLs per sitemap file.

## Tools

- Robots.txt Generator (Coming soon): Create a robots.txt file that allows or blocks crawlers by path and lists your sitemap.
- XML Sitemap Generator (Coming soon): Crawl a public website and build an XML sitemap of its pages to submit to search engines.

---

Category: [Web & SEO Tools](https://techolm.com/seo-tools)