Crawler (bot)
An automated program that fetches web pages. Search engines and AI systems use crawlers to read the web. Crawlers read the raw page and often do not run all of a site's scripts.
Updated August 21, 2026
A crawler fetches the raw HTML your server returns, and many crawlers do not run all of your JavaScript. If your headline, prices or copy are only assembled by scripts in the browser, a crawler can see an almost empty page. Server-rendered or static content, and a robots.txt that does not block your real pages, keep crawlers able to read you.
Related terms
- Retrieval. The step where a system finds and fetches your page. If a page is blocked, unreachable, or clearly not about the topic asked, it is never retrieved, and nothing else matters.
- Structured data (schema). Hidden labels in your page's code that state facts in a machine-readable format, for example that a block of text is your business's name, address and phone number. It removes guesswork for machines.
- Thin content and doorway pages. Mass-produced pages with little unique value, for example the same page repeated with only a city name swapped. Search engines demote these, and AI systems ignore them. The fix is genuine, unique content per page, not cosmetic variation.