The Power of IndexNow and Crawler Hints

Search engines find changes by recrawling your site on their own schedule. A new post can sit unnoticed for days. IndexNow flips that around: instead of waiting to be crawled, you tell search engines which URLs changed. Cloudflare Crawler Hints does the same thing automatically, without any code.

What IndexNow is

IndexNow is an open protocol for telling search engines that a page was added, updated or deleted. You send one HTTP request with the affected URLs, and the engine that receives it shares them with the other participating engines.

How it works

1. You host a key file on your domain to prove ownership.
2. When content changes, you send the changed URLs to an IndexNow endpoint.
3. The search engine checks the key, then schedules those URLs for crawling.

Who supports it

Submitting to one endpoint notifies all participants. The IndexNow FAQ lists Amazon, Bing, Naver, Seznam.cz, Yandex and Yep.

Google is not a participant. It tested the protocol in 2021 but hasn’t adopted it, and I found no sign that has changed as of October 2026. For Google you still rely on sitemaps and Search Console.

What it doesn’t do

• It doesn’t guarantee indexing. The FAQ says submitting a URL “does not guarantee immediate indexing”. It signals a change, and each engine decides whether to crawl and index.
• It isn’t for unchanged pages. Resubmitting URLs that haven’t changed wastes crawl quota and can get you a 422 or 429 response.

Setting it up

1. Create and host a key

Make up a key of 8 to 128 characters (letters, numbers and dashes), then serve it from your site root as a text file whose content is the key itself:

https://example.com/a1b2c3d4e5f6a7b8.txt

a1b2c3d4e5f6a7b8

In an Astro site, put the file in public/ and it’s served as-is. The key proves you own the domain, so it isn’t a secret.

2. Submit a URL

For a single URL, a plain GET request is enough:

curl "https://api.indexnow.org/indexnow?url=https://example.com/blog/my-post/&key=a1b2c3d4e5f6a7b8"

For several URLs, send a POST with up to 10,000 of them:

curl -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "host": "example.com",
    "key": "a1b2c3d4e5f6a7b8",
    "keyLocation": "https://example.com/a1b2c3d4e5f6a7b8.txt",
    "urlList": [
      "https://example.com/blog/my-post/",
      "https://example.com/blog/"
    ]
  }'

3. Read the response

4. Automate it on deploy

Submitting by hand doesn’t scale. Run the request as a step after each deploy, sending only the URLs that changed. Don’t resubmit your whole site on every build. The FAQ advises against resubmitting unchanged URLs and suggests waiting at least 5 to 10 minutes between updates to the same URL.

What Crawler Hints is

Crawler Hints is a Cloudflare feature. Cloudflare treats a cache MISS on your zone as a sign that content probably changed, and reports it through IndexNow. Responses with status codes above 4xx aren’t reported. You don’t write code or host a key.

You can’t set page priorities with it, and it doesn’t process sitemaps or structured data. It sends “this URL changed” signals and nothing else.

To turn it on, go to your zone in the Cloudflare dashboard and open Caching → Configuration → Crawler Hints. It’s available on the Free, Pro, Business and Enterprise plans. It’s a zone-wide setting, so you can’t enable it for only some pages. To keep pages out of the index, send an X-Robots-Tag: noindex header or a noindex meta tag.

Which one to use

• Your site is behind Cloudflare: turn on Crawler Hints. It takes a minute.
• You want control over what is submitted, or you’re not on Cloudflare: call IndexNow yourself from your deploy pipeline.

Did it work?

Bing Webmaster Tools has an IndexNow section that shows which URLs were submitted and when they were crawled. Compare the time between publishing and crawling before and after you turn it on. Ignore generic SEO metrics like bounce rate, because they won’t tell you anything about this.