Rankelle-Audit only visits a site when somebody with access to that site asks us to audit it, or lists it as a competitor to watch. It does not crawl the web at large and it has no index. If you did not expect it, the block instructions are two sections down and we would rather you used them than wondered.
What it is
The full user agent string is Rankelle-Audit/1.0 (+https://www.rankelle.com/bot).
It fetches pages, reads the HTML, and reports what is broken: dead links, redirect chains, duplicate titles, pages missing from your sitemap, and about seventy other things. The findings go to whoever asked for the audit. We are not a search engine, we do not rank anything, and nothing it reads is published.
It does not execute JavaScript, load images, or request anything other than HTML, robots.txt and XML sitemaps.
How it behaves
- It obeys robots.txtFetched once at the start of every crawl, parsed to RFC 9309, and honoured. We read up to 500 KiB of it, which is the minimum the RFC requires parsers to handle.
- Between 5 and 20 requests at a timeIt starts at ten and adjusts. Responses slower than 10 seconds make it back off; consistently fast ones let it rise, to a ceiling of twenty. It also holds a total bandwidth ceiling across everything in flight, so a site of large pages gets fewer simultaneous requests rather than the same number of bigger ones.
- A ceiling on pages2,000 per crawl by default, and the person who asked for the audit can set it lower. It stops there whether or not it has finished.
- Not continuouslyA crawl runs when a site is added, when its owner asks for one, when their deploy hook fires, and as a safety net for any site nothing has looked at in 30 days. There is no constant background crawling.
- It does not keep your textPage bodies are not stored. What is kept is structural: titles, headings, status codes, link targets, word counts and a content hash used to notice that a page changed. That is a deliberate limit, and it means we can report that a page does not target a topic, and can never claim to know what it discusses.
How to block it
No appeal, no contact and no account needed. Blocking takes effect on the next crawl.
User-agent: Rankelle-Audit Disallow: /Block it entirely
Add this to your robots.txt. That is the whole process.
User-agent: Rankelle-Audit Disallow: /admin/Keep it out of one area
The same directive with a narrower path. Worth knowing that a crawler which finds nothing linked from your own pages will not reach them anyway.
HTTP/1.1 429 Too Many RequestsSlow it down instead
Crawl-delay is not part of RFC 9309 and we do not read it. Returning 429 or 503 is the mechanism that works: the crawler treats a slow or failing origin as a reason to cut concurrency.
Common questions
Why is it crawling my site when I never signed up?
Someone with a Rankelle account added your site, either because they work on it, or as a competitor they are tracking. We do not verify the first case, which is why blocking it is deliberately a one-line change you can make without talking to us.
Will blocking it affect my Google rankings?
No. Rankelle-Audit is unrelated to Google, Bing or any search engine. Blocking it changes nothing about how you are indexed or ranked anywhere.
Is it the same as the agent that reads AI answers?
No. Rankelle-Audit only fetches pages. Measuring whether AI answer engines mention a brand is done by asking those engines, not by crawling anybody.
Do you publish what you find?
No. Findings go to the account that requested the audit. There is no public index, no directory and no database of sites we have crawled.
Can I see what it saw?
If you are the site owner, add your own site and run a crawl: you get the full list. If you are not, the account that ran it owns those findings and we will not share them.
Try it on your own site, free
The whole audit of one website, no card and no expiry. Pay only when you want the proof.