CarpathianBot is the web crawler that builds the index behind Carpathian Search at search.carpathian.ai. If you've found it in your server logs, it identifies itself with this user agent:
It fetches public pages so they can appear in search results. It makes one request at a time to your site, waits at least 1.5 seconds between requests, follows robots.txt, and never submits forms, logs in, or runs JavaScript. You can block it, slow it down, or keep your pages in search while opting them out of AI training, all from robots.txt.
The rest of this guide covers how CarpathianBot crawls, which rules it reads, how it finds your pages, what it keeps from each one, and how Carpathian Search decides the order of results. If you're comparing search engines rather than managing a site, Independent Search Engines: Which Have Their Own Index? explains why an engine with its own crawler matters at all.
What is CarpathianBot?
CarpathianBot is the crawler for Carpathian Search, a web search engine that runs its own index instead of showing results licensed from another engine. A crawler is a program that downloads web pages and follows the links it finds. Everything Carpathian Search returns was fetched by CarpathianBot first, so a page it never reaches can't appear in results.
The bot's own page at search.carpathian.ai/bot is the reference for its behavior, and the user agent string links there so anyone reading a log can check what it is. Seeing it in your logs means your site is being added to, or refreshed in, the index.
How does CarpathianBot crawl a site?
CarpathianBot crawls politely by default: one connection per host, at least 1.5 seconds between requests, and longer if your robots.txt asks for it. It backs off when a site struggles, and it refetches pages on a schedule that follows how often each one changes, so a page that never changes is fetched less over time.
What that looks like from your server:
One request at a time. CarpathianBot never opens several connections to the same host at once.
A minimum gap. Requests are at least 1.5 seconds apart, or your Crawl-delay value if that's longer.
Backoff on errors. A 429 or 5xx response slows it down exponentially, and a Retry-After header is honored.
Conditional requests. It asks whether a page changed since the last fetch, so an unchanged page answers with a 304 and no body.
Adaptive refetching. Pages that change often are checked more often, and pages that don't are checked less.
It also has a short list of things it won't do. It doesn't submit forms, log in, execute JavaScript, or request private or reserved IP ranges, and it never fetches past your robots.txt rules or crawl delay, even for a URL someone submitted.
Which robots.txt rules does CarpathianBot follow?
CarpathianBot reads robots.txt under RFC 9309, the Robots Exclusion Protocol standard published in September 2022, and caches it for 24 hours. A group addressed to CarpathianBot is followed exactly. Without one, it follows your Googlebot group, then Bingbot, then *, the same fallback Applebot uses.
That fallback means most sites need no change at all. If you've already written rules for Google, CarpathianBot obeys them. The standard itself sets the caching limit, saying crawlers "SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable," so a change you make today reaches CarpathianBot within a day.
Crawl-delay is read from whichever group CarpathianBot ends up following. To give it rules of its own, add a group with its name:
If your robots.txt can't be reached, CarpathianBot uses the last copy it fetched successfully. If it has never had a copy, it treats the site as allowed only after the file has been unreachable for a day on a site verified in Search Engine Console, or 30 days on any other site, and only while your front page answers.
How do you block CarpathianBot or opt out of AI training?
To keep your whole site out of Carpathian Search, disallow CarpathianBot in robots.txt. To stay in search but keep your pages out of AI training datasets built from the crawl, disallow Carpathian-AI instead. The two are separate, so blocking one doesn't affect the other.
Out of search entirely:
User-agent: CarpathianBot
Disallow: /
In search, out of training:
User-agent: Carpathian-AI
Disallow: /
The opt-out page lists the other signals CarpathianBot treats as an AI training opt-out, so a site that already uses one of them doesn't have to add anything:
A robots.txt rule blocking any named AI crawler.
noai or noimageai in a robots meta tag.
The TDM-Reservation: 1 header.
For a single page, a noindex robots meta tag or X-Robots-Tag header works too. CarpathianBot still fetches the page, reads the tag, and leaves it out of the index. A page that's removed, disallowed, or marked noindex drops out of the index on its next fetch.
How does CarpathianBot find your pages?
CarpathianBot finds pages through the sitemaps your robots.txt declares, through links from other sites, through sitemaps and sites submitted on its sitemap page, and through IndexNow notices your site sends. Submitting gets a site fetched quickly, but how much of it gets crawled depends on the links pointing at it.
Declare your sitemap in robots.txt with one line:
Sitemap: https://example.com/sitemap.xml
If there's no Sitemap line, CarpathianBot requests /sitemap.xml on its own. It reads sitemap files and sitemap indexes, plain or gzipped, up to 50,000 URLs per file and 1,000 sitemap files per host. It ignores URLs on other sites, and it uses lastmod to refetch changed pages sooner. RSS and Atom feeds your pages advertise are read the same way.
A submission on the sitemap page is queued within a minute. CarpathianBot fetches robots.txt first, then your front page, the submitted sitemap, and any sitemaps robots.txt declares. Submissions are limited to 5 an hour per IP address and one per site per day.
For pages that change often, IndexNow is faster. Carpathian Search accepts IndexNow submissions at https://search.carpathian.ai/indexnow, either a single URL as a GET or up to 10,000 URLs in one POST. The crawler reads your key file from your site before it queues anything, so a submission only counts when the key matches.
The sitemap checklist covers the mistakes that keep pages out: URLs that redirect, URLs marked noindex, URLs robots.txt disallows, and a lastmod that changes when the content didn't.
What does CarpathianBot index, and what does it skip?
CarpathianBot indexes HTML and plain-text pages up to 5 MB: the main text, title, published date, outbound links, the page type from structured data, and the anchor text other sites use to link to the page. It skips anything rendered by JavaScript, because it doesn't run scripts.
That last point is the one most likely to affect you. A site that sends an empty HTML shell and builds its content in the browser looks blank to CarpathianBot, and it won't rank for anything. Server-rendered or static pages are indexed in full.
Pages marked noindex, and pages whose canonical URL points to another page on the same site.
Duplicates. Of several pages with identical or near-identical text, one is indexed.
Links marked nofollow, sponsored, or ugc, which aren't followed for discovery.
Endless URL spaces, such as calendars and filter combinations, past a fixed number of URLs per pattern.
The index is also younger and smaller than those of the largest engines, so a new or very small site may wait longer to be found than it would on Google or Bing. Links from other sites are what move that along.
How does Carpathian Search rank results?
Carpathian Search ranks by relevance first. Each page is scored on how well its body, title, and inbound anchor text match your query, blended with how close the page is in meaning. Quality signals, such as ad load, trackers, and authorship, only reorder pages that are already about equally relevant.
The relevance score uses BM25F, a standard text-matching formula that weighs a word more when few pages in the index contain it and when it appears in an important field like the title. Carpathian's own language model adds a meaning score, so a page that says the same thing in different words still matches.
Among pages of similar relevance, the ranking page lists what moves one above another. The ones a site owner can most directly affect:
Cleanliness. No ad networks, popups, cookie walls, trackers, or affiliate links. The ad and tracker penalties page lists what's detected.
Weight. More of the page is text and less is scripts and markup.
Integrity. No affiliate listicles, generated filler, keyword stuffing, or a company's own "best of" marketing.
Authorship. An author, a date, a feed, or an about page.
Citations. Outbound links from the body text, weighted toward reference sources.
Authority. The number and weight of distinct sites linking in.
Title match. Query words in the title, URL, and headings.
Weights also change with the kind of query. A question favors pages where the search terms sit close together, a navigational search puts the named site's front page first, and freshness only counts for searches about recent events. Queries with words like "best," "vs," or "review" raise first-hand accounts and discussion threads and penalize affiliate content harder.
Readers count too. Pages that people's searches end on rise for that query, and pages they skip or return from quickly fall, though it takes many different people before the signal counts. None of these signals can lift an irrelevant page above a relevant one.
How do you see what CarpathianBot found on your site?
Verify your site in Search Engine Console to see which of your pages Carpathian Search has indexed, the searches they appear for, and their impressions, clicks, and average position by page, country, and device. From there you can submit a sitemap, queue a recrawl, or remove a page from results.
Search Engine Console has no charge. It shows counts only, never who searched. Every result on the search page also carries a "Why here" link with its per-signal breakdown, which shows how one of your pages scored against a query without verifying anything.