Debugging is a loop: paste the error, try the fix, paste the next error. A message cap ends that loop before the bug does. Carpathian AI helps with it all.
To get your website indexed, a search engine has to find your pages, be allowed to fetch them, and decide to keep each one. Most indexing problems come from one of those three steps. A page nothing links to is never found. A page your robots.txt turns away is never fetched. A page that says noindex, points its canonical somewhere else, or returns an error page with a 200 status is fetched and then dropped.
So the work is mostly checking. Link every page from somewhere a crawler already visits, publish a sitemap, keep robots.txt from blocking what you want found, make sure each page returns its own text with a 200, and tell engines when something changes.
What's the difference between crawling and indexing?
Crawling is a search engine's program (a crawler, or bot) downloading your page. Indexing is the engine deciding to keep that page and filing its words so a search can find it. A page can be crawled and never indexed, and the fix for each is different, so it helps to know which one failed.
Think of a library. Crawling is a librarian picking up your book. Indexing is putting it on the shelf with a catalog card. A librarian can read your book and still decide it's a duplicate of one already shelved, or that the cover says it's a draft.
Every engine draws this line. A crawler downloads far more pages than the engine keeps, because some pages are copies of others, some are empty, and some ask not to be listed.
How do crawlers find your pages?
Crawlers find pages by following links: from pages they already know, from your sitemap, and from your site's feeds. A brand-new page with no link pointing to it is invisible until something names it. The fastest way to get a page found is to link it from a page the crawler already visits, like your home page or a blog index.
There are three main routes in:
Links from other sites. These also tell an engine your site matters. A site nobody links to tends to get crawled less deeply, because a crawler has finite time and spends it where the web points.
Links within your own site. Your navigation, your article index and the links inside your posts. An orphaned page, one no other page on your site links to, is the most common reason a single page never shows up.
Sitemaps and feeds. A sitemap is an XML file listing the pages you want crawled. The allows up to 50,000 URLs and 50 MB (uncompressed) per file, and a sitemap index can point to several files. An RSS or Atom feed does the same job for new posts, and a crawler that reads feeds can pick up a post from it as soon as it's listed.
A page gets crawled but not indexed when it tells the engine not to keep it, when the engine reads it as a copy of another page, or when there's nothing on it to keep. Look for a noindex, a canonical pointing elsewhere, near-duplicate text, a near-empty page, or an error that came back with a success status.
Here's what to look for, roughly in the order it bites people:
A noindex you forgot about. A staging site's <meta name="robots" content="noindex"> that made it to production, or an X-Robots-Tag: noindex header set by a plugin or a proxy. Check the page source and the response headers, since either one is enough.
A canonical pointing somewhere else.<link rel="canonical"> tells engines which address is the main copy of a page. If every page's canonical points to your home page (a common template bug), every page but the home page is read as a duplicate.
Duplicate or near-duplicate text. Print views, tag pages and pages that differ only by a tracking parameter all hold the same words. Engines keep one copy and drop the rest, and they won't always keep the one you'd pick unless the canonical says so.
Too little text. A page that is a heading and a photo has little an index can file. Pages built entirely in JavaScript can look empty to a crawler that doesn't run scripts, and not every crawler does. If your words only appear after JavaScript runs, render them on the server.
An error page that says 200. A "something went wrong" screen, a login wall or a bot-challenge page returned with a success status looks to a crawler like your page's content. Return a 404 for missing pages and a 503 for maintenance, so engines keep the version they had.
Blocked by robots.txt. A blocked page isn't fetched, so its content can't be indexed. Google's documentation adds a catch: robots.txt "is not a mechanism for keeping a web page out of Google," and a blocked page can still be listed by its address if other pages link to it (Google Search Central).
That last point trips up people trying to remove a page. To keep a page out of an index, let crawlers fetch it and mark it noindex. Google says it plainly: "For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file," because otherwise "the crawler will never see the noindex rule" (Google Search Central).
How do you tell search engines about new or changed pages?
Tell engines about changes in three ways: keep your sitemap's lastmod dates true, send changed addresses through IndexNow to the engines that support it, and use each engine's site tools to request a crawl. Crawlers revisit pages on their own schedule, so these signals shorten the wait, they don't replace it.
Accurate lastmod dates. The sitemap protocol says the date "must be set to the date the linked page was last modified, not when the sitemap is generated." A sitemap that stamps every page with today's date teaches crawlers to ignore the field, and you lose the one hint that points them at your fresh pages.
IndexNow. IndexNow is a protocol for telling search engines a URL was added, updated or deleted, the moment it happens. You prove you own the site by hosting a key file at your site's root, then post the changed addresses: up to 10,000 URLs per request, according to the IndexNow documentation. Participating engines agree that "submitted URLs will be automatically shared with all other participating search engines," so one submission reaches all of them. Many site platforms and SEO plugins send IndexNow pings for you.
Each engine's site tools. Most search engines offer a console where you prove ownership of a site, submit a sitemap, request a recrawl of one address, and see which pages were indexed and why others weren't. These are the only place you'll see an engine's own reasons for skipping a page.
How long does indexing take?
There's no fixed time. A page on a well-linked site that's in the sitemap and sent through IndexNow can be crawled within hours. A new site with no inbound links can wait weeks, because crawlers ration their visits to sites the web hasn't pointed at yet. The wait shrinks as other sites link to you.
A few things speed it up, and none of them involve tricks:
Link the new page from your home page or another page that changes often.
Get one or two links from sites in your field. A mention on a community forum, a directory for your niche, or a partner's page all count.
Keep the server fast and reliable. Crawlers back off from a site that answers slowly or with errors, and some slow down for good after a site asks them to (a 429 Too Many Requests response).
Fit matters too. If you're running a small personal site, you may never need more than a sitemap and a few good links. Search consoles and IndexNow pay off once you publish often, or once a missed page costs you something.
What's on an indexing checklist?
Work through this list for any page that isn't showing up. It runs from the problems that block the most pages to the ones that affect one page at a time, so start at the top.
Robots.txt allows the page. Open https://yoursite/robots.txt and check that no Disallow line covers the page's path for * or for the crawler you care about.
Robots.txt itself loads. Under RFC 9309, the robots.txt standard, a crawler that gets a server error (5xx) fetching robots.txt "MUST assume complete disallow." A broken robots.txt can hide an entire site.
The page returns 200 with its own content. Not a redirect chain, not a login wall, not a challenge page.
No noindex in the HTML or the response headers.
The canonical names the page itself, or is left out.
The text is in the HTML the server sends, not only added later by JavaScript.
Another page links to it, ideally one in your navigation or a recent post.
It's in your sitemap, with a true lastmod, and the sitemap is named in robots.txt.
Changes are announced through IndexNow or the engine's console.
It isn't a near copy of another page on your site. If it is, pick one address and point the other's canonical at it.
If all ten pass and a page still isn't indexed, check the engine's own console. That's where you'll find its reason.
Carpathian's Search Engine
Carpathian runs its own web search engine, Carpathian Search, with its own crawler (CarpathianBot) and index. It reads robots.txt under RFC 9309, honors noindex and canonicals, reads the sitemaps your robots.txt names, and accepts IndexNow submissions.
Search Engine Console is the site owner's side of it, free with a Carpathian account. You prove you own a site with a DNS TXT record or a meta tag, then see how your pages perform in Carpathian Search, which pages are indexed and which were skipped and why, how the crawler has been visiting, and the IndexNow submissions it received. From there you can submit a sitemap, ask for a page to be crawled again, or remove one from results.
It only reports on Carpathian Search. It can't tell you why a page is missing from any other engine's index, so keep using each engine's own tools for those. If you'd like to see your site the way Carpathian's crawler does, create an account and add Search Engine Console from Services. CarpathianBot's page explains what the crawler fetches and how to address it in robots.txt.
How to Get Your Website Indexed by Search Engines | Carpathian
