How to use AI to pull data from images and PDFs

Pulling typed fields out of a scanned invoice, a photographed receipt, or a decade-old PDF is a two-stage job, and keeping the two stages apart is what makes it dependable. The first stage is optical character recognition, or OCR: software reads the pixels of an image and returns the text it can make out. The second stage hands that text to a language model with a schema and asks for named fields, the invoice date, the total, each line item, shaped as JSON your application can store. OCR turns the pixels of an image into text; a language model turns that text into the fields your app keeps. Neither step does the other's job well, so you build and test them separately.

The document pipeline behind Carpathian's AI assistants already does the first stage for you, which matters when you are assembling your own retrieval-augmented generation (RAG) setup. A user uploads a PDF or a photo, and it comes back as text a model can read. This guide covers how that extraction works, how to turn the resulting text into typed fields, how to catch a bad read before it reaches your database, and when a vision-capable model should be used over plain OCR. For the wider picture on our AI work, start with the AI overview and our notes on AI application development.

How does OCR turn an image into text?

OCR reads the shapes in an image and decides which characters they are. You hand it a photo, a scan, or a rasterized PDF page, and it returns the text it found in roughly reading order. Carpathian's pipeline uses Tesseract, a open-source OCR engine that runs as a local binary rather than a hosted call.

In code, an uploaded image is opened with Pillow, converted to grayscale or RGB, and passed to Tesseract, which returns a string. PDFs get handled differently. I read each page's built-in text layer first, because a digital PDF already carries its text, and running OCR over it would only add error to something already exact. A page falls through to OCR only when its text layer is nearly empty, under roughly twenty characters, which is the signal that it is a scan rather than digital text.

When a page is image-only, I reccomend rendering it to a PNG at 300 dots per inch (DPI) before OCR runs. Resolution drives accuracy more than any other single factor, and Tesseract does far better on a clean 300 DPI render than on a small, blurry one. For pages that do carry a text layer, pull the text as positioned blocks and sort them top to bottom, left to right. That ordering keeps multi-column pages and slide decks readable instead of interleaving figure labels into the middle of a sentence.

Raw OCR output carries noise, and it needs cleaning before anything downstream reads it. Cleaning means stripping the font glyphs that surface as stray bullets and checkboxes, dropping lines that are nothing but rows of dashes or equals signs, and removing the headers and footers that repeat on every page. Left in, that boilerplate buries the content you care about and spends tokens the model then has to process.

Why run OCR locally instead of sending files to an API?

Running OCR on our hardware means the document never leaves it. That is the main reason the pipeline keeps Tesseract local rather than calling a hosted 3rd party OCR service. When someone uploads a contract, a medical form, or an ID, the pixels become text on Carpathian's own US-based infrastructure, and the file stays there.

A hosted OCR API can be accurate and quick to start with, and it is a fine choice when the data is not sensitive. For regulated or private records, every page you send is a page that has left your control, and local extraction sidesteps that. It also carries no per-page fee, so a batch of ten thousand scanned pages costs compute time rather than a metered bill. The tradeoff is ownership: you run the setup, the tuning, and the CPU it burns.

OCR is CPU-bound, which makes it easy to swamp a server if every upload runs at once. Each Tesseract call is pinned to a single thread, the number of documents extracting in parallel is capped, and short-lived locks make sure a crashed job frees its slot instead of holding it. Extraction runs as a background job rather than inline with the upload request, so a large scan never blocks the page the user is looking at.

A few limits exist for safety rather than speed. A crafted PDF can declare an enormous page size that would exhaust memory the moment you rasterize it, so the render caps the pixel area of any page and refuses documents past a page-count ceiling. Treat every uploaded file as hostile until you have bounded what it can cost you.

How do you extract structured data from images into clean JSON?

OCR gives you text, not fields. To get from a wall of characters to typed values your application can store, send the OCR text to a language model with a schema you define and ask it to fill that schema. Most model APIs support a JSON or function-calling mode that constrains the reply to the shape you specify, so you receive structured data instead of a paragraph of prose.

Carpathian exposes its models through an OpenAI-compatible inference API, so the extraction call looks the same whether it points at a Carpathian-hosted model or a provider you bring, such as OpenAI or Anthropic. You describe the fields you want, an invoice number as a string, a date in ISO format, a total as a number, a list of line items, and the model reads the messy text and returns them. This is the step where you extract structured data from images into values code can act on, and it stays clean only because the OCR stage handed the model text instead of raw pixels. Keep the two jobs separate and each one gets easier to test and repair.

{
  "invoice_number": "string",
  "invoice_date": "YYYY-MM-DD",
  "currency": "string",
  "total": "number",
  "line_items": [
    { "description": "string", "quantity": "number", "amount": "number" }
  ]
}

Define the schema once, then ask the model to return exactly that shape from the OCR text. The same pattern carries over to receipts, purchase orders, lab results, shipping labels, and forms, with only the field list changing.

To keep the output reliable, give the model the cleaned OCR text rather than the raw dump, so repeated headers do not distract it. And keep the prompt narrow: ask for the listed fields and nothing else, with an explicit instruction to leave a field null when the document does not contain it. A model told to guess will guess, so instruct it to return null when it is unsure, and it hands you a gap you can flag rather than a fabricated value.

If your task is scoring or classifying rather than pulling fields, the same structured-output approach applies. I walked through that shape in how to build an AI content scoring tool.

Inside our assistant the extracted text feeds the model along a different path, because the goal there is answering questions rather than filling a form. The cleaned text is split into chunks, each chunk becomes an embedding vector, and at question time the closest chunks are pulled back and placed in the model's prompt with a page citation. That retrieval step is its own subject, and I covered the mechanics in how to build semantic search with embeddings.

What do you do with a bad OCR read?

Assume some reads will be wrong and design for it. OCR is imperfect, and a language model will happily turn garbled text into confident, wrong fields. Validation is the fix. Once the model returns JSON, check it against your schema before you trust it: is the date a date, is the total a number, do the line items sum to the total. When a check fails, you do not write the row, you retry or route the document to a person. Keep a link back to the source page or image on every extraction, so a reviewer can glance at the original in seconds instead of hunting for it. You cannot make OCR flawless, so the aim is to catch bad reads cheaply, before they reach your database.

The hard cases are predictable. Handwriting is the largest one, and Tesseract is weak on it. Faxes, photos taken at an angle, low-contrast scans, and unusual fonts all drag accuracy down as well. In one published benchmark from AIMultiple, Tesseract topped 95% accuracy on printed documents once handwriting was removed from the test set, across a 300-document set (AIMultiple). Printed text is where classic OCR is strong; handwriting and messy captures are where it struggles.

So watch for the signs of a bad read. Very short output from a page that clearly held content, a wall of nonsense characters, or a model that returns nearly every field as null usually points to OCR failing upstream rather than the model. Score confidence where you can, and treat a low score as a reason to send the page for review instead of storing its output. Checking a hundred flagged pages costs far less than unwinding a thousand wrong records later.

When should you use a vision model instead of Tesseract?

Reach for a vision-capable model when the document is more than clean printed text. A multimodal model reads the image directly and handles handwriting, complex tables, stamps, and photos far better than classic OCR, and because it understands layout, it can tell a total from a subtotal by where each sits on the page. That capability costs more per page and in latency, and it sends the image to whichever service runs the model.

For clean, printed, high-volume documents, plain Tesseract is usually the better call: it is fast, it carries no per-page fee, and it runs locally. For a stack of handwritten forms or phone photos of receipts, a vision model saves you the accuracy fight. Many teams run both, cheap local OCR for the easy majority and a vision model only for the pages that fail a confidence check. Because the extraction call already goes through an OpenAI-compatible API, pointing the hard pages at a multimodal model is a routing decision rather than a rebuild.

To optomize yout cost, you can try sending everything through local OCR first, validate the result, and escalate only the pages that fail, the low-confidence reads and the ones your schema check rejects, to the more expensive vision path. Most document sets are mostly clean, so the majority never touches the pricier model and your bill tracks the hard cases rather than the whole pile.

A few things that might push you toward a vision model earlier are heavy handwriting, dense tables where column alignment carries meaning, mixed languages on one page, or layouts where position matters as much as the words. A few things keep you on Tesseract: privacy rules that forbid sending files out, high volume where per-page fees accumulate, and clean digital PDFs where you should be reading the text layer and skipping OCR anyway.

How to build a document pipeline you can trust

A working version of this can be genrally built in a few days and the pieces are small, and each one is testable on its own.

1. Take the upload and check its type and size before you touch it. Reject anything you do not handle, and cap the byte size and page count.
2. For PDFs, read the text layer first, and OCR only the pages that come back nearly empty. For images, open or render at roughly 300 DPI and OCR directly.
3. Clean the text. Strip repeated headers and footers, separator lines, and stray glyphs before anything downstream sees it.
4. Send the cleaned text to a model with a schema and ask for JSON. Tell it to return null for anything the document does not contain.
5. Validate the JSON against the schema. Store the clean rows, and flag the rest for a person with a link back to the source page.
6. Watch your confidence signals, and escalate only the failures to a vision model if you use one.

Keep extraction and understanding as two separate steps. When OCR and the model are tangled together, a wrong field is hard to trace. When they are split, you can read the text between them and see exactly where it went wrong. Get clean text out of the document, then let the model turn it into fields, and the rest follows from those two.

Good document handling is the kind you stop noticing. It reads the easy pages for almost nothing, quietly flags the hard ones for a human, and never lets a bad scan become a bad record. If you need help setting up your own local OCR model, you can contact one of our architects or use Carpathian's models.