We built a tool that scores titles and headlines with an LLM. Here is how to make the model grade consistently, hold a strict format, and stop it refusing.
GeneralBy Samuel Malkasian | FounderJuly 31, 202611 min read
An AI content scoring tool takes a piece of short text, a headline, a blog title, an email subject line, and returns a graded read on how well it works alongside a few stronger rewrites. We built one called Evaluate My Title, where the platform takes user-generated content and improve it by treating the language model as a grader rather than a chatbot. Give the model fixed input, force its reply into a strict output format, and keep every number that has to be exact in plain code rather than the prompt. Built that way, the scores hold steady from one run to the next, which is the difference between a tool people trust and one whose output shifts every time they run it.
What is an AI content scoring tool?
An AI content scoring tool is software that grades short text against a goal, then explains the grade. It judges a title, an ad line, or an email subject against an outcome like clicks or opens, and returns a number from 0 to 100, a short breakdown of why, and a few suggested rewrites. It is the pattern behind headline analyzers, SEO title checkers, and writing-feedback apps. Evaluate My Title grades across relevance, engagement, and SEO, then rewrites the title two ways.
Headlines carry a lot of weight, especially when creating articles for their online presence. On average, 8 out of 10 people read the headline and only 2 read the rest, a line Copyblogger traces to David Ogilvy. Grading that one line before you publish is cheap insurance against a piece nobody opens. The tool will not write the article for you, but it tells you whether the headline earns the click.
The same design is used in tools like CoSchedule's Headline Studio and the Hemingway Editor: text goes in, a score and suggestions come out. The scoring engine underneath can be a set of rules, a machine-learning model, or an LLM. I went with an LLM because it reads a title the way a person does and can explain its reasoning in plain language, which rules-only tools cannot. Evaluate My Title is one of the projects listed in our work, and it covers blog posts, videos, podcasts, books, email subject lines, and a handful of ad and social formats, each graded on what matters for that specific format.
Why treat the LLM as a scoring engine?
Treating the model as a scoring engine rather than a chatbot comes down to control. A chatbot is open-ended: the user steers it anywhere and the shape of the output changes every turn. A scoring engine is closed. The input is fixed, a title and a topic, the job is fixed, grade it, and the output has to match one template every time. I built the tool the second way on purpose. Every request sends the same instructions and one title, and gets back the same sections in the same order. That constraint is what makes the results usable: you can parse them, render them in a consistent layout, and compare two titles fairly because they were graded the same way. When a number has to be exact, I do not ask the model for it. The app computes it in code and drops it into the reply, so the model does the judgment and the code does the arithmetic.
Written by
Samuel Malkasian | Founder
Samuel Malkasian is the founder and lead cloud architect at Carpathian, where he designed the platform's core architecture along with a range of client enterprise systems and open-source tools for AI workflows and integration. He serves as a Cyber Warfare Officer in the U.S. Army and has a background in machine learning and data science. He is currently focused on building AI infrastructure that is secure, efficient, and low-power by design.
Under the hood, each request is an OpenAI-compatible chat completion sent to a Carpathian inference endpoint. The model id is set per content type in config, so a format can point at a different model without any change to the app. I run it at a low temperature so the judgment does not swing around between runs, and I cap the output tokens so a reply cannot balloon into an essay. The engine framing keeps every one of those choices small, because there is only ever one job to do.
How do you set up a scoring persona for each content type?
A blog title and a video title get judged on different things, so each content type carries its own config. I keep one JSON file per format: blogs, videos, podcasts, books, email subject lines, and several ad and social formats. Each file holds a system prompt that defines the grader's job for that format, plus one baked-in example, a sample title in and a fully formatted score out. That example does quiet work as a few-shot demonstration, so the model copies the shape of a good answer instead of guessing at it. The blog config rewards a number and a clear benefit, while the video config rewards curiosity and a strong hook, and the engine underneath both is identical. At request time the app loads the file that matches the content type, appends the user's title to the messages, and sends it. Adding a new content type is a new JSON file rather than new code, which is the part of the design most worth copying.
The system prompt pins an exact output format, and the baked-in example shows the model what that looks like:
Nothing about the format is left to chance, because the worked example already showed the model what a correct answer looks like. Because the persona lives outside the code, tuning a grader means editing text, not shipping a deploy.
How do you get consistent scores from an LLM?
Consistency comes from splitting the work: the model judges, and the code owns every number that has to be stable. Language models are shaky at arithmetic. Ask a small model for a readability score or a keyword-density percentage and it returns a plausible number that changes on the next run. So the tool does not ask. The model returns the qualitative parts, the 0-to-100 judgment, the reasons, and the rewrites, and then the server computes the exact metrics, Flesch reading ease, keyword density, and word count, and appends them to the reply. The prompt tells the model not to output those numbers at all, because the app adds them. This is hybrid scoring, and it is the single most important choice in the tool. A qualitative score can wobble a few points and stay useful, but a readability number that jumps from 62 to 78 on the same title is broken. Pinning the hard numbers to a formula in code makes them come out the same every time, for every user.
The readability figure is plain Flesch reading ease: 206.835, minus 1.015 times words per sentence, minus 84.6 times syllables per word, clamped between 0 and 100, with a small heuristic for counting syllables. Keyword density is the top few content words, stopwords removed, as a share of the total. These attach only to the title-scoring tools, the blog, book, podcast, and video graders, where the thing being measured is a single title. The ad and social tools return freeform ideas, and a keyword count over freeform text would mislead more than it helps, so those skip the metrics block. A low temperature handles the rest on the judgment side, keeping repeated runs close together.
How do you stop the model from refusing or rambling?
Two failure modes show up with small models, one that refuses a fine input and one that pads the answer, and both get handled in the prompt. Small models tend to hedge: give one a blank topic and it will often refuse, ask a clarifying question, or open with three paragraphs of preamble, none of which belongs in a scoring tool. A few blunt prompt rules fix it. A refuse gate uses a sentinel: when the input clearly is not a title, the model replies with a single line starting INVALID, and the app catches that word and shows a clean message. An explicit rule tells the model never to refuse just because the topic is missing, but to infer it from the title and grade anyway. A hard instruction to answer in the exact format and nothing else kills the rambling. The API call also caps the response length, so even a chatty reply cannot run long. The prompt does most of the work, and the token cap is the backstop.
On the way back, the app inspects the start of the reply. If it begins with INVALID, the response is treated as a decline, the sentinel word is stripped, and the user sees a short human message instead of a broken score. Older phrase-matching stays in as a fallback for models that ignore the sentinel rule. One more wrinkle: some backend models are reasoning models that emit a hidden thinking section before the answer. I ask the model to skip that step for a scoring task and strip any that slips through, so the user only ever sees the finished grade.
How do you handle free users and paid speed?
Two things keep a public AI tool from being overrun: a usage cap and a faster path for paying users. Both run off the plan. Every visitor, including one who never signs up, becomes a guest with a small daily quota tracked in the database and reset each day. Guests get a couple of scores to try it, registered free users get more, and paid plans get considerably more. When the quota reaches zero, the app returns a clean out-of-tokens message instead of burning compute on abuse. Speed is the second piece. Free and guest traffic goes to a standard inference instance, and paid plans route to a separate, faster one. The request shape and the code path are identical, with one branch that picks the endpoint and key based on the plan. That split is what lets a free tier exist without starving the people who pay for it.
The quota lives in a small usage table keyed to the user or guest, with a daily reset, so nobody has to sign up to try the tool and nobody can hammer it for free forever. The paid route is a single conditional: a paid plan uses the fast endpoint and key, everything else uses the standard pair. The tool has to stay cheap to run and quick for paying users, and that one branch covers both.
When does LLM scoring fall short?
LLM scoring is strong at the kind of judgment a person would also make quickly: is this headline clear, does it promise something, does it read well. It is weak anywhere the truth is fixed and checkable. If a rule can answer the question, the rule beats the model every time, and it is faster and free. Character counts, keyword presence, and reading ease all belong in code, which is exactly why they moved out of the prompt. The model is also no authority on what will get clicks; it reflects patterns in its training data, not your audience, and two runs on a borderline title can land a few points apart. So the number is a strong hint, not a verdict, and the parts that must be exact stay far from the model. Use it to compare and improve, not to certify.
A quick check before opting to use an LLM: if you can write the check as an if-statement, write the if-statement. Save the model for the calls a rule cannot make, like whether a title sounds dull or overpromises. That line, judgment on one side and fixed facts on the other, is the seam the whole design is built around.
We use the same concepts across other AI features we've implemented. Pulling clean fields out of a messy document and structured output from a model that would rather chat are covered in how to extract structured data from images. So is adding AI content moderation, where the model classifies and the code enforces.
Applying the pattern to a problem of your own is the kind of work we do at Carpathian (AI application development). If you need help with developing, troubleshooting, or implementing, our architects are available to help you build AI into your applications.
How to build an AI that ranks content appeal | Carpathian
