Backend Notes 4: PDF to LaTeX with an LLM in the Loop

Exam PDFs are made for printing. A page looks clean to a student, but to a program it is a scatter of glyphs, line fragments and images with no idea where one question ends and the next begins. A question bank needs each item as a record: stem, options, answer, figures, and metadata such as topic, skill and difficulty, so that items can be searched, assembled into tests and, later, calibrated.

In 2024, on a remote contract with Noraco, I automated a pipeline that turned written exams into structured, database-ready items, more than 5,000 questions in total. It converted PDFs to LaTeX, used LLM APIs through OpenRouter to generate topic, skill and difficulty metadata, and ended in FastAPI services and a React review console for LaTeX validation, QA and publishing. This note describes how a pipeline like that is put together, how I think about the tagging step, and why the human review step is not optional.

The stages

1. Ingest. Store the original file, hash it to catch duplicates, and record where it came from: which exam, which year, which page.
2. Extract. Get text and math out of each page, and render every page as an image regardless, because reviewers will need it.
3. Segment. Split pages into items and attach what belongs to each one: figures, shared reading passages, answer options.
4. Convert. Turn each item into LaTeX using a small, agreed set of packages and macros.
5. Validate. Compile every item on its own and record the result.
6. Tag. Ask an LLM for topic, skill and difficulty, constrained to a controlled vocabulary.
7. Review and publish. A person checks the item against the source and approves, edits or rejects it.

Each stage writes its output and status to the database, keyed by item and stage version. A failure in tagging does not redo extraction, and when the logic of one stage improves you can rerun just that stage over the items it affects. With thousands of items, that property matters more than the speed of any single step.

Extraction is where the errors start

PDFs come in two kinds. Born-digital files have a text layer, and libraries such as PyMuPDF will hand you the text. Math is the problem: fractions come out as separate lines, superscripts flatten into the baseline, and symbol fonts map to the wrong characters. Scanned files have no text layer at all and need OCR, ideally a math-aware model that outputs LaTeX directly.

Neither route is reliable enough to trust blindly. The rule I would not give up is to keep a crop of the original page region next to every extracted item, all the way through the pipeline. Every later decision, by a script or a person, can then be checked against what the exam actually printed.

Validation by compiling

The cheapest useful test of a LaTeX fragment is to compile it. Wrap the item in a minimal document, run the compiler with a timeout, and keep the error lines from the log:

import subprocess
import tempfile
from pathlib import Path

TEMPLATE = r"""\documentclass{article}
\usepackage{amsmath,amssymb}
\begin{document}
%s
\end{document}
"""

def compile_item(body: str, timeout_s: int = 20) -> tuple[bool, str]:
    """Compile one item in isolation; return (ok, first error lines)."""
    with tempfile.TemporaryDirectory() as tmp:
        Path(tmp, "item.tex").write_text(TEMPLATE % body, encoding="utf-8")
        try:
            subprocess.run(
                ["pdflatex", "-interaction=nonstopmode", "-halt-on-error", "-no-shell-escape", "item.tex"],
                cwd=tmp, capture_output=True, timeout=timeout_s, check=False,
            )
        except subprocess.TimeoutExpired:
            return False, "timeout"
        log_path = Path(tmp, "item.log")
        log = log_path.read_text(encoding="utf-8", errors="replace") if log_path.exists() else ""
        errors = [line for line in log.splitlines() if line.startswith("!")]
        return Path(tmp, "item.pdf").exists() and not errors, "\n".join(errors[:5])

This catches unbalanced braces, unknown macros, missing math delimiters and packages outside the agreed set. It does not catch meaning. A dropped minus sign, an exponent attached to the wrong term or two options in swapped order all compile perfectly. A passing compile only tells you the item is not broken.

Treat the input as untrusted. -no-shell-escape stops the document from running commands, but LaTeX can still read files on the machine, so the compile step belongs in a locked-down container without network access, with a timeout so a runaway macro cannot hold a worker forever. If your items use non-Latin scripts, XeLaTeX or LuaLaTeX is the better engine.

Tagging with an LLM: schema first

Tags are only useful if they come from a fixed vocabulary. A model asked "what topic is this?" in free text will invent a new label every few hundred items, and a label that exists only once is worthless for search or test assembly. So the taxonomy lives in the database, and the prompt and the output schema are both generated from it.

OpenRouter exposes an OpenAI-compatible API, so the standard openai Python client works with a different base URL. For models that support it, response_format with a JSON schema constrains the output. OpenRouter's documentation is clear that enforcement varies by provider, so the output is validated again in code. A JSON schema can say that skill is one of a list of strings; the rule that the skill must belong to the chosen topic is easier to enforce with Pydantic:

import json
import os
from typing import Literal

from openai import OpenAI
from pydantic import BaseModel, Field, ValidationError, model_validator

client = OpenAI(base_url="https://openrouter.ai/api/v1", api_key=os.environ["OPENROUTER_API_KEY"])

TAXONOMY = {                      # loaded from the database in the real service
    "algebra": ["linear-equations", "quadratics", "inequalities"],
    "geometry": ["triangles", "circles", "coordinate-geometry"],
}

SYSTEM_PROMPT = (
    "You classify exam questions. Choose topic and skill only from this taxonomy:\n"
    + json.dumps(TAXONOMY, indent=2)
    + "\nDifficulty is your estimate for the target students: easy, medium or hard."
    " If no skill fits well, choose the closest one and set needs_review to true."
    " Keep the rationale to two sentences."
)

SCHEMA = {
    "type": "object",
    "properties": {
        "topic": {"type": "string", "enum": sorted(TAXONOMY)},
        "skill": {"type": "string", "enum": sorted({s for v in TAXONOMY.values() for s in v})},
        "difficulty": {"type": "string", "enum": ["easy", "medium", "hard"]},
        "needs_review": {"type": "boolean"},
        "rationale": {"type": "string"},
    },
    "required": ["topic", "skill", "difficulty", "needs_review", "rationale"],
    "additionalProperties": False,
}

class ItemTags(BaseModel):
    topic: str
    skill: str
    difficulty: Literal["easy", "medium", "hard"]
    needs_review: bool
    rationale: str = Field(max_length=500)

    @model_validator(mode="after")
    def skill_belongs_to_topic(self):
        if self.skill not in TAXONOMY.get(self.topic, []):
            raise ValueError(f"skill '{self.skill}' is not under topic '{self.topic}'")
        return self

def tag_item(item_latex: str, model: str) -> ItemTags | None:
    messages = [{"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": item_latex}]
    for _ in range(2):                                   # one retry, showing the model its error
        resp = client.chat.completions.create(
            model=model,
            temperature=0,
            messages=messages,
            response_format={"type": "json_schema",
                             "json_schema": {"name": "item_tags", "strict": True, "schema": SCHEMA}},
        )
        raw = resp.choices[0].message.content or ""
        try:
            return ItemTags.model_validate_json(raw)
        except ValidationError as err:
            messages += [{"role": "assistant", "content": raw},
                         {"role": "user", "content": f"Invalid output: {err}. Answer again with valid JSON."}]
    return None                                          # goes to the review queue untagged

A few design choices are deliberate. The model gets an explicit way out, needs_review, instead of being forced to pretend that every item fits. Self-reported uncertainty is poorly calibrated, so that flag can add items to the review queue but never remove them from it. The rationale is short, and it is shown to the reviewer next to the tags. Temperature 0 reduces variation but does not make output deterministic, so every result is stored with the model id, the prompt version and the raw response. When either the model or the prompt changes, rerun a fixed set of already reviewed items and compare before switching.

Where models go wrong

None of these are exotic; they follow from what the model can and cannot see.

• Difficulty. A model's "hard" is not your students' "hard". Real difficulty comes from response data, the b parameter from the adaptive testing note. An LLM estimate is a reasonable prior for assembling early tests and a useful sort key for reviewers, and it should be replaced by calibration as soon as responses exist.
• Neighbouring categories. An inequality solved through a quadratic could sit under either skill. If two teachers would disagree about an item, the model will be inconsistent about it too. Fixing the taxonomy, with clear definitions and examples, helps more than rewording the prompt.
• Missing context. An item that refers to a figure, a shared passage or "your answer to question 3" cannot be tagged, or even used, on its own. That is a segmentation problem showing up in the tagging step.
• Plausible but wrong conversion. Models are good at producing LaTeX that looks right. When the OCR or the model quietly "fixes" notation, the result compiles and reads well, and only a comparison with the source catches it.

The review console

This is why the last stage is a person, and why the console deserves real engineering. A good review screen comes down to four things: the original page crop on one side, the rendered LaTeX on the other, an editor for the LaTeX with its compile status, and the proposed tags with the model's rationale. The actions are approve, edit or reject.

Behind the screen, a few rules keep the bank trustworthy:

• Every item has an explicit state, such as extracted, validated, tagged, in review, approved and published, and the API only allows legal transitions.
• An edit triggers validation again, and approval requires a passing compile.
• Publishing writes a new version of the item with the reviewer's name, so the bank has an audit trail.
• The queue is ordered: compile failures and needs_review items first, then a random sample of the confident ones. Sampling the confident items is how you find systematic errors, the kind a model makes with full conviction.
• Slow work, such as compiling and calling the LLM, runs in background workers, not inside the request, so the console stays quick while a new batch is processing.

In this pipeline the LLM moves the reviewer's job from typing metadata to checking it, which is faster and less tedious, as long as the checking is real. If you have a pile of documents that needs to become structured data with a model in the loop, that is work I take on. The next note is a personal one: what moving from tensor research to backend engineering taught me.

This note is also available in Persian: read the Persian version.