AI Question Ingestion: from exam PDFs to structured LaTeX items

Mahdi Molavi October 4, 2026 4 min read

From January to December 2024 I worked on a remote contract for Noraco, based in Australia. I automated a pipeline that takes written exams as PDFs and turns them into structured, database-ready LaTeX items, 5,000+ questions in all. LLM APIs generate topic, skill and difficulty metadata, and a React review console covers validation, QA and publishing.

Context

Exam questions often exist only as documents: PDFs prepared for printing, full of equations, numbered parts, tables and answer options. That format works for a paper exam and poorly for almost everything else. You cannot search it by topic, filter it by difficulty, build new tests from it or display it reliably on the web.

The aim was to turn that material into a question bank. Each question becomes a separate, structured item with metadata, stored in a database and ready to be published online.

The engineering problem

Turning exam PDFs into clean items is harder than it sounds, for reasons that apply to any project of this kind:

PDF is a layout format. It records where glyphs sit on a page, not what they mean. Plain text extraction flattens mathematics, drops superscripts and fractions, and blurs the boundaries between questions.
Math needs a real representation. For exam content with formulas, LaTeX is the natural target. It is precise, it can be checked by compiling or rendering it, and web pages can display it.
Metadata does not come with the document. Topic, skill and difficulty are what make a question bank useful. Tagging thousands of items by hand is slow, and different people tag the same question differently.
Automation makes mistakes. Any automatic conversion or tagging step will sometimes be wrong. Across thousands of items, even a small error rate adds up to many bad items unless something catches them.

What I built

The ingestion pipeline. I automated the PDF-to-LaTeX conversion that turns written exams into structured, database-ready items. It produced 5,000+ questions.
LLM metadata. I integrated LLM APIs through OpenRouter to generate topic, skill and difficulty metadata for the questions.
Backend services. I delivered the services as FastAPI applications.
A review console. I delivered a React review console for LaTeX validation, QA and online publishing.

Design notes

The notes below give the general reasoning for a pipeline like this one. They are not a description of Noraco's internal setup.

LaTeX as the item format

LaTeX gives every question a text form that is exact for mathematics and still readable by people. It also offers a cheap first check: if an item's LaTeX does not compile or render, something went wrong, and the system can flag it before a person looks at it. That turns part of the question "is this conversion correct?" into a mechanical test.

It does not answer the whole question. LaTeX that renders can still say the wrong thing, so a passing check is a filter, not a substitute for review.

LLMs for metadata, treated as drafts

Labelling a question by topic, skill and difficulty is a language task, and LLMs can do it at a volume that manual tagging cannot match. But a model's labels are suggestions. Difficulty in particular is an estimate until real students have answered the question. In a pipeline like this, generated metadata is best treated as a draft that a reviewer can correct before an item goes live.

Why OpenRouter

OpenRouter puts many models from different providers behind one API. For a pipeline that runs over thousands of items, that keeps the integration code stable while the choice of model stays open, whether the reason to switch is quality on a given subject, cost or availability.

Why a review console matters

Automation handles the volume; people handle the judgment. A review console matters because conversion and tagging will sometimes be wrong, and the cheapest place to catch that is a tool built for the job, where a reviewer can check and correct an item without going back through files. Putting publishing in the same console keeps a clear gate between "converted" and "live": nothing goes online just because a script finished.

FastAPI on the backend

Document processing and LLM client libraries are well served in Python. FastAPI is a light way to expose Python code as HTTP services, with typed request and response models and generated API documentation. Those help when a review frontend and its backend are developed side by side, because the contract between them is written down in one place.

Stack

Python and FastAPI
LLM APIs via OpenRouter
LaTeX
React for the review console

Outcome

Written exams that existed only as PDFs became 5,000+ structured, database-ready questions with LLM-generated topic, skill and difficulty metadata. The React console gives the project one place for LaTeX validation, QA and online publishing.

Item banks with structured metadata are the raw material of adaptive testing in general; for that side of assessment work, see the adaptive testing platform case study. If you have a similar project in mind, see Work with me.

Frequently asked questions

What did the Noraco question ingestion pipeline do?

It turned written exams in PDF form into structured, database-ready LaTeX items: 5,000+ questions in total.

How were LLMs used in the pipeline?

Mahdi Molavi integrated LLM APIs through OpenRouter to generate topic, skill and difficulty metadata for the questions.

What was the review console for?

It is a React application for LaTeX validation, quality assurance and online publishing of the questions, backed by FastAPI services.

More in this section

Projects