After part 5, the next request was simpler to state: one workbook, lesson level, no question rows, comparing each lesson across 1403, 1404 and 1405. Part 5 had shown that N alone moves this estimator's coefficients, so the comparison had to hold N fixed. That part was achievable. Making the years comparable in the stronger sense people tend to assume was not, and this last part is about where that line falls.
An intermediate version kept the original 1405 forms from part 1 and used one N per track: 434 for Mathematics, 521 for Experimental Sciences and 1,134 for Humanities. That version is superseded, and its counts should not be paired with the final results. The current-year forms were then replaced with three later 1405 forms, one per track, and the 1403 and 1404 input snapshots were reused unchanged. A new N meant new calibrations on the old pools too; old fits made at other sample sizes were not substituted.
Equal N within a track
Official attendance on the replacement forms was 1,950 for Mathematics, 2,038 for Experimental Sciences and 2,302 for Humanities. The first two worked directly as per-lesson sample sizes. For Mathematics, for example, the three lesson pools on the new form held 1,963 to 1,965 usable people.
Humanities did not. After the valid-lesson filtering from part 2, Arabic had 2,216 usable people and history/geography only 2,198. Using 2,302 for every lesson would have meant dropping lessons or duplicating people, and both were unacceptable. I raised the problem explicitly instead of picking a size myself, and the approved size was 2,198 for all eight Humanities subjects, which keeps every subject in the comparison.
| Track | Final per-lesson sample N |
|---|---|
| Mathematics | 1,950 |
| Experimental Sciences | 2,038 |
| Humanities | 2,198 |
N is equal within a track across the three years, not across tracks. The Humanities value is the common feasible size that retains all eight lessons, and it is never relabeled as attendance.
Seventeen of thirty questions
The new Humanities literature lesson had 30 questions in its catalog. Selecting response rows whose lesson key was the main lesson found only 17 of them. The other 13 had real responses stored under subtopic lesson keys, and some questions appeared under both the main and a subtopic key. A missing-question assertion stopped preparation before any fit ran, which is exactly what it was there for.
The fix was structural:
- Build the catalog of active, nondeleted main-lesson questions from the exam's question and lesson dimensions.
- Fetch responses through the active request and question joins.
- Map each content item to its canonical main lesson among the selected lessons.
- Assert that no content item maps to more than one selected main lesson.
- For repeated copies of the same person, content and main lesson, require identical answer statuses.
- Prefer a direct main-lesson row if one exists, then collapse the consistent copies.
- Apply the valid-score check against the canonical main lesson.
- Anonymize students, then verify full question coverage, unique person/content pairs and observed responses.
The Humanities join audit recorded 599,324 mapped rows, 373,120 distinct response rows and 226,204 consistent duplicate copies collapsed. Those are response-record counts, not students or questions. No answer was fabricated or changed, and all 30 literature questions reached the final inputs. Step 5 is what separates this from ordinary deduplication on whichever row comes first: a conflicting copy would have stopped the run.
An older stored current-year output for literature had coefficients for only 18 of its 30 content items, and I recorded that as unresolved. Nothing here shows that a similar fix was applied retroactively to older stored results or exploratory reports.
The final run
The replacement study had 16 standard-track lessons per year (three in Mathematics, five in Experimental Sciences, eight in Humanities), so 48 lesson-year records and 960 requested executions at 20 replicates each. There were 941 distinct fits. The other 19 requests belonged to the new history/geography lesson, whose pool and sample size were both 2,198 with a fixed training seed. Twenty permutations cannot produce twenty different subsets when every person is selected, so its 20 requests resolved to one fit.
Final status was 960 successful requested executions, 941 distinct fits, no failures and no pending work, in about 4,795 seconds (roughly 80 minutes) of saved calibration time for this run only. All 48 input hashes were preserved. A final verifier recomputed the lesson means from the saved item arrays, checked the 96 coefficient cells in the workbook, and confirmed that the previous current-year forms were excluded.
Three charts, three different questions
The workbook has a hidden source sheet with 96 rows (48 lesson-year groups times two metrics, a and b) and one visible native Excel PivotTable with 32 lesson/metric rows and three year columns. It was built, reopened and refreshed with native Excel automation, and a separate check of the file's XML compared cached pivot values with the source. I kept it away from generic importers, which can flatten a native pivot into a static table.
The charts are where aggregation choices start to matter. Within a lesson and year, the lesson mean is the average over 20 fits of the mean coefficient across that lesson's questions. The original overview bar for a track is the mean of those lesson means, so every common lesson gets equal weight. Each lesson was fitted separately; the bar is not a pooled score, a pooled multidimensional fit, or a mean weighted by question.
| Track | N per lesson | Mean a, 1403 | Mean a, 1404 | Mean a, 1405 |
|---|---|---|---|---|
| Mathematics | 1,950 | 0.965353 | 1.009673 | 1.003773 |
| Experimental Sciences | 2,038 | 1.017133 | 1.019009 | 1.034005 |
| Humanities | 2,198 | 1.017059 | 1.037192 | 1.008924 |
These are equal-lesson means of the local narrow-prior estimator, with the replacement forms in the 1405 column. They do not answer part 1's puzzle directly, which concerned maxima on the original forms at very different sizes. What they show, descriptively, is that with N held equal within a track, the three years' overview values sit within about 0.05 of each other. On unlinked scales that closeness is not evidence that the years are equivalent, for reasons later in this part.
Histograms answered a different request: the distribution of coefficients for Humanities psychology (15 questions each year) and mathematics (20 each year). Each question contributes one value, its coefficient averaged across the 20 saved fits. The fits are not pooled as 20 independent copies of each question. Bins are shared per parameter across both lessons and all years. With 15 or 20 questions, an uneven histogram says nothing either way about whether the underlying item population is normal.
The bar charts were later converted to a four-page boxplot report: one overview page and one per track, 114 boxes in all (18 overview boxes and 96 lesson-level boxes). A lesson-level box holds that lesson's per-question 20-fit means. An overview box concatenates every question value from the common lessons of a track, one weight per question: 105 Mathematics questions, 155 Experimental Sciences questions and 160 Humanities questions per year. So the overview boxes show an all-question distribution where the old bars showed equally weighted lesson means, and a box's median should not be expected to match the old bar's height. The overview also mixes independently calibrated lessons, so it is descriptive and identifies no common latent scale for a track.
Axis limits are shared across the report and every outlier stays visible. Persian labels went through Arabic reshaping and bidirectional handling with an installed Tahoma font, and I checked the rendered pages by eye.
What a circle means
Q1 = 25th percentile Q3 = 75th percentile IQR = Q3 - Q1 lower fence = Q1 - 1.5 * IQR upper fence = Q3 + 1.5 * IQR
The box spans Q1 to Q3. Whiskers end at the most extreme observed values still inside the fences, which need not be the fences themselves, and circles mark observed values beyond the whiskers. If Q1 = 1 and Q3 = 1.4, the IQR is 0.4 and the fences are 0.4 and 2.0. A coefficient of 0.3 is outside, and if the smallest value inside is 0.6, the lower whisker stops at 0.6. The quartiles, whiskers and outlier counts were checked against independent calculations; NIST's box plot page and the Matplotlib specification describe the convention.
The 1.5 is a conventional Tukey-style exploratory multiplier. It is not an IRT parameter, a significance test, a confidence level or a pass/fail rule. A smaller multiplier circles more points and a larger one fewer. A circled question may be unusually informative, unusually easy, unsuitable, or simply different from its neighbors, and the plot cannot tell you which.
Relative information is a rank
An earlier request asked how to judge questions from existing a, b and estimated abilities, without expert scores. The diagnostic I implemented uses 2PL item information, the same quantity that drives item selection in adaptive testing (Backend Notes 2 covers that use):
P_i(theta) = sigmoid(a_i * (theta - b_i))
I_i(theta) = a_i^2 * P_i(theta) * (1 - P_i(theta))
Score_i = mean of I_i(theta_s) over the stored abilities in that lesson
percentile = (number of lower scores + 0.5 * number of tied scores)
/ number of valid questions
Abilities enter with their actual frequencies, and ties share a midrank percentile within a lesson and exam. The top quarter is labeled HIGH, the middle half MEDIUM and the bottom quarter LOW, using main lessons so that sublessons do not count the same content twice. The labels are ranks inside one lesson. The a² term cannot detect a reversed key or a negative discrimination, the abilities were estimated from the same questions, and information says nothing about factual correctness, curriculum coverage, fairness, local dependence or security. Shrinkage and scale issues feed into the calculation too. A HIGH label is a reason to look at a question first, and it certifies nothing.
Why equal N is not equating
Equal N controls one important source of estimator behavior. It does not put different questions, cohorts, curricula or years on an empirically linked scale. The 2PL scale has a free origin and unit. Under a positive linear transformation of ability:
theta_new = A * theta_old + B b_new = A * b_old + B a_new = a_old / A
Changing θ alone without transforming a and b changes the response curve. And setting each year's ability mean to 0 and SD to 1 independently is a within-group standardization, which neither proves that the groups are equal nor recovers a link between years. Each year here was calibrated on its own students. Humanities psychology's mean b went from 0.756026 in 1403 to 0.246514 in 1405. Read naively, the questions got easier. On unlinked scales, a lower b could equally reflect a stronger cohort, different questions, or both, and these data cannot separate the explanations.
The established tool is a common-item anchor: shared questions used to estimate or maintain the link. Useful anchors need adequate content and difficulty coverage, stable behavior and enough response information. The same content identifier showing up somewhere in two years does not make a linking design. We discussed whether three anchors would be enough in an 80-question test, which is 3.75% of the questions. Estimating a link from a small set is possible in principle. Three items, though, give little coverage and let one unstable item move the whole link. This project validated no universal anchor count or percentage, and anchor-based linking was discussed, never performed. The figures in this series are not equated comparisons.
Other designs came up in the same discussion, and none were implemented: a common-person design where some people take both forms (with attention to learning, order and fatigue); randomly assigned equivalent groups taking different forms under one sampling design; a connected concurrent or multiple-group calibration with defensible identification constraints; and an existing calibrated item bank whose reference scale is maintained over time. Concurrent estimation alone cannot fix disconnected data with different items and unrelated people unless more identifying assumptions are supplied, and text similarity, matching aggregate means or dividing a by sample size cannot manufacture the link. With no shared items, no common people, no equivalent-group design and no defensible external reference, the response data alone do not identify a unique transformation between exams.
NAEP's documentation describes marginal maximum likelihood item estimation with explicit choices of scale origin and unit. PISA's scale construction retains trend items, initially fixes their parameters to previous-cycle values, checks fit and releases constraints that do not hold. They illustrate what scale maintenance involves; this project reproduced neither.
What was not done
The completed work did not:
- replace the deployed estimator, revise its prior, write results back to a database, or establish the historical identity of deployed code and runtime;
- prove that every observed sample-size effect has a single cause;
- perform a formal anchor, common-person or equivalent-group link between years;
- establish a common latent dimension across the lessons of a track;
- validate every item's fit, differential item functioning, distractors, key or content coverage;
- deliver a universal good/bad rule for questions, or show that 200, 250, 434 or any other N guarantees sufficient accuracy;
- use synthetic true abilities in real-data calibrations, treat the full-population fit as ground truth, or count repeated full-pool selections as new samples;
- turn 20 sample quantiles into definitive confidence intervals.
What I would do next
These are proposals for a later validation program. None of them were carried out here.
- Define the intended measurement population and a stable key system for items, lessons and curricula.
- Store request identity, extraction timestamp, source version, model configuration and effective per-item N with every production calibration result.
- Validate priors and identification constraints with parameter-recovery experiments across plausible N, ability distributions, missingness and item ranges.
- Compare candidate estimators with repeated held-out predictive checks, calibration diagnostics, boundary behavior and interval coverage, never by which one yields larger a values.
- Review unidimensionality, local dependence, distractor patterns, possible key errors and differential item functioning.
- Design a genuine scale link before claiming any year-to-year trend in quality or difficulty.
- Report a stability or uncertainty label beside each parameter, and avoid ranking questions only by an exam's maximum a.
- Automate the existing data-quality assertions and provenance records before automating any decision about item quality.
Looking back over the six parts, the low discrimination that started this was uninterpretable as a statement about question quality until the pipeline behind it had been examined. A narrow prior labeled vague, the definition of who entered the calibration, latent-scale behavior and reporting aggregation all moved the numbers. Equal sample sizes, frozen inputs and recorded provenance made the year comparisons reproducible and more controlled. What remains open is the harder part: a formal link between years' scales, and a production estimator validated well enough to replace the current one.
This note is also available in Persian: read the Persian version.