WEIAN DATA 惟安数据科技
HOME / METHODS

The methods behind large assessments,
in plain language

So you know exactly what you're buying: what each method solves, why it matters, and what goes wrong when it's done carelessly.

§1 CTT vs. IRT §2 Model choice §3 Equating §4 Weighting §5 Unit merging §6 DIF
§ 1 — FOUNDATIONS

What's the difference between CTT and IRT?

Classical test theory (CTT) treats the total score as the ability. Simple and direct — but the result is tied to the specific test form: change the items, and scores are no longer comparable.

IRT (item response theory) models each item's difficulty and discrimination separately, converting responses into ability estimates that don't depend on the particular item mix. Cross-form comparison, item banking, and adaptive testing all become possible — which is why PISA and national literacy surveys are built on IRT.

How WEIAN does it: our in-house IRTC engine covers Rasch/1PL, 2PL, PCM, RSM, GPCM and multidimensional models, validated on million-scale samples.
§ 2 — MODELING

The IRT model family: which one to use?

Right/wrong items usually call for Rasch (1PL) or 2PL; polytomous items (Likert scales, partial credit) call for PCM, RSM, or GPCM. Complex assessments may need between-item multidimensional models, latent regression, and multi-group analysis.

The choice depends on item types, sample size, and research purpose. Picking the wrong model throws no error — it just quietly, systematically distorts every downstream ability estimate.

How WEIAN does it: model choices come with written justification and fit diagnostics — never blind defaults.
§ 3 — COMPARABILITY

What is equating & linking?

Scores from different years or different test forms can't be compared directly — the forms differ in difficulty. Equating uses anchor-item designs and parameter linking to put different tests on the same ruler. Every "trend" conclusion rests on it.

Done wrong, an apparent "improvement" may just mean the test got easier. For surveys that inform decisions this is not a technical detail — it changes the conclusion.

How WEIAN does it: anchor-design review, linking-constant diagnostics, and a fully reproducible, documented equating pipeline.
§ 4 — REPRESENTATIVENESS

What is calibration weighting?

When the sample structure doesn't match the population, weights must be adjusted so weighted estimates match population margins or target rates — while distorting the original sampling weights as little as possible.

Manual reweighting is neither rigorous nor reproducible. The rigorous approach formalizes it as an optimization problem — following Deville & Särndal's (1992) calibration theory — with effective-sample-size and design-effect diagnostics.

How WEIAN does it: our ratecalib package (on CRAN) solves it as bounded convex QP, with soft/exact modes and built-in diagnostics.
§ 5 — POST-PROCESSING

How are sparse population cells handled?

Cross-tabulating by province × gender × urban/rural × age × education leaves many cells with tiny or zero samples, making proportion estimates unstable. Cells must be merged — but every final unit needs positive samples, and subgroup grade shares must stay within target intervals.

That is a constraint-heavy combinatorial optimization problem. Ad-hoc merging silently corrupts the estimates. It deserves a deterministic optimal solution, not cell-by-cell judgment calls.

How WEIAN does it: our mergecalib package solves it with set-partitioning integer programming — and reports honestly when constraints are infeasible.
§ 6 — FAIRNESS

What is DIF (differential item functioning)?

If respondents of equal ability but from different groups (urban/rural, gender) perform systematically differently on an item, the item may be biased — it's measuring group membership, not ability.

DIF analysis finds such items. It's a key fairness step — especially for national surveys covering diverse populations.

How WEIAN does it: DIF screening is built into our standard modeling workflow; flagged items are reviewed jointly with your content experts.

A methodology question
on your project?

Email us →
Initial feasibility assessment is free — describe your project and we'll respond with a technical plan.
See these methods as shipped software in Tools & Cases.

contact@weiandata.com