diff --git a/AGENTS.md b/AGENTS.md index 63f46ef4..1a961475 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -109,4 +109,19 @@ Applies to every agent (Claude, Codex, Cursor, opencode, ...) working in this re - For substantive calibration/linking changes, cite the relevant IRT and psychometrics literature. Commit paper PDFs only when redistribution is permitted; otherwise cite, link, and summarize. +- Method decisions are recorded in `docs/adr/`. Verified APA 7th records + and DOIs are in `docs/papers/README.md`. Do not invent bibliographic + records or leave empty `DOI:` placeholders. +- The implemented linking contract is FIPC (Kim, 2006): anchors keep + old-form values. `autoFIPC()` does not estimate a Stocking–Lord (1983) + or Haebara (1980) transformation (ADR-0001). Numerical estimation is + delegated to `mirt` (ADR-0002), but the returned models are not all one + MML-EM path: raw old/new fits may recover through QMCEM, MHRM, and + `surveyFA` variants, while the linked fit and IPD/DIF path use EM for + nominal items or `tryEM = TRUE` and MHRM otherwise. IPD/DIF screening is + not a published invariance claim (ADR-0003). +- Do not restore Kim and Kolen (2010), "Linking item parameters to a + base scale," *Journal of Educational Measurement*. That attribution was + incorrect and was removed. The title is Kang and Petersen (2012). A real + Kim and Kolen FIPC paper is Kim and Kolen (2019). diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 59672533..45433d76 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -20,6 +20,9 @@ metadata and CI are wired, and which parts are safe to evolve. - `.github/CODEOWNERS` - code ownership map for reviews - `.github/dependabot.yml` - Automated Actions dependency updates - `.cursor/` - Cloud Agent environment (`environment.json` + `install.sh`) +- `docs/adr/` - architecture decision records (FIPC contract, mirt engine) +- `docs/fixed-parameter-item-calibration.md` - Kim (2006) linking contract +- `docs/papers/README.md` - verified bibliographic sources and DOIs - `docs/coderabbit/review-commands.md` - CodeRabbit command quick reference - `docs/operations/maintenance-runbook.md` - recurring maintainer operations checklist - `README.md` - User/developer entrypoint @@ -40,11 +43,17 @@ R objects (data frame/matrix/model), and outputs are returned as an R list. - Path: `R/aFIPC.R` - Responsibility: - - Calibrate old/new forms using `mirt` + - Calibrate old/new forms using `mirt`, with bounded QMCEM/MHRM/`surveyFA` + recovery when an initial raw-data fit is unacceptable - Optionally detect item parameter drift (IPD) - Apply fixed common-item constraints for test linking + - Fit the linked model with EM for nominal items or `tryEM = TRUE`, and MHRM + otherwise - Produce linked model and score/theta artifacts - Key dependency: `mirt` +- Method decision: FIPC (Kim, 2006), not Stocking-Lord (1983) or + Haebara (1980) transformation estimation; see + `docs/adr/0001-fipc-linking-contract.md` ### 3.2 Package Metadata and API Surface @@ -105,15 +114,83 @@ package metadata, and CI workflow definitions in Git. - Evaluate migration path from historical `packrat/` to a modern lock workflow. -## 10. Project Identification +## 10. Bibliographic grounding + +`autoFIPC()` orchestrates FIPC (Kim, 2006): anchors keep old-form +values and the new form is calibrated onto that scale. That contract +is an alternative to separate calibration plus Stocking and Lord +(1983) or Haebara (1980) characteristic-curve linking, and to +concurrent calibration (Kolen & Brennan, 2014). This repository does +not implement those transformation estimators. + +Numerical estimation lives in `mirt` (Chalmers, 2012). The linked FIPC +fit uses MML-EM (Bock & Aitkin, 1981) for nominal items or when +`tryEM = TRUE`, and MHRM otherwise. Separately fitted old/new raw-data +models can recover through QMCEM, MHRM, and `surveyFA` variants after +an unacceptable initial fit, so a returned aFIPC result must not be +summarized as if every model artifact used MML-EM. Optional IPD +screening calls `mirt::multipleGroup` and `mirt::DIF` with the same +EM-versus-MHRM selection rule as the linked fit; it is not a published +invariance claim (see `docs/adr/0003-ipd-dif-screening-delegation.md`). +Score-scale interpretation is bounded by AERA, APA, and NCME (2014). + +An earlier draft incorrectly attributed "Linking item parameters to a +base scale" to Kim and Kolen (2010) in JEM; that attribution was removed. +The title belongs to Kang and Petersen (2012). Kim and Kolen (2019) is +a separate, real later FIPC application paper. + +Full APA 7th records and DOIs: `docs/papers/README.md`. Accepted +method ADRs: `docs/adr/`. + +Kim, S. (2006). A comparative study of IRT fixed parameter calibration +methods. *Journal of Educational Measurement, 43*(4), 355-381. + + +Stocking, M. L., & Lord, F. M. (1983). Developing a common metric in +item response theory. *Applied Psychological Measurement, 7*(2), +201-210. + +Haebara, T. (1980). Equating logistic ability scales by a weighted +least squares method. *Japanese Psychological Research, 22*(3), +144-149. + +Kolen, M. J., & Brennan, R. L. (2014). *Test equating, scaling, and +linking: Methods and practices* (3rd ed.). Springer. + + +Kim, S., & Kolen, M. J. (2019). Application of IRT fixed parameter +calibration to multiple-group test data. *Applied Measurement in +Education, 32*(4), 310-324. + + +Kang, T., & Petersen, N. S. (2012). Linking item parameters to a base +scale. *Asia Pacific Education Review, 13*(2), 311-321. + + +Chalmers, R. P. (2012). mirt: A multidimensional item response theory +package for the R environment. *Journal of Statistical Software, +48*(6), 1-29. + +Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood +estimation of item parameters: Application of an EM algorithm. +*Psychometrika, 46*(4), 443-459. + + +American Educational Research Association, American Psychological +Association, & National Council on Measurement in Education. (2014). +*Standards for educational and psychological testing*. American +Educational Research Association. + +## 11. Project Identification - Project Name: aFIPC - Repository URL: `https://github.com/ContextualWisdomLab/aFIPC` - Primary Contact: Seongho Bae - Date of Last Update: 2026-08-16 -## 11. Glossary / Acronyms +## 12. Glossary / Acronyms - FIPC: Fixed Item Parameter Calibration - IPD: Item Parameter Drift - IRT: Item Response Theory +- MML-EM: Marginal Maximum Likelihood via the EM algorithm diff --git a/CLAUDE.md b/CLAUDE.md index 6a11a23e..1c149798 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -88,8 +88,9 @@ score/theta outputs, returned as an R list. The linking contract (Kim, 2006): anchor items keep their old-form parameter values fixed during new-form calibration so the new form is calibrated -directly onto the established scale. It is documented in -`docs/fixed-parameter-item-calibration.md` and enforced by +directly onto the established scale. That is FIPC, not a Stocking–Lord or +Haebara transformation. It is documented in +`docs/fixed-parameter-item-calibration.md` and `docs/adr/`, and enforced by `tests/testthat/test-fixed-parameter-calibration.R`. ## Key conventions @@ -125,3 +126,5 @@ Summarized from `AGENTS.md` and `CONTRIBUTING.md`; read those for detail. - `CONTRIBUTING.md` — contribution process and verification baseline - `docs/operations/maintenance-runbook.md` — recurring maintainer operations - `docs/fixed-parameter-item-calibration.md` — calibration/linking basis +- `docs/adr/` — FIPC, mirt MML-EM, and IPD/DIF decision records +- `docs/papers/README.md` — verified source papers and DOIs diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cfc9dbbe..99c6e5a3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -12,6 +12,9 @@ over feature velocity. 2. Prefer small, auditable pull requests. 3. Keep CI/security/docs healthy (`.github/workflows/`, `README.md`, `ARCHITECTURE.md`, `AGENTS.md`). +4. Methodological decisions (linking contract, estimation engine) are + recorded in `docs/adr/`. Cite verified sources from + `docs/papers/README.md`; do not invent bibliographic records. ## Development Setup diff --git a/README.md b/README.md index 7e5816f2..f6394087 100644 --- a/README.md +++ b/README.md @@ -1,55 +1,181 @@ # aFIPC -Automated Fixed Item Parameter Calibration (FIPC) for IRT test linking. - -This package contains the original graduate-school implementation used to -produce accurate fixed-item linking results. The current maintenance goal is to -preserve numerical behavior while modernizing repository operations -(documentation, CI, and dependency hygiene). - -## What this repository contains - -- `R/aFIPC.R`: core `autoFIPC()` implementation -- `DESCRIPTION`, `NAMESPACE`, `man/`: package metadata and generated docs -- `packrat/`: historical dependency lock/vendor directory -- `.github/workflows/`: CI/security automation - -## Development status - -- Algorithmic core is legacy but trusted for historical outputs. -- Operational guardrails are now maintained via GitHub Actions and Dependabot. -- Legacy `packrat` bootstrap is opt-in via `AFIPC_ENABLE_PACKRAT=true`. -- Broken host-specific `packrat/lib-R` symlinks were removed for portable builds. -- Architectural and agent operation docs are available in: - - `ARCHITECTURE.md` - - `AGENTS.md` - - `CLAUDE.md` - - `CONTRIBUTING.md` - - `.github/SECURITY.md` - -## Collaboration workflow - -- Pull request template: `.github/PULL_REQUEST_TEMPLATE.md` -- Issue templates: `.github/ISSUE_TEMPLATE/` -- Code ownership: `.github/CODEOWNERS` -- Code quality checks: `.github/workflows/code-quality.yml` -- Security checks (private-safe): `.github/workflows/security-audit.yml` -- Secret-scan policy config: `.gitleaks.toml` -- CodeRabbit command reference: `docs/coderabbit/review-commands.md` -- Maintainer operations runbook: `docs/operations/maintenance-runbook.md` - -## Local package check +[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ContextualWisdomLab/aFIPC) + +**Fixed-item IRT calibration for keeping test forms on a common score scale.** + +aFIPC automates fixed-item parameter calibration (FIPC) for linking and equating +workflows. It helps psychometric teams carry anchor-item information from a +reference form into calibration of a new form, inspect item-parameter drift, and +obtain linked model/score artifacts without turning the repository into a +general assessment platform. + +## When aFIPC fits + +Use aFIPC when you have a reference form, a newly administered form, and an +explicitly reviewed set of common items whose parameters should anchor the +linking design. The package owns the in-process calibration/linking workflow. +Test delivery, source-system data collection, operational score policy, and +downstream decision authority remain outside this repository. + +The numerical implementation is compatibility-sensitive. Changes to calibration +behavior should be backed by regression evidence rather than incidental +refactoring. + +## Core workflow + +1. Prepare old/reference-form and new-form response data, or compatible fitted + model objects. +2. Identify the corresponding common-item names on both forms. +3. Run `autoFIPC()` with the intended item model and explicit common-item + confirmation. +4. Review convergence, item-parameter-drift evidence, and linked outputs before + downstream use. + +A non-interactive raw-data API shape is: + +```r +result <- autoFIPC( + newformXData = new_form, + oldformYData = reference_form, + newformCommonItemNames = common_new, + oldformCommonItemNames = common_old, + newformBILOGprior = FALSE, + oldformBILOGprior = FALSE, + confirmCommonItems = TRUE +) +``` + +The explicit BILOG-prior choices matter for the default 3PL path: leaving either +choice as `NULL` can require interactive input when raw response data are fitted. +If callers use already fitted compatible model objects, review the complete +argument contract before omitting those raw-data choices. + +`autoFIPC()` returns the base-form, new-form, and linked-model artifacts as an R +list. See `man/autoFIPC.Rd` for the complete argument contract. + +## Methodological boundary + +`autoFIPC()` implements FIPC: common items keep reference-form parameter values +while the new form is calibrated onto that scale (Kim, 2006). This differs from +separate calibration followed by Stocking-Lord or Haebara characteristic-curve +transformations, and from concurrent calibration. aFIPC does not estimate +Stocking-Lord or Haebara linking constants. + +The current estimation path delegates IRT estimation to `mirt` rather than +owning an independent numerical IRT engine. The accepted method decisions are +recorded in [the ADR index](docs/adr/README.md), and the verified APA records and +DOIs are maintained in [the research index](docs/papers/README.md). + +Key sources include: + +- Kim, S. (2006). A comparative study of IRT fixed parameter calibration + methods. *Journal of Educational Measurement, 43*(4), 355-381. + +- Chalmers, R. P. (2012). mirt: A multidimensional item response theory package + for the R environment. *Journal of Statistical Software, 48*(6), 1-29. + +- Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of + item parameters: Application of an EM algorithm. *Psychometrika, 46*(4), + 443-459. + +Linking quality still depends on anchor quality, model fit, calibration +assumptions, and the comparability of the forms being linked. Repository tests +and cited methodology are implementation evidence; they do not make arbitrary +forms automatically comparable. + +## Evaluate the current source + +This repository currently provides source rather than an immutable GitHub +release. It also has a known GPL-family runtime blocker: `DESCRIPTION` imports +`mirt`, and `rcmdcheck` does **not** install that dependency for the package under +check. Therefore this README does not present a fresh `install.packages("mirt")` +bootstrap as a commercially acceptable onboarding path. + +If you are maintaining the existing legacy development environment and its +current dependency graph has already been provisioned for license-diligence or +compatibility work, the repository check itself is: ```bash R_PROFILE_USER=/dev/null Rscript -e \ -'install.packages(c("rcmdcheck"), repos="https://cloud.r-project.org")' +'install.packages("rcmdcheck", repos="https://cloud.r-project.org")' + R_PROFILE_USER=/dev/null Rscript -e \ 'rcmdcheck::rcmdcheck(args = c("--no-manual", "--as-cran"), error_on = "warning")' ``` -## Maintenance policy +This is **not** a clean commercial-install recipe: `rcmdcheck` expects the +package's declared runtime dependencies, including the currently disallowed +`mirt`, to already exist. A commercially compatible clean setup is blocked until +issue #320 replaces/removes that runtime path and the final package graph is +revalidated. + +The historical `packrat/` tree is retained for compatibility archaeology. Its +bootstrap is opt-in via `AFIPC_ENABLE_PACKRAT=true` and should not be treated as +the preferred dependency workflow. + +## Architecture and responsibility + +The runtime is single-process and fileless: callers provide in-memory R +data/model objects and receive R model artifacts. `R/aFIPC.R` owns the main +fixed-item linking workflow, while `R/surveyFA.R` contains supporting analytical +routines. Package metadata and generated reference docs live in `DESCRIPTION`, +`NAMESPACE`, and `man/`. + +The current calibration engine calls the external `mirt` package for IRT +estimation and parameter/model operations. That dependency is a material runtime +boundary, not an implementation detail that the repository license can override. + +See [ARCHITECTURE.md](ARCHITECTURE.md) for the component map and maintenance +boundaries. + +## Quality and change control + +Repository automation covers R package checks, code/document quality, +workflow/security auditing, dependency review where the platform supports it, +and supply-chain posture. Before modifying estimation/linking logic, read +[CONTRIBUTING.md](CONTRIBUTING.md) and preserve the documented regression and +review discipline. + +Current source metadata is `0.1.0`; the repository does not currently publish +GitHub Releases. A source version or passing development check is therefore not +presented as an immutable supported release. + +## Commercial licensing status + +**Not currently cleared for ContextualWisdomLab commercial +intake/distribution.** + +`DESCRIPTION` presently declares `GPL-3 | file LICENSE`, while the runtime +directly imports `mirt`, whose current CRAN distribution is GPL-family licensed. +ContextualWisdomLab's commercial intake policy does not accept GPL/LGPL/AGPL +family software as the normal dependency baseline. The repository-authored +source grant and third-party runtime obligation are separate questions; neither +can be made policy-compliant by README wording alone. + +Issue #320 owns the required source-provenance/relicensing review and replacement +of the GPL-family runtime path while preserving the actual fixed-item +calibration/linking estimand and regression behavior. Until that work is +integrated and verified, do not present this repository as Apache-2.0/MIT +cleared or commercially policy-compliant. + +## Documentation + +- [Public documentation home](docs/index.md) - product scope, workflow, + architecture, and change-control entry point. +- [FIPC linking contract](docs/fixed-parameter-item-calibration.md) - what is + fixed, what is estimated, and how the linked scale is defined. +- [Architecture decisions](docs/adr/README.md) - FIPC and estimation decisions. +- [Research sources](docs/papers/README.md) - verified methodological references + and DOIs. +- [Architecture](ARCHITECTURE.md) - component and responsibility boundaries. +- [Generated R reference](man/autoFIPC.Rd) - `autoFIPC()` API contract. +- [Contributing](CONTRIBUTING.md) - contributor workflow and verification + expectations. +- [.github/SECURITY.md](.github/SECURITY.md) - vulnerability-reporting guidance. +- [Ask DeepWiki](https://deepwiki.com/ContextualWisdomLab/aFIPC) - + repository-aware navigation. -- Prefer preserving equation/calibration behavior over refactoring. -- Avoid silent behavioral changes in `autoFIPC()` without explicit regression - evidence. -- Keep CI green on supported runners and keep Actions pinned/updated. +For scientific or operational changes, open an issue or pull request with the +exact design assumptions, affected calibration behavior, and reproducible +verification evidence. diff --git a/docs/adr/0000-template.md b/docs/adr/0000-template.md new file mode 100644 index 00000000..0a5bbe63 --- /dev/null +++ b/docs/adr/0000-template.md @@ -0,0 +1,33 @@ +# ADR-NNNN: Short decision title + +- Status: Proposed | Accepted | Deprecated | Superseded by ADR-NNNN +- Date: YYYY-MM-DD +- Deciders: maintainers + +## Context + +What question is being decided, and which constraints apply? Cite verified +sources. Do not invent bibliographic records. + +## Decision + +State the choice as a contract the package will keep. + +## Alternatives considered + +What was rejected and why, with sources when the alternative is a published +method. + +## Consequences + +Positive and negative follow-through for maintainers and callers. + +## Claim boundary + +What this decision does and does not claim. Separate orchestration in this +package from estimation or methodology that lives elsewhere. + +## References + +Use APA 7th. Include a DOI or publisher URL when one exists. Leave no empty +`DOI:` placeholders. diff --git a/docs/adr/0001-fipc-linking-contract.md b/docs/adr/0001-fipc-linking-contract.md new file mode 100644 index 00000000..eaf8e75a --- /dev/null +++ b/docs/adr/0001-fipc-linking-contract.md @@ -0,0 +1,117 @@ +# ADR-0001: FIPC as the linking contract + +- Status: Accepted +- Date: 2026-08-16 +- Deciders: maintainers + +## Context + +`autoFIPC()` places a new IRT form onto an established old-form scale by +declaring common items and calibrating the new form while those anchors +keep their old-form parameter values. The scientific question is which +published linking family that workflow implements. + +Kim (2006) frames fixed item parameter calibration (FIPC): old +operational or anchor parameters are treated as known during new-form +calibration so the new form is estimated directly on the base scale. +Kolen and Brennan (2014) survey the broader equating and linking +toolkit, including separate calibration plus a characteristic-curve +transformation, concurrent calibration, and fixed-parameter approaches. +The *Standards for Educational and Psychological Testing* +(AERA, APA, & NCME, 2014) limit how linked scores may be interpreted. + +A withdrawn repository cite attributed "Linking item parameters to a +base scale" to Kim and Kolen (2010) in the *Journal of Educational +Measurement*. That record is not a real JEM article. The title belongs +to Kang and Petersen (2012). A real Kim and Kolen FIPC paper is their +2019 *Applied Measurement in Education* application to multiple-group +data. + +## Decision + +This package implements FIPC as specified by Kim (2006) and restated in +`docs/fixed-parameter-item-calibration.md`: + +1. For each eligible anchor pair, copy the old-form item parameter + vector onto the matching new-form item. +2. Hold those copied parameters fixed (`est := FALSE`) during the + linked calibration. +3. Estimate only non-anchor new-form parameters on the scale defined by + the fixed anchors. + +`autoFIPC()` orchestrates that contract. It does not estimate a +Stocking–Lord or Haebara transformation. Inspection of `R/aFIPC.R` +shows no characteristic-curve linking objective; the linked call is +`mirt::mirt(..., pars = NewScaleParms)` after the copy-and-fix step. + +## Alternatives considered + +- **Separate calibration + Stocking and Lord (1983).** Calibrate each + form freely, then find a linear transformation that matches test + characteristic curves. Canonical characteristic-curve equating; not + what `autoFIPC()` computes. +- **Separate calibration + Haebara (1980).** Calibrate each form + freely, then match item characteristic curves by weighted least + squares. Also a post-calibration transformation; not implemented + here. +- **Concurrent calibration.** Estimate both forms in one run with + shared parameters for common items. `autoFIPC()` instead calibrates + forms separately and then fixes anchors (Kim, 2006). +- **Kang and Petersen (2012).** Correct source for the title "Linking + item parameters to a base scale." Useful background on placing + parameters onto a base scale; not the FIPC contract this package + implements. +- **Kim and Kolen (2019).** Later FIPC application to multiple-group + test data. Supports FIPC as a published method family; does not + replace Kim (2006) as the contract implemented here. + +## Consequences + +- Maintainers must preserve the copy-and-fix invariant unless a + regression fixture and explicit maintainer intent say otherwise. +- Docs must contrast FIPC with Stocking–Lord and Haebara so readers do + not infer that `autoFIPC()` returns those transformation constants. +- Score-scale claims stay inside AERA/APA/NCME (2014) limits: linking + does not by itself justify interchangeable high-stakes + interpretations. + +## Claim boundary + +This package orchestrates FIPC. Estimation of item-response +probabilities, the MML-EM cycles, and scores lives in `mirt` (see +ADR-0002). Accepting FIPC here is not a claim that Stocking–Lord, +Haebara, or concurrent calibration are inferior; they are different +published designs. It is also not a claim that linked scores meet a +particular testing-program validity argument. + +## References + +Kim, S. (2006). A comparative study of IRT fixed parameter calibration +methods. *Journal of Educational Measurement, 43*(4), 355–381. + + +Stocking, M. L., & Lord, F. M. (1983). Developing a common metric in +item response theory. *Applied Psychological Measurement, 7*(2), +201–210. + +Haebara, T. (1980). Equating logistic ability scales by a weighted +least squares method. *Japanese Psychological Research, 22*(3), +144–149. + +Kolen, M. J., & Brennan, R. L. (2014). *Test equating, scaling, and +linking: Methods and practices* (3rd ed.). Springer. + + +Kim, S., & Kolen, M. J. (2019). Application of IRT fixed parameter +calibration to multiple-group test data. *Applied Measurement in +Education, 32*(4), 310–324. + + +Kang, T., & Petersen, N. S. (2012). Linking item parameters to a base +scale. *Asia Pacific Education Review, 13*(2), 311–321. + + +American Educational Research Association, American Psychological +Association, & National Council on Measurement in Education. (2014). +*Standards for educational and psychological testing*. American +Educational Research Association. diff --git a/docs/adr/0002-mirt-mml-em-engine.md b/docs/adr/0002-mirt-mml-em-engine.md new file mode 100644 index 00000000..2d327171 --- /dev/null +++ b/docs/adr/0002-mirt-mml-em-engine.md @@ -0,0 +1,116 @@ +# ADR-0002: mirt as the estimation engine + +- Status: Accepted +- Date: 2026-08-16 +- Deciders: maintainers + +## Context + +FIPC (ADR-0001) is a linking contract: which parameters are copied from +the old form and held fixed. Someone still has to estimate the free +item parameters and the ability distribution. This package is an R +orchestrator, not a new IRT estimator. + +Chalmers (2012) describes `mirt`, the package `DESCRIPTION` imports. +Bock and Aitkin (1981) give the marginal maximum-likelihood EM +(MML-EM) basis used when `mirt` is called with `method = "EM"`. + +The implementation does not use one estimation method for every model +artifact. The method policy has three distinct stages: + +1. When raw old/new form data must first be fitted, `autoFIPC()` starts with + the ordinary `mirt` path. If the current fit is still unacceptable and the + corresponding `tryFitwholeOldItems` or `tryFitwholeNewItems` flag is true, + that flag gates only the direct QMCEM retry followed by the direct MHRM + retry. If the model remains unacceptable after that stage—or if the direct + whole-form retry flag is false—the later `surveyFA()` recovery sequence is + still evaluated independently: `forceUIRT`, then `forceNormalEM`, then the + `unstable` path, then `forceMHRM`, with each later step attempted only while + the current model remains unacceptable. These raw-form recovery gates are + independent of the later linked-fit `tryEM` choice. +2. The linked FIPC fit uses EM when `itemtype == "nominal"` or `tryEM` is + true. Otherwise it uses MHRM. +3. IPD/DIF screening follows the same EM-versus-MHRM selection rule as the + linked fit: EM for nominal items or `tryEM = TRUE`, MHRM otherwise. + +Accordingly, an `autoFIPC()` result can legitimately contain old/new form +models fitted by QMCEM, MHRM, or a `surveyFA()` recovery path even when the +linked model uses EM. Setting `tryFitwholeOldItems = FALSE` or +`tryFitwholeNewItems = FALSE` suppresses only the corresponding direct +QMCEM/MHRM whole-form retries; it does not suppress later `surveyFA()` recovery. +Documentation must not label every returned model as MML-EM. + +## Decision + +Use `mirt` as the estimation engine while keeping the method of each model +artifact explicit: + +- Separate old-form and new-form raw-data fits call `mirt::mirt`. Their + `tryFitwhole*` flags gate the direct QMCEM-then-MHRM retries only; if the + current model is still unacceptable, `surveyFA()` recovery remains a + separate subsequent gate sequence. +- The linked fit calls `mirt::mirt` with `pars` after the FIPC copy-and-fix + step. +- Default linked estimation uses `method = "EM"` because `tryEM` defaults to + true; non-nominal linked estimation uses `method = "MHRM"` when + `tryEM = FALSE`. +- IPD/DIF uses `mirt::multipleGroup` and `mirt::DIF` with the same explicit + EM/MHRM branch. +- Scores and expected-score artifacts use `mirt` helpers such as `fscores` + and `expected.test`. + +This package does not reimplement the likelihood, quadrature, EM, QMCEM, or +MHRM algorithms. + +## Alternatives considered + +- **A custom estimator in `R/aFIPC.R`.** This would duplicate a maintained + estimator and risk silent numerical drift. Rejected. +- **A different IRT package.** Historical outputs were produced with `mirt`. + Changing engines would be a scientific behavior change, not a docs fix. +- **Treating `tryFitwhole* = FALSE` as disabling every later recovery.** + Rejected because current source places the `surveyFA()` recovery sequence + outside those direct retry gates. +- **Calling every result “MML-EM.”** Rejected because source permits QMCEM and + MHRM for raw-form recovery and MHRM for the linked/IPD branch when the + explicit method policy selects it. + +## Consequences + +- Numerical changes in `mirt` can change `autoFIPC()` output even when this + repository's R sources are untouched. +- Evidence about a returned model should record its actual estimation path; + `tryEM = TRUE` alone does not prove the separately fitted old/new models used + EM after all recovery attempts. +- Disabling `tryFitwholeOldItems` or `tryFitwholeNewItems` must not be described + as disabling all raw-form recovery unless runtime behavior is changed in a + dedicated behavior PR with regression evidence. +- Formula-integrity reviews in + `docs/fixed-parameter-item-calibration.md` apply to orchestration only. + Estimation mathematics stay in `mirt`. +- `man/autoFIPC.Rd` remains roxygen-generated from `R/aFIPC.R`. Method + citations belong in these Markdown ADRs and `docs/papers/README.md` unless + the roxygen `@references` block is updated in the same change. + +## Claim boundary + +Choosing `mirt` is an engineering dependency decision. The EM/MHRM method +selection and raw-form recovery sequence are implementation behavior, not a +claim that this package contributes a new estimation algorithm or that one +method is universally more accurate. Linking-scale interpretation remains +bounded by ADR-0001 and AERA/APA/NCME (2014). + +## References + +Chalmers, R. P. (2012). mirt: A multidimensional item response theory +package for the R environment. *Journal of Statistical Software, +48*(6), 1-29. + +Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood +estimation of item parameters: Application of an EM algorithm. +*Psychometrika, 46*(4), 443-459. + + +Kim, S. (2006). A comparative study of IRT fixed parameter calibration +methods. *Journal of Educational Measurement, 43*(4), 355-381. + diff --git a/docs/adr/0003-ipd-dif-screening-delegation.md b/docs/adr/0003-ipd-dif-screening-delegation.md new file mode 100644 index 00000000..2d905944 --- /dev/null +++ b/docs/adr/0003-ipd-dif-screening-delegation.md @@ -0,0 +1,74 @@ +# ADR-0003: IPD/DIF screening is delegated to mirt + +- Status: Accepted +- Date: 2026-08-16 +- Deciders: maintainers + +## Context + +`autoFIPC()` can optionally screen common items for item parameter +drift (IPD) before the FIPC copy-and-fix step (`checkIPD`, default +true). The implementation builds a two-group response matrix and calls +`mirt::multipleGroup` plus `mirt::DIF`. Items flagged by that screen +may be dropped from the anchor set. + +That workflow is an operational convenience around `mirt`. It is not +itself a published invariance, DIF, or IPD methodology paper, and it +must not be documented as one. + +## Decision + +Treat IPD/DIF screening as delegated `mirt` machinery: + +- When `checkIPD` is true, screening uses `mirt::multipleGroup` and + `mirt::DIF` on the declared common items. +- Items retained after the screen become the anchors for the Kim + (2006) FIPC contract (ADR-0001). +- Documentation may describe the calls and the effect on the anchor + list. It must not present `autoFIPC()` as a new DIF/IPD statistic + or as evidence that anchors are invariant in a testing-program + sense. + +## Alternatives considered + +- **No IPD screen.** Callers can set `checkIPD = FALSE` and supply + anchors they have already reviewed. +- **A package-local DIF/IPD statistic.** Would be a new methodological + claim and a behavior change. Out of scope for documentation work + and not present in `R/aFIPC.R`. +- **Citing a security standard (NIST, OWASP) for this control.** + Those sources apply to security ADRs. IPD screening is a + psychometric operations step, not a security control. + +## Consequences + +- IPD results inherit `mirt` defaults, version behavior, and the + arguments `autoFIPC()` passes through. Changes in `mirt` can change + which anchors survive. +- Reviewers should not treat a clean IPD screen as a published + invariance argument (AERA, APA, & NCME, 2014). +- Tests that pin FIPC (anchors fixed to old-form values) are separate + from any claim about the DIF screen's Type I error or power. + +## Claim boundary + +This ADR records delegation. It does not claim that the `mirt` DIF +screen equals a named published IPD procedure, that surviving anchors +are drift-free, or that linked scores are interchangeable. The linking +contract remains FIPC (ADR-0001); estimation remains `mirt` MML-EM +(ADR-0002). + +## References + +Chalmers, R. P. (2012). mirt: A multidimensional item response theory +package for the R environment. *Journal of Statistical Software, +48*(6), 1–29. + +Kim, S. (2006). A comparative study of IRT fixed parameter calibration +methods. *Journal of Educational Measurement, 43*(4), 355–381. + + +American Educational Research Association, American Psychological +Association, & National Council on Measurement in Education. (2014). +*Standards for educational and psychological testing*. American +Educational Research Association. diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 00000000..6f4e25f8 --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,28 @@ +# Architecture decision records + +This folder records methodological and estimation decisions for `aFIPC`. +Each ADR states a contract, the alternatives considered, and a claim +boundary so maintainers do not treat orchestration in this package as a +new psychometric method. + +Use `0000-template.md` for new records. Status values are Proposed, +Accepted, Deprecated, or Superseded. Do not invent bibliographic records; +cite verified sources only. + +These ADRs are documentation. They do not change `autoFIPC()` numerical +behavior. + +## Index + +| ID | Title | Status | +| --- | --- | --- | +| [ADR-0001](0001-fipc-linking-contract.md) | FIPC as the linking contract | Accepted | +| [ADR-0002](0002-mirt-mml-em-engine.md) | mirt as the estimation engine | Accepted | +| [ADR-0003](0003-ipd-dif-screening-delegation.md) | IPD/DIF screening is delegated to mirt | Accepted | + +## Related documents + +- Linking contract restatement: + [`docs/fixed-parameter-item-calibration.md`](../fixed-parameter-item-calibration.md) +- Source-paper list with DOIs: [`docs/papers/README.md`](../papers/README.md) +- Repository map: [`ARCHITECTURE.md`](../../ARCHITECTURE.md) diff --git a/docs/fixed-parameter-item-calibration.md b/docs/fixed-parameter-item-calibration.md index efad6448..e37654cc 100644 --- a/docs/fixed-parameter-item-calibration.md +++ b/docs/fixed-parameter-item-calibration.md @@ -9,8 +9,16 @@ should move onto that base scale. This follows the fixed parameter calibration framing in Kim (2006): old operational or anchor item parameters are treated as known values during the new-form calibration so the new form is calibrated directly on the established -scale. The package test `test-fixed-parameter-calibration.R` reproduces this -contract with generated 2PL data: +scale. FIPC is one published linking design among others. Separate calibration +plus a Stocking and Lord (1983) or Haebara (1980) characteristic-curve +transformation, and concurrent calibration of both forms, are alternatives +surveyed by Kolen and Brennan (2014). `autoFIPC()` implements FIPC only: it +copies old-form anchor values, holds them fixed, and re-estimates free new-form +parameters in `mirt`. It does not estimate a Stocking-Lord or Haebara linking +transformation. + +The package test `test-fixed-parameter-calibration.R` reproduces this contract +with generated 2PL data: 1. Generate old-form and new-form responses from known true item parameters. The generated forms include all-zero and all-one response rows plus missing @@ -41,13 +49,41 @@ eligible when both items share the same number of scored response categories, i.e. `n_cat(x_j) == n_cat(y_j)` where `n_cat` counts distinct non-missing responses. +## Estimation-path boundary + +`mirt` owns the numerical estimation algorithms. aFIPC chooses among those +algorithms at distinct stages rather than applying one method label to every +returned model: + +- If old/new inputs are raw response data, `autoFIPC()` first builds separate + form models. If a current form fit is unacceptable and the corresponding + `tryFitwholeOldItems` or `tryFitwholeNewItems` flag is true, that flag gates + only the direct QMCEM retry followed by the direct MHRM retry. +- If the current raw-form model remains unacceptable after that stage—or if the + corresponding direct whole-form retry flag is false—the later `surveyFA()` + recovery sequence is evaluated independently. Source proceeds through + `forceUIRT`, `forceNormalEM`, `unstable`, and `forceMHRM` variants as needed, + stopping when the current model becomes acceptable. These raw-form recovery + choices are independent of the later linked-fit `tryEM` choice. +- The linked FIPC model uses `method = "EM"` when the item type is nominal or + `tryEM = TRUE`; for non-nominal items with `tryEM = FALSE`, it uses MHRM. +- IPD/DIF screening follows the same EM-versus-MHRM selection rule as the + linked fit. + +Therefore, `tryEM = TRUE` is not evidence that the separately returned old/new +form models were ultimately fitted by MML-EM. Likewise, +`tryFitwholeOldItems = FALSE` or `tryFitwholeNewItems = FALSE` suppresses the +direct QMCEM/MHRM retry for that form, not every later `surveyFA()` recovery. +Reproducibility evidence should record the actual fitted-model path when method +identity matters. + ## Formula-integrity audit of performance refactors -The estimation mathematics (item-response probabilities, the MML-EM cycles, -`fscores`, `expected.test`, and the DIF/IPD statistics) live in `mirt`; this -package only orchestrates the linking contract above. The following merged -performance refactors were reviewed against that contract and confirmed -**mathematically equivalent** (no term, margin, or constant changed): +The estimation mathematics (item-response probabilities, EM/QMCEM/MHRM +algorithms, `fscores`, `expected.test`, and DIF/IPD statistics) live in `mirt`; +this package only orchestrates the linking and recovery contracts above. The +following merged performance refactors were reviewed against that contract and +confirmed **mathematically equivalent** (no term, margin, or constant changed): - **#48 / #52** (`82fa77d`, `762b8a9`): hoist `fscores(..., method = 'MAP')` into a variable reused by `expected.test` instead of recomputing it. MAP @@ -70,12 +106,88 @@ pinned to hand-computed reference values in contract (anchors fixed to old-form values, non-anchors left free) is pinned in `tests/testthat/test-fixed-parameter-calibration.R`. +## Relation to other linking methods + +Kolen and Brennan (2014) organize common IRT linking designs as: + +- **Separate calibration + characteristic-curve transformation.** Each form is + calibrated freely. A linear transformation is then chosen to match test + characteristic curves (Stocking & Lord, 1983) or item characteristic curves + (Haebara, 1980). +- **Concurrent calibration.** Both forms are estimated in one run with shared + parameters for common items. +- **Fixed item parameter calibration (FIPC).** Anchor parameters from the old + form are treated as known and held fixed while the new form is calibrated + onto that scale (Kim, 2006; see also Kim & Kolen, 2019, for a later + multiple-group FIPC application). + +`R/aFIPC.R` implements the third design. There is no Stocking-Lord or Haebara +objective, and no post-calibration slope/intercept estimator. An earlier draft +incorrectly attributed "Linking item parameters to a base scale" to Kim and +Kolen (2010) in the *Journal of Educational Measurement*; that attribution was +removed. The title belongs to Kang and Petersen (2012). + +Linked scores still fall under the interpretation limits in the *Standards for +Educational and Psychological Testing* (AERA, APA, & NCME, 2014). The linked +fit defaults to `mirt` MML-EM because `tryEM` defaults to true, while the +explicit non-EM and raw-form recovery paths above remain valid implementation +behavior. See `docs/adr/` for the accepted method decisions. + +## mirt documentation/version evidence + +`DESCRIPTION` imports `mirt` without pinning an exact package version, so this +document must not imply one historical version is permanently authoritative. +At the 2026-09-02 documentation review, current CRAN package/check metadata was +reverified as `mirt` 1.47. Reproducibility evidence should record the installed +version actually used for a calibration run rather than treating that review +snapshot as a permanent pin. + +- CRAN package record: +- CRAN reference manual: + +The upstream `fixedCalib` documentation describes fixed-item calibration based +on Kim (2006) and points to `multipleGroup` for more flexible anchor-item +calibration. The Chalmers (2012) package citation below is the scientific +package reference; the CRAN record/manual are the executable documentation +locators. + ## References -- Kim, S. (2006). A comparative study of IRT fixed parameter calibration - methods. Journal of Educational Measurement, 43(4), 355-381. -- Chalmers, R. P. `mirt::fixedCalib` documentation. The implementation note - describes fixed-item calibration methods based on Kim (2006) and points to - `multipleGroup` for more flexible anchor-item calibration. -- Kim, S., & Kolen, M. J. (2010). Linking item parameters to a base scale. - Journal of Educational Measurement, 47(2), 164-181. +Kim, S. (2006). A comparative study of IRT fixed parameter calibration +methods. *Journal of Educational Measurement, 43*(4), 355-381. + + +Stocking, M. L., & Lord, F. M. (1983). Developing a common metric in +item response theory. *Applied Psychological Measurement, 7*(2), +201-210. + +Haebara, T. (1980). Equating logistic ability scales by a weighted +least squares method. *Japanese Psychological Research, 22*(3), +144-149. + +Kolen, M. J., & Brennan, R. L. (2014). *Test equating, scaling, and +linking: Methods and practices* (3rd ed.). Springer. + + +Kim, S., & Kolen, M. J. (2019). Application of IRT fixed parameter +calibration to multiple-group test data. *Applied Measurement in +Education, 32*(4), 310-324. + + +Kang, T., & Petersen, N. S. (2012). Linking item parameters to a base +scale. *Asia Pacific Education Review, 13*(2), 311-321. + + +Chalmers, R. P. (2012). mirt: A multidimensional item response theory +package for the R environment. *Journal of Statistical Software, +48*(6), 1-29. + +Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood +estimation of item parameters: Application of an EM algorithm. +*Psychometrika, 46*(4), 443-459. + + +American Educational Research Association, American Psychological +Association, & National Council on Measurement in Education. (2014). +*Standards for educational and psychological testing*. American +Educational Research Association. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 00000000..21f91fbd --- /dev/null +++ b/docs/index.md @@ -0,0 +1,105 @@ +# aFIPC + +[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ContextualWisdomLab/aFIPC) + +Automated Fixed Item Parameter Calibration for item-response-theory linking and +equating. + +## Product responsibility + +aFIPC preserves a common measurement scale across test forms by combining +anchor-item information from a reference form with calibration of newly +administered items. It is intended for psychometric workflows where score +comparability across administrations matters and fixed-item linking is the +chosen design. + +The repository owns the in-process R calibration/linking workflow. Test +delivery, source-system data collection, operational score policy, and +downstream decision authority remain outside this package. + +## Core workflow + +1. Prepare reference-form and new-form response data or compatible fitted model + objects. +2. Define the reviewed common-item correspondence between forms. +3. Run `autoFIPC()` with the intended item model and explicit common-item + confirmation. For non-interactive raw-data use of the default 3PL path, + explicitly set both BILOG-prior choices instead of leaving them `NULL`. +4. Review convergence, item-parameter-drift evidence, and linked outputs before + downstream score reporting or operational use. + +## Method and architecture + +- `R/aFIPC.R` contains the main fixed-item linking workflow. +- `R/surveyFA.R` contains supporting analytical routines used by the package. +- `DESCRIPTION`, `NAMESPACE`, and `man/` define package metadata and generated + reference documentation. +- `docs/adr/` records reviewed method/architecture decisions. +- `docs/papers/` retains verified methodological references and DOIs. +- `.github/workflows/` provides continuous-integration and security checks. + +The current calibration engine directly uses the external `mirt` runtime for +IRT estimation and parameter/model operations. FIPC is distinct from separate +calibration plus Stocking-Lord/Haebara transformations and from concurrent +calibration. See the FIPC contract and ADR index for the precise boundary. + +## Onboarding and verification + +The current source graph is not commercially intake-cleared because it directly +imports GPL-family `mirt`. `rcmdcheck` does not install the package-under-check's +runtime dependencies, so there is no honest commercially compatible clean-R +bootstrap to advertise while issue #320 remains open. + +For maintainers working in an already provisioned legacy compatibility or +license-diligence environment, the repository check itself is: + +```bash +R_PROFILE_USER=/dev/null Rscript -e \ +'install.packages("rcmdcheck", repos="https://cloud.r-project.org")' +R_PROFILE_USER=/dev/null Rscript -e \ +'rcmdcheck::rcmdcheck(args = c("--no-manual", "--as-cran"), error_on = "warning")' +``` + +That command assumes the package's existing declared dependencies are already +available; it is not a recommendation to add `mirt` to a new commercial stack. +A clean commercial setup becomes publishable only after #320 replaces/removes +the GPL-family runtime path and the resulting graph passes package, numerical, +security, and provenance verification. + +For contributor expectations and architectural context, read `CONTRIBUTING.md` +and `ARCHITECTURE.md` before modifying estimation or linking logic. + +## Release and commercial-license status + +The repository currently has no published GitHub Release. Source metadata +`0.1.0` and development checks are not immutable release evidence. + +The current source/dependency graph is also **not cleared for +ContextualWisdomLab commercial intake/distribution**: package metadata declares +`GPL-3 | file LICENSE`, and the runtime directly imports the GPL-family `mirt` +package. Issue #320 owns the source-provenance/relicensing review plus +replacement of that runtime dependency while preserving the actual fixed-item +calibration/linking contract. Until that work is complete, do not describe +aFIPC as Apache-2.0/MIT-cleared or commercially policy-compliant. + +## Documentation and support + +- [Repository README](https://github.com/ContextualWisdomLab/aFIPC/blob/master/README.md) + - product, usage, status, and contributor entry point. +- [Architecture](https://github.com/ContextualWisdomLab/aFIPC/blob/master/ARCHITECTURE.md) + - runtime and maintenance boundaries. +- [FIPC linking contract](fixed-parameter-item-calibration.md) - linking design. +- [Architecture decisions](adr/README.md) - method and ownership decisions. +- [Research sources](papers/README.md) - verified references and DOIs. +- [Contributing](https://github.com/ContextualWisdomLab/aFIPC/blob/master/CONTRIBUTING.md) + - development and verification expectations. +- [Ask DeepWiki](https://deepwiki.com/ContextualWisdomLab/aFIPC) - repository-aware + documentation and code navigation. + +Linking quality depends on anchor quality, model fit, calibration assumptions, +and the comparability of the forms being linked. Repository tests and cited +methodology are implementation evidence; they do not make arbitrary test forms +automatically comparable. + +This file is a Pages-ready source only. It is not evidence that GitHub Pages is +published; publication requires repository settings and live HTTPS verification. diff --git a/docs/papers/README.md b/docs/papers/README.md index 75b62728..5f2553ad 100644 --- a/docs/papers/README.md +++ b/docs/papers/README.md @@ -1,46 +1,102 @@ # Source papers for the calibration and linking mathematics -`autoFIPC()` implements the fixed item parameter calibration (FIPC) linking -contract; the underlying IRT estimation is delegated to `mirt`. The canonical -equations this package must match are documented in the sources below. +`autoFIPC()` implements the fixed item parameter calibration (FIPC) +linking contract; the underlying IRT estimation is delegated to `mirt`. +The canonical sources this package must match are listed below. -No PDFs are committed here: the two primary FIPC references (Kim, 2006; Kim & -Kolen, 2010) are published in the *Journal of Educational Measurement* and are -not open access, so they are cited by DOI rather than redistributed, to respect -copyright. The `mirt` reference is open access. +No PDFs are committed here. Copyrighted articles are cited by DOI +rather than redistributed. The `mirt` reference is open access. + +A previous draft listed Kim and Kolen (2010), "Linking item parameters +to a base scale," *Journal of Educational Measurement, 47*(2), +164–181. That bibliographic record is not a real JEM article and is +not kept. The title belongs to Kang and Petersen (2012). A real Kim +and Kolen FIPC paper is Kim and Kolen (2019). ## Primary source — the FIPC linking contract -- **Kim, S. (2006). A comparative study of IRT fixed parameter calibration - methods.** *Journal of Educational Measurement, 43*(4), 355-381. +- **Kim, S. (2006). A comparative study of IRT fixed parameter + calibration methods.** *Journal of Educational Measurement, 43*(4), + 355–381. DOI: - - Canonical rule implemented: old-form (anchor) item parameters are treated as - known and held fixed while the new form is calibrated directly onto the - established base scale. + - Canonical rule implemented: old-form (anchor) item parameters are + treated as known and held fixed while the new form is calibrated + directly onto the established base scale. + +- **Kim, S., & Kolen, M. J. (2019). Application of IRT fixed parameter + calibration to multiple-group test data.** *Applied Measurement in + Education, 32*(4), 310–324. + DOI: + - Later FIPC application to multiple-group data. Supports FIPC as a + published method family; does not replace Kim (2006) as the + contract `autoFIPC()` implements. + +## Characteristic-curve equating (not implemented here) + +`autoFIPC()` implements FIPC, not a Stocking–Lord or Haebara +transformation estimator. These papers are the canonical +characteristic-curve methods that FIPC is an alternative to (separate +calibration plus a linking transformation, versus concurrent +calibration, versus FIPC). + +- **Stocking, M. L., & Lord, F. M. (1983). Developing a common metric + in item response theory.** *Applied Psychological Measurement, + 7*(2), 201–210. + DOI: + - Test characteristic-curve linking after separate calibration. -- **Kim, S., & Kolen, M. J. (2010). Linking item parameters to a base scale.** - *Journal of Educational Measurement, 47*(2), 164-181. - DOI: - - Basis for treating the old-form scale as the fixed base onto which new-form - parameters are placed. +- **Haebara, T. (1980). Equating logistic ability scales by a weighted + least squares method.** *Japanese Psychological Research, 22*(3), + 144–149. + DOI: + - Item characteristic-curve linking by weighted least squares after + separate calibration. + +## Equating and linking handbook + +- **Kolen, M. J., & Brennan, R. L. (2014). *Test equating, scaling, + and linking: Methods and practices* (3rd ed.).** Springer. + DOI: + - Handbook survey of equating, scaling, and linking designs, + including the families contrasted above. + +## Correct source for the withdrawn title + +- **Kang, T., & Petersen, N. S. (2012). Linking item parameters to a + base scale.** *Asia Pacific Education Review, 13*(2), 311–321. + DOI: + - Real paper with this title (also circulated as ACT Research Report + 2009-2). Not the FIPC contract implemented here. ## Estimation engine (open access) -- **Chalmers, R. P. (2012). mirt: A Multidimensional Item Response Theory - Package for the R Environment.** *Journal of Statistical Software, 48*(6), - 1-29. DOI: (open access) - - Provides the MML-EM estimation, `fscores` (MAP), `expected.test`, and the - `multipleGroup`/`DIF` machinery used for item parameter drift screening. - - `?mirt::fixedCalib` documents fixed-item calibration methods based on - Kim (2006). +- **Chalmers, R. P. (2012). mirt: A multidimensional item response + theory package for the R environment.** *Journal of Statistical + Software, 48*(6), 1–29. + DOI: (open access) + - Provides the MML-EM estimation, `fscores` (MAP), `expected.test`, + and the `multipleGroup`/`DIF` machinery used for item parameter + drift screening. + - `?mirt::fixedCalib` documents fixed-item calibration methods based + on Kim (2006). + +- **Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood + estimation of item parameters: Application of an EM algorithm.** + *Psychometrika, 46*(4), 443–459. + DOI: + - The MML-EM algorithm underlying `mirt`'s `method = "EM"` + estimation. -## Supporting reference +## Score-scale interpretation limits -- **Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of - item parameters: Application of an EM algorithm.** *Psychometrika, 46*(4), - 443-459. DOI: - - The MML-EM algorithm underlying `mirt`'s `method = "EM"` estimation. +- **American Educational Research Association, American Psychological + Association, & National Council on Measurement in Education. + (2014). *Standards for educational and psychological testing*.** + American Educational Research Association. + - Limits on how linked scores and scales may be interpreted. No DOI + is used here; none was verified for this edition. -See `../fixed-parameter-item-calibration.md` for the equation restatement, the -formula-integrity audit of the performance refactors, and the tests that pin -these formulas to reference values. +See `../fixed-parameter-item-calibration.md` for the equation +restatement, the formula-integrity audit of the performance refactors, +and the tests that pin these formulas to reference values. Method +decisions are recorded in `../adr/`. diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md new file mode 100644 index 00000000..440e7f0c --- /dev/null +++ b/docs/product-technical-gap-baseline.md @@ -0,0 +1,78 @@ +# Product / Technical Gap Baseline + +This document records code-current commercial and technical gaps for aFIPC. It is not a roadmap promise and does not turn an open proposal into an accepted architecture decision. + +## Product boundary + +aFIPC owns fixed-item-parameter calibration/linking behavior and the domain meaning of its form, common-item, calibration, prior, linking, and recovery contracts. Shared psychometric kernels may be consumed through released/versioned boundaries when they preserve that product contract; aFIPC does not copy a sibling repository's source or depend on a mutable sibling head. + +Protected repository authority at this baseline is `master@f87c2324f1686135e57d8730c1b0b9420874f300`. Open PRs are evidence/proposals until they satisfy the live protected-branch gates and merge normally. + +## Commercial blocker: runtime and repository licensing + +Issue #320 is the current commercial-intake blocker. The package metadata declares GPL-3 and the core calibration path imports/calls `mirt`, whose CRAN license is GPL (>= 3). Those are separate obligations: changing repository prose or a root license file cannot make the current runtime dependency graph compatible with ContextualWisdomLab's no-GPL-family intake baseline. + +Acceptance for a commercially cleared successor is therefore behavioral and provenance-based, not merely textual: + +- establish first-party/contributor provenance before changing the repository grant; +- remove or replace GPL/LGPL/AGPL-family runtime, build, vendored, copied-source, or derivative obligations under the intended distribution model; +- preserve the actual fixed-item calibration/linking estimand rather than substituting a different psychometric method to obtain a different license; +- keep package metadata, root license bytes, dependency inventory, SBOM and release evidence mutually consistent; +- retain normal R compatibility only as an explicit boundary while any Rust-first replacement kernel is validated. + +A released `fast-mlsirm` capability may be evaluated as an optional canonical kernel boundary when it actually provides the required estimand and numerical contract. Until then it is a read-only dependency candidate, not a source-copy or mutable-head shortcut. + +## Numerical and psychometric evidence gap + +A replacement calibration runtime is not GREEN merely because unit tests execute. Acceptance must preserve observable aFIPC outputs over right-cleared representative product data and, where simulation is scientifically appropriate, report true-parameter recovery rather than synthetic-data success alone. + +Required evidence for the replacement/calibration boundary: + +- fixed common-item identities and ordering; +- linked item-parameter selection and scale behavior; +- convergence/recovery-path attribution across the currently supported estimation/recovery paths; +- reproducibility under pinned runtime/package/kernel versions; +- simulation studies that report true-parameter bias, RMSE and interval coverage with the data-generating design stated explicitly; +- real/right-cleared product-form replay for commercial acceptance, with synthetic data confined to unit/property/recovery tests. + +Any material numerical difference requires an explicit ADR with the estimand, constraint, alternatives, reason for acceptance/rejection, measured effect, migration risk and rollback path. + +## Current correctness lane: interactive choice domain + +The three interactive yes/no menu boundaries document `1` or `2`. PR #337 is the current repaired candidate that narrows admission before integer coercion and carries deterministic production-path regressions. It remains Draft until one unchanged exact head has terminal R CMD/test, applicable security/SAST/CodeQL evidence, current review findings resolved, protected-base compatibility, and qualifying independent review. + +The supported finding is input-contract/reliability correctness. Oversized decimal coercion to `NA` is not, without additional evidence, proof of integer overflow, memory corruption, remote exploitation, or a specific security severity. + +## Current performance lanes + +Several open branches propose R-level micro-optimizations in the calibration/linking path. Their admissible evidence is narrower than their generated performance language: + +- direct model-column lookup must preserve the source/model column-membership validation that the protected data-frame projection currently provides; +- distinct non-missing category refactors must preserve observed-category semantics, including NA/NaN/factor edge cases; +- local expression or allocation changes are not buyer-visible performance improvements until the actual `autoFIPC()` path is measured. + +Performance promotion requires representative item/respondent/form cardinalities, pinned R/runtime/dependency state, warm-up policy, repeated wall-time distribution including median and p95, allocation/GC or equivalent profile evidence, and numerical-equivalence checks. A unit test, coverage result, or isolated operation-count benchmark is correctness evidence only. + +If profiling shows a material psychometric hot path rather than incidental R overhead, prefer an auditable Rust-first kernel boundary with vector/linear/matrix operations and CPU multithreading, while preserving the product-owned R/API/domain contract through an adapter. Do not migrate a hot path on style grounds alone. + +## Release and rollback gap + +No open branch is a release by itself. A release-ready protected generation must have: + +- one exact protected head after normal reviewed merge; +- terminal required repository and organization checks on that generation; +- numerical/behavioral evidence appropriate to the changed domain boundary; +- version and CHANGELOG aligned with the shipped behavior; +- immutable tag/package plus SBOM and provenance; +- reproducibility instructions and a tested rollback/recovery path. + +Until the licensing blocker and the applicable correctness/numerical gates are resolved, documentation must not describe aFIPC as commercially intake-cleared or a replacement runtime as behaviorally equivalent. + +## Traceability + +- Commercial/runtime licensing owner: issue #320. +- Product/method documentation owner lane: PR #261. +- Interactive choice-domain repair candidate: PR #337. +- Model-column performance/validation candidate: PR #335. +- Protected repository authority recorded above remains the merge/release source of truth. +- CRAN `mirt` package metadata is the primary external license reference; Chalmers (2012), *Journal of Statistical Software, 48*(6), documents the `mirt` estimation framework. External references support the decision record but do not supersede live repository code, dependency bytes, or protected-branch rules.