TIMX 655 - handle deletes for researchdatabases (AZ)

[ghukill]

Purpose and background context

Why these changes are being introduced:

It turns out that the Springshare OAI endpoint that we harvest AZ items (research databases) has never produced deletes. Without those deleted records / tombstones in OAI, we were not successfully removing records from TIMDEX.

Transmogrifier is currently the place in the TIMDEX ETL ecosystem where records are first written to the dataset, and most commonly, if they are records to index or delete. There may come a time when explicit pre-transform work is performed to establish source records in the TIMDEX dataset, then we transform them, but Transmogrifier has historically been responsible for that double duty and continues to be at this time.

Ultimately, we need to identify AZ items that are no longer publicly available and establish action=delete records in the TIMDEX dataset to have them removed. The OAI harvester is less per-source opinionated than Transmogrifier, making it a poor choice for this.

How this addresses that need:

A new ResearchDatabases transformer class has been created.

Formerly, the TIMDEX source researchdatabases used the Springshare transformer class as a naive OAIDC XML transformation. This new class changes nothing about the metadata transformation, but allows for a place to identify records that were formerly indexed in TIMDEX but are no longer publicly accessible.

There is precedence here in the libguides source, which also performs some additional work via the Springshare API. With that scaffolding already present, it was a relatively simple addition to have researchdatabases do a bit of extra work beyond the OAI XML records provided.

The ResearchDatabases transformer does something unique: while yielding the OAI XML records provided by the harvester like normal, it also queries the Springshare API to get the current list of public AZ item identifiers and queries the TIMDEX dataset for current researchdatabases records. Any identifiers present in the TIMDEX dataset but absent from the current Springshare API results are no longer public, and for these a synthetic OAI XML record is injected into the records yielded by this class for transformation. Those are handled by pre-existing logic and ultimately get written to the TIMDEX dataset with action=delete, prompting a removal from Opensearch downstream.

This approach — querying the TIMDEX dataset for "previous" known identifiers — replaces an earlier design that relied on a managed text file to track identifiers between runs. Querying the dataset directly avoids the need for an additional file to manage and keeps the source of truth in one place.

Lastly, delete detection is optional. If any of the required env vars [LIBGUIDES_API_TOKEN, LIBGUIDES_CLIENT_ID, TIMDEX_DATASET_LOCATION] are not set, delete detection is skipped and the transformer behaves as a plain OAI XML transformation, keeping the source fully backwards compatible.

How can a reviewer manually see the effects of these changes?

I created two records in the dev TIMDEX dataset for researchdatabases that were not present in the Springshare API response (nonsense identifiers). As such, the following full StepFunction ETL run in Dev, when running Transmogrifier, picked up these records should be deleted:

2026-08-12 18:25:30,915 INFO transmogrifier.sources.xml.researchdatabases._yield_api_records_for_deleting(): 2 identifiers identified for deletion: [None, 'i-am-old-record']
...
...
2026-08-12 18:25:35,206 INFO transmogrifier.cli.main(): Completed transform, total records processed: 938, transformed records: 936, skipped records: 0, deleted records: 2

NOTE: you can ignore the None in [None, 'i-am-old-record'], that was just as mistake mocking some records.

The actual synthetic source records injected into the TIMDEX dataset, via Transmogrifier, looked like this:

<?xml version="1.0" ?>
<record>
  <header status="deleted">
    <identifier>oai:libguides.com:az/i-am-old-record</identifier>
    <datestamp>2026-08-12T18:25:30Z</datestamp>
    <setSpec>az</setSpec>
  </header>
  <note>This is a synthetic delete record created by Transmogrifier.</note>
  <metadata/>
</record>

This synthetic source record is helpful for understanding why/what/how the record is deleted, but it's important to focus that just the presence of a row in the dataset with action=delete was sufficient for the rest of the ETL process to remove it from Opensearch.

Includes new or updated dependencies?

NO

Changes expectations for external applications?

YES

• The researchdatabases source now requires the Springshare API
  credentials and TIMDEX_DATASET_LOCATION env var to enable delete
  detection, though these are optional for backwards compatibility
• If AZ items are deleted or hidden, they should get removed from
  TIMDEX now.

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/TIMX-655

Code review

• Code review best practices are documented here and you are encouraged to have a constructive dialogue with your reviewers about their preferences and expectations.

[ghukill]

This is conceptually the most important part to be clear how it works.

The goal of parse_source_file(), for any transformer, is to parse records from the source file it receives -- OAI XML, browsertrix JSONL, you name it -- that will be transformed and added to the TIMDEX dataset.

Part of Transmogrifier's responsibility has always been sniffing out if the record is "deleted" or not. Because the OAI feed will never provide deleted records, our shim here is to inject records as if they had existed in the OAI XML input file.

This admiteddly deviates from other sources where a source file has X records and so X records are written to the dataset, but not entirely. Other sources already skip records, so the records written to the dataset may be less.

In theory, I still think that it would be ideal if Transmog was just transformation of metadata, and less how records "make it into" the TIMDEX dataset, but it's currently the only opinionated place to do that in our ecosystem without really polluting the harvesters.

[Copilot]

Pull request overview

This PR introduces delete detection for the researchdatabases (AZ) source by adding a dedicated transformer that supplements harvested OAI records with synthetic “deleted” OAI records derived from comparing current Springshare API identifiers against existing TIMDEX dataset records.

Changes:

• Add ResearchDatabases transformer to inject synthetic delete records when AZ items are no longer public.
• Refactor LibGuidesAPIClient into transmogrifier/helpers.py and extend it with AZ-fetching helpers.
• Wire researchdatabases source config to use the new transformer and add tests/fixtures for the new behavior.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
transmogrifier/sources/xmltransformer.py	Allows subclass override of XML parsing entrypoint by removing @final.
transmogrifier/sources/xml/springshare.py	Fixes link-generation logging to only warn when identifier is missing.
transmogrifier/sources/xml/researchdatabases.py	New transformer that yields harvested OAI records plus synthetic delete records.
transmogrifier/sources/json/libguides.py	Switches to shared LibGuidesAPIClient from helpers.
transmogrifier/helpers.py	Adds LibGuidesAPIClient and AZ identifier retrieval methods.
transmogrifier/config.py	Points researchdatabases at the new transformer; adds TIMDEX_DATASET_LOCATION.
tests/sources/xml/test_researchdatabases.py	Adds tests for combined OAI + synthetic delete yielding and env-var skip behavior.
tests/sources/json/test_libguides.py	Updates request-mocking path after moving API client code.
tests/fixtures/researchdatabases/researchdatabases-2026-08-11-full-extracted-records-to-index.xml	Adds fixture XML for researchdatabases OAI parsing tests.
README.md	Documents TIMDEX_DATASET_LOCATION env var.

Suppressed comments (2)

transmogrifier/helpers.py:246

• get_current_az_identifiers() returns list(non_hidden_az_df.id) without normalizing types; depending on the API payload, pandas may represent IDs as ints, which would make set comparisons against dataset string IDs behave incorrectly (potentially triggering widespread deletes). Casting IDs to str here makes the comparison stable.

    def get_current_az_identifiers(self) -> list[str]:
        """Get list of identifiers for non-hidden / public AZ items.

        When filtering to enable_hidden = 0, the count matches the OAI-PMH full harvest
        for AZ items.
        """
        az_df = self.fetch_az(self.get_api_token())
        non_hidden_az_df = az_df[az_df.enable_hidden == "0"]
        return list(non_hidden_az_df.id)

transmogrifier/sources/xml/researchdatabases.py:127

• The dataset query can yield NULL az_identifier values (e.g., if a timdex_record_id doesn't contain az-), which then leads to synthetic deletes like ...:az/None. Filter out NULL identifiers at the query level to prevent accidental deletes for invalid IDs.

                """
                select
                    string_split(timdex_record_id, 'az-')[2] as az_identifier
                from metadata.current_records
                where source = 'researchdatabases'
                and action='index';
                """

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ehanson8]

While this lack of deleted records is unfortunate, I like your solution for it, great work! One test naming suggestion

[ehanson8]

Maybe test_researchdatabases_synthetic_deleted_record_created to follow our usual test naming convention?