Merge entities

[paulduchesne]

An important element of federating datasets is the ability to merge like-entities. This process is already supported by the transformation from source identifier to a FIAFcore UUID, which grants the opportunity to enable multiple input entities to point to a single output identifier.

In the current (as of writing) dataset, there are three different FIAFcore UUIDs for Friedrich Wilhelm Murnau. The following script merges these into pointing towards a single new identifier.

import pandas
import pathlib

# input uris, to be merged.

input_uris = [
    'https://dev.fiafcore.org/dd481e5d-79c4-4ff0-9d41-5b262590d8ea',
    'https://dev.fiafcore.org/73f251b4-7a79-40f9-8275-93236b106d39',
    'https://dev.fiafcore.org/fcf4faae-4bed-4167-bf5d-287e20e7b772'
]

# output uri, which will replace all inputs.
# note this does not have to be a **new** identifier, but this seems like a good, agnostic, choice.

output_uri = 'https://dev.fiafcore.org/84bf0af2-db6b-4fb5-910e-8a0040b38af5'

# auth files for processing.

auth_files = [
    pathlib.Path.home() / 'git' / 'fiafcore-bfi' / 'auth.parquet',
    pathlib.Path.home() / 'git' / 'fiafcore-bundesarchiv' / 'auth.parquet'
]

# verify auth files exist.

for a in auth_files:
    if not a.exists():
        raise Exception(f'{a} not found.')

# find and replace.

for a in auth_files:
    df = pandas.read_parquet(a)
    df.loc[df.fiafcore.isin(input_uris), 'fiafcore'] = output_uri
    df.to_parquet(a)

# report on result.

for a in auth_files:
    df = pandas.read_parquet(a)
    print(a.parent.stem)
    print('input:', len(df.loc[df.fiafcore.isin(input_uris)]))
    print('output:', len(df.loc[df.fiafcore.isin([output_uri])]))