Export file reading functionality

[johncmerfeld]

Provide a file reader that can be imported and used with compiled Earthmover configs to turn an input path into a dataframe. Runway needs this in order to manipulate source files consistently with Earthmover's logic.

read_compiled_source takes in a named Earthmover source (i.e. not a path, but something like input) and a compiled Earthmover file and returns a dataframe (the caller can ask for an unmaterialized Dask dataframe). Requiring that the config file be compiled makes the code lighter and more predictable, and seems like a reasonable setup step, especially for our intended use case.

Also exposes read_file, which takes in a path and a dict of Earthmover config. This is the primitive that read_compiled_source uses. We could leave it out of the init file but callers would still be able to use it if they wanted to.

Open to other file naming / organization.

You can test the functionality out with something like this:

# reader_demo.py

import logging
import os
import subprocess
import sys

import earthmover

REPO_ROOT = os.path.dirname(os.path.abspath(__file__))
PROJECT_DIR = os.path.join(REPO_ROOT, "example_projects", "11_composition")
COMPILED = os.path.join(PROJECT_DIR, "earthmover_compiled.yaml")


def compile_project():
    """Install packages and compile, so earthmover_compiled.yaml is up to date."""
    # Ensure the subprocess can import earthmover when run from a source checkout.
    env = {**os.environ, "PYTHONPATH": os.pathsep.join([REPO_ROOT, os.environ.get("PYTHONPATH", "")])}
    for command in ("deps", "compile"):
        subprocess.run(
            [sys.executable, "-m", "earthmover", command],
            cwd=PROJECT_DIR, env=env, check=True,
        )


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    logger = logging.getLogger("demo")
    compile_project()
    # `schools` is defined in the project; `students_anytown` comes from a package --
    # both are readable by name once compiled.
    for source_name in ("schools", "students_anytown"):
        df = earthmover.read_compiled_source(source_name, COMPILED, logger=logger)
        print(f"\n{source_name}: {df.shape[0]} rows x {df.shape[1]} cols")
        print(df.head().to_string(index=False))
    # read_file works too, given a path and a config dict (no project required).
    schools_csv = os.path.join(PROJECT_DIR, "sources", "schools.csv")
    df = earthmover.read_file(schools_csv, {"type": "csv", "header_rows": 1}, logger=logger)
    print(f"\nread_file(schools.csv): {df.shape[0]} rows x {df.shape[1]} cols")

[snyk-io-us]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.