U.S. Fish & Wildlife Service — Alaska Region
A set of tools for discovering, exploring, downloading, and quality-checking the Alaska Region's wildlife data assets. Everything is driven by your existing mdEditor metadata and network file server — no new data entry is required.
Note: All code files on this repository, including this
ReadMeand.gitignorewith small edits, where produced using Anthropic's Claude 4.8 AI agent during the Department of Interior's trial period that began on 15 June 2026. Most functionality was complete on that day, with various enhancements completed through 23 June 2026, as usage limits allowed. The maintainer of this respository (Erik Osnas) has no experience programming in Python, html, JavaScript, or css and has no technical knowledge of REST APIs.
Most staff only need the Catalog. Once you've done the one-time setup, here's how to start it:
- Make sure you are connected to the FWS network or VPN
- Open the Miniforge Prompt (from the Start menu)
- Go to the project folder and start the Catalog:
cd "C:\Users\YOURNAME\YOURPATH\RDR_Catalog" python catalog_app.py "\\YOUR-SERVER\path\to\data" "\\YOUR-SERVER\path\to\contacts.json"
- Wait for the message
✅ Loaded N records and ready to serve. - Open your browser to http://localhost:8000
To stop the Catalog, press Ctrl+C in the Miniforge Prompt (or just close the window).
If you've never set this up before, read First-Time Setup first.
Tip: That command is long. Many staff save it in a small local startup file (a
.baton Windows) so they can launch the Catalog with a double-click. Those files are kept local and are not part of the shared code — see Optional: a local shortcut.
- What These Tools Are
- The Problem They Solve
- The Four Components
- First-Time Setup
- Using the Catalog (Most Staff)
- Using the Data Portal (Public-Facing)
- Utility Scripts for Data Managers
- Optional: a Local Shortcut
- The Proof of Concept — Public Drupal Portal
- The Path to Production
- Technical Architecture
- Frequently Asked Questions
This project provides a data library catalog for the Alaska Region's file server. Instead of digging through network folders to find data, staff can type a keyword and instantly find relevant projects, read their full metadata, view their location on a map, and download files directly in a web browser.
There are two web applications and three utility scripts:
| Tool | Audience | Purpose |
|---|---|---|
| Catalog | FWS staff (internal) | The main day-to-day tool for finding and downloading data |
| Data Portal | The public (future) | A polished, fws.gov-style public discovery site |
| Metadata Quality Report | Data managers | Audits metadata quality across the whole repository |
| Restricted Access Scanner | Data managers | Lists records flagged as restricted |
| Change & Integrity Monitor | Data managers | Scans the repository for new, changed, and deleted data |
The Alaska Region collects and manages large amounts of wildlife monitoring data. Currently:
- Data files live in project folders on a network drive (
\\YOUR-SERVER\path\to\data) - Metadata is documented in mdEditor following the ISO 19115 standard
- But there is no easy way to search, discover, or share these assets
Staff must know exactly where to look, partners have no access, and leadership has no clear picture of what data the program holds. Data that took years and significant funding to collect is effectively invisible.
These tools solve that by surfacing your existing data and metadata through searchable interfaces — with no duplication of effort and no new data entry.
The primary internal tool. It scans the network drive, reads every mdEditor metadata file, and serves a rich searchable website at http://localhost:8000. Features include full-text search, faceted filtering, interactive maps, contact cards, smart file downloads, project-to-products navigation, and CSV export.
A public-facing version modeled after fws.gov. It queries the Catalog's data, adds public/internal access control (so restricted records stay hidden from the public), and also searches external sources like GBIF. This is the prototype for an eventual public website.
A command-line audit tool for data managers. It checks every record for 12 types of metadata problems and writes a CSV report so the team can systematically improve metadata quality.
A command-line tool that lists all records flagged as having restricted access, so data managers can quickly confirm which assets are protected.
A command-line tool that scans the repository for new, changed, and deleted data and checks data-governance integrity: when data changes in a project, did its metadata record and the project's changelog.txt also get updated?
You only need to do this once.
These tools run on Python. The easiest way to get Python at FWS is Miniforge, available from the conda-forge.org:
- Go to https://conda-forge.org/download/ and download Miniforge.
- Double click the to install on your FWS computer (no admin rights needed).
After installation, you'll have a "Miniforge Prompt" in your Start menu. This is the terminal you'll use.
The code lives on GitHub at:
https://github.com/USFWS/RDR_Catalog
Download it one of two ways:
- Easiest: On the GitHub page, click the green Code button → Download ZIP, then unzip it to a folder like
Documents\RDR_Catalog - With Git:
git clone https://github.com/USFWS/RDR_Catalog.git
Open the Miniforge Prompt and run this once:
pip install fastapi uvicorn flask requests truststore --userOptional: If you are a data manager who will run the metadata quality report's Active Directory check, also run
pip install ldap3 --user.
The Portal needs a config.py file with your settings. A template is provided so your real settings (and any local paths) never get committed to GitHub:
- In the
alaska-portalfolder, copy or renameconfig.example.pytoconfig.py - Open
config.pyand edit the values for your environment (mainlySSL_CERT) (this is optional now)
The Catalog itself takes its network paths on the command line (Step 3 of Quick Start), so it needs no config file.
You're now ready to use the Catalog.
Open the Miniforge Prompt, go to the project folder, and run:
cd "C:\Users\YOURNAME\YOURPATH\RDR_Catalog"
python catalog_app.py "\\YOUR-SERVER\path\to\data" "\\YOUR-SERVER\path\to\contacts.json"Wait for ✅ Loaded N records and ready to serve., then open http://localhost:8000 in your browser. The first scan of the network drive may take up to a minute.
- Search — type keywords in the search box (searches titles, abstracts, keywords, and species)
- Filter — use the left sidebar to filter by resource type (project, dataset, report, etc.) or status (completed, ongoing, etc.)
- Sort — order results by title, most recently updated, or relevance
- Default view — the Catalog opens showing projects first
Click any card to open its detail panel, which shows:
- Full abstract and overview metadata
- Point of contact (name, position, organization, email, phone)
- External links to ScienceBase, ServCat, and DOI
- Keywords and species
- An interactive map of the data's geographic extent
- File locations with download buttons
Click a download button next to any file. The Catalog checks the file at that moment and shows:
| What you see | What it means |
|---|---|
| ⬇ Download | File is available — it downloads to your computer |
| 📁 Folder icon | The path is a folder (normal for projects) |
| ⚠ Metadata issue | The path points to a folder when it should be a file |
| ❌ File not found | The file is missing on the server |
| ⚠ Network unavailable | You're not connected to the network drive |
| 🔒 Access Restricted | The record is restricted — no download offered |
On a project record, click "View N associated data products" to see all data linked to that project, with breadcrumb navigation back to the project.
Click ⬇ Export CSV above the results to download your current filtered view as a spreadsheet.
Press Ctrl+C in the Miniforge Prompt, or close the window.
The Data Portal is the prototype for a future public website. It presents the same Alaska Region data in an fws.gov-style interface, adds public/internal access control, and also searches GBIF.
Most internal staff will prefer the Catalog. The Portal exists to demonstrate what a public-facing version would look like.
The Portal reads from the Catalog, so you run two commands in two Miniforge Prompt windows.
Window 1 — start the Catalog (if it isn't already running):
cd "C:\Users\YOURNAME\YOURPATH\RDR_Catalog"
python catalog_app.py "\\YOUR-SERVER\path\to\data" "\\YOUR-SERVER\path\to\contacts.json"Window 2 — start the Portal:
cd "C:\Users\YOURNAME\YOURPATH\RDR_Catalog\alaska-portal"
python app.pyThen open http://localhost:5000 in your browser.
- Not logged in → you see only public records (restricted and draft records are hidden)
- Logged in (Staff Login) → you see everything
Locally, the Staff Login accepts any username/password. In a real deployment, this would use your FWS Active Directory (Windows) login automatically.
Press Ctrl+C in each Miniforge Prompt window (or close both windows).
The first two scripts read from the running Catalog, so catalog_app.py must already be running at http://localhost:8000 before you run them. They do not re-scan the files themselves — they use the data the Catalog has already parsed. The last one scans the files fresh and does not depend on the Catalog.
Run them from the Miniforge Prompt, in the project folder.
Lists every record flagged as restricted, so you can confirm which assets are protected.
python scan_restricted.pyExample output:
Fetching catalog from http://localhost:8000/api/catalog...
Scanned 1690 records.
Found 3 restricted record(s):
Title : Alaska YKD Raw Salinity Data 2006 - 2019
Type : tabularDataset
Status : completed
Source file : \\YOUR-SERVER\path\to\data...
File paths :
[RESTRICTED] file://YOUR-SERVER/path/to/data...
Scans every record and writes a CSV report (metadata_quality_report.csv) of metadata quality issues across the whole repository. This is a data-management audit tool — it helps the team systematically improve metadata rather than discovering problems one record at a time.
python metadata_quality_report.py| Check | What it flags |
|---|---|
| Missing abstract | Record has no abstract text |
| Missing time period | Record has no start date |
| No keywords | Record has no keywords (hard to find via search) |
| No point of contact resolved | Contact UUID doesn't match any entry in the contacts file |
| Contact not in Active Directory | Contact email isn't found in USFWS AD (requires ldap3 + AD config) |
| Contact has no email | Contact exists but has no email to check |
| No file paths recorded | Data asset has metadata but no associated files |
| Restricted record — file still accessible | Record is flagged restricted but the file can actually be opened |
| File path not found on server | A non-restricted file path doesn't exist on the drive |
| Path points to a folder, not a file | A non-project record points to a directory instead of a file |
| DOI does not resolve | A DOI identifier doesn't resolve to a live webpage |
| No external identifiers | Record has no ScienceBase, DOI, or ServCat link |
The report is a CSV with one row per issue, with columns: record_id, title, resource_type, source_file, issue, and detail. Open it in Excel to sort and filter.
The script also prints a summary to the terminal:
Issue summary:
1052 No external identifiers (ScienceBase, DOI, ServCat)
752 No file paths recorded
580 File path not found on server
342 Missing time period
338 No point of contact resolved
254 Path points to a folder, not a file
148 Missing abstract
133 Contact not found in USFWS Active Directory
20 Contact has no email — cannot check Active Directory
14 DOI does not resolve
5 No keywords
The AD check verifies that each record's point-of-contact email exists in the USFWS Active Directory.
If ldap3 isn't installed, or the connection fails, the AD check is skipped and all other checks still run. Check with IT for the correct LDAP server and authentication method.
Note: DOI checks make a live web request for each unique DOI and are rate-limited to be polite. A repository with many DOIs may take several minutes.
Scans the repository for new, changed, and deleted data and checks data-governance integrity: when data changes in a project, did its metadata record and the project's changelog.txt also get updated?
Unlike the other two utility scripts, this one scans the file system directly and does not require catalog_app.py to be running.
The tool works in two steps. First you build a baseline snapshot of the repository; later you compare the current state against that snapshot to see what changed.
Step 1 — build the baseline (run once to start):
python change_monitor.py "<repo_path>" --snapshotStep 2 — check for changes (run anytime after):
python change_monitor.py "<repo_path>"The snapshot is saved to repo_snapshot.json in the current directory. After you've reviewed a set of changes and want to reset the baseline to the current state, just run the --snapshot command again.
For every project folder, the monitor tracks four things:
| Item | Location |
|---|---|
| Data files / collections | The main project folders (excluding metadata/ and incoming/) |
| Metadata files | The metadata/ subfolder (mdEditor JSON) |
| Incoming data | The incoming/ subfolder |
| Changelog | changelog.txt in the project root |
When data is added or changed in a project's main folders, the monitor checks whether the metadata and changelog were updated to match, and reports one of:
| Result | Meaning |
|---|---|
| ✓ documented | Data changed, and both the metadata and changelog.txt were also updated — a properly documented change, no action needed |
| ⚠ needs attention | Data changed but the metadata and/or changelog.txt were not updated — the report explains exactly what's missing and suggests how to fix it |
Because only the data manager can write to the main folders, a "needs attention" flag is effectively a reminder that you changed data and may have forgotten to update the metadata or changelog.
The incoming/ folder is where project PIs deposit new data for the data manager to review. Changes there are expected, so they are reported as informational awareness flags, never as integrity errors:
| Flag | Meaning |
|---|---|
| 📥 NEW in incoming/ | A PI added new data awaiting review |
| 📥 CHANGED in incoming/ | A PI modified data already in incoming/ |
| 📤 REMOVED from incoming/ | Data left incoming/ — likely moved into the project after review |
All file types are tracked in incoming/ (not just recognized data formats), so nothing a PI deposits is missed.
Folders containing more than 50 files (for example, a collection of thousands of photos) are automatically rolled up into a single collection entry that records the file count, total size, and most recent modification time. This keeps the scan fast and the report readable — a 12,000-photo folder appears as one line rather than 12,000. If files are added, removed, or modified in a collection, it is flagged as a changed collection with a note about how many files were added or removed.
By default the monitor uses each file's size and modification time to detect change, which is fast even over the network because it does not read file contents. For byte-level certainty on a particular run, add the --hash flag (slower, as it reads every file):
python change_monitor.py "<repo_path>" --snapshot --hash
python change_monitor.py "<repo_path>" --hashThe monitor prints a summary to the terminal — highlighting the items that need attention and the incoming/ activity — and writes a full report to change_report.csv with columns: project, category, item, detail, and suggestion.
Python 3.8+ standard library only — no additional packages needed.
Note:
repo_snapshot.jsonandchange_report.csvare local working files and are excluded from version control via.gitignore.
The startup commands are a bit long to type each time. If you'd like a one-click launcher, you can create a small Windows batch file on your own computer.
This is entirely optional and kept local — batch files are intentionally excluded from the shared GitHub repository (via .gitignore) because they contain machine-specific paths and some networks flag .bat downloads.
To make one, open a text editor, paste the following (editing the paths to match your computer), and save it as start_catalog.bat in your project folder:
@echo off
title Alaska Regional Data Catalog
cd /d "C:\Users\YOURNAME\YOURPATH\RDR_Catalog"
start "" "http://localhost:8000"
python catalog_app.py "\\YOUR-SERVER\path\to\data" "\\YOUR-SERVER\path\to\contacts.json"Double-clicking it will open your browser and start the Catalog. Because it lives only on your machine, it never affects the shared code.
Alongside these Python tools, a production-ready proof of concept was developed using Drupal — the same content management system that runs fws.gov. This demonstrates what a publicly accessible version of the portal would look like deployed on official FWS web infrastructure.
1. Public portal theme (fws_data_portal) — an fws.gov-styled theme with multi-source search, tabbed results, dataset detail pages, downloads, citations, and Section 508 accessibility.
2. ServCat proxy module (fws_servcat_proxy) — proxies the FWS ServCat API server-side (avoiding CORS issues), with caching and three endpoints for search, record detail, and download.
3. Private data proxy module (fws_private_data_proxy) — scans the Alaska network drive, parses mdEditor metadata, syncs records into Drupal via a drush alaska:sync command, gates downloads behind FWS Active Directory (SAML), streams files securely, and logs downloads for audit.
4. Drupal Twig templates — the full template set for dataset cards, detail pages with multi-format downloads and auth gating, search results, and pagination.
The Python tools and the Drupal modules do the same jobs in different environments:
| Function | Python (now) | Drupal (production) |
|---|---|---|
| Metadata parsing | catalog_app.py |
MdJsonParser.php |
| File scanning | catalog_app.py |
FileSystemScanner.php |
| Search API | FastAPI /api/catalog |
Drupal Views + REST routes |
| Web interface | Flask / Jinja2 / HTML | Drupal theme + Twig |
| Authentication | Local session | FWS Active Directory (SAML) |
| File downloads | Python streaming | BinaryFileResponse (PHP) |
| Hosting | A staff laptop | FWS web servers |
Nothing built here is throwaway work. The Python version is a fully functional prototype proving the concept works with real Alaska Region data. The Drupal version is the production architecture, ready for deployment when organizational approval is obtained.
- ✅ Catalog running locally against the real network drive
- ✅ 1,690+ Alaska Region records indexed from mdEditor metadata
- ✅ Maps, contacts, smart downloads, project navigation
- ✅ Quality and restricted-access audit scripts
- ✅ Public Portal prototype with access gating
- Deploy the Catalog on an internal FWS server (staff-wide, no public access)
- Connect to FWS Active Directory for real authentication
- Gather staff feedback; expand to other data programs
- Work with the FWS Web team to deploy the Drupal modules
- Mount the Alaska network drive on the Drupal server
- Schedule automatic metadata syncing
- Implement SAML authentication
- Review records for public-release eligibility
- Coordinate with Communications and Regional leadership
- Complete Section 508 accessibility review
- Launch under fws.gov/alaska
┌───────────────────────────────────────────────────────────┐
│ Staff Browser │
│ Catalog: localhost:8000 · Portal: localhost:5000 │
└───────────────┬────────────────────────┬──────────────────┘
│ │
┌───────────────▼──────────┐ ┌──────────▼──────────────────┐
│ Catalog (catalog_app) │ │ Data Portal (app.py) │
│ localhost:8000 │◄─┤ localhost:5000 │
│ │ │ │
│ • Scans network drive │ │ • Public-facing interface │
│ • Parses mdEditor JSON │ │ • Public/internal gating │
│ • Serves /api/catalog │ │ • Also searches GBIF │
│ • /download endpoint │ │ • Reads from the Catalog │
└───────────────┬──────────┘ └─────────────────────────────┘
│
│ reads (read-only)
┌───────────────▼───────────────────────────────────────────┐
│ FWS Alaska Region Network Server │
│ \\YOUR-SERVER\path\to\data │
│ │
│ ProjectFolder/ │
│ ├── metadata/ ← mdEditor JSON metadata │
│ └── data/ ← CSV, Excel, Shapefile, PDF, etc. │
└───────────────────────────────────────────────────────────┘
Utility scripts (scan_restricted.py, metadata_quality_report.py)
read from the running Catalog's API at localhost:8000.
Do I need to be a programmer to use this?
No. Install Miniforge from the Company Portal once and copy config.example.py to config.py. After that you start the Catalog with a single command and do everything else in your web browser. If typing the command each time is a hassle, you can make an optional local shortcut (see Optional: a Local Shortcut).
Does this require any new data entry? No. The tools read your existing mdEditor metadata directly. The richer your mdEditor records, the more useful the Catalog — but there's no separate catalog to maintain.
Does this put our data on the internet? No. The Catalog and Portal run only on your computer and are only accessible to you. A public deployment would require FWS IT involvement and leadership approval first.
What happens to restricted or draft records? In the Catalog (staff tool), restricted records appear but their files show a 🔒 lock instead of a download. In the public Portal, restricted and draft records are hidden entirely from users who aren't logged in.
Where does the code live? On GitHub at https://github.com/USFWS/RDR_Catalog. Download it as a ZIP or clone it with Git.
Can this work for other Alaska Region programs?
Yes. Point the Catalog at any folder of projects that follow the standard structure (a metadata/ subfolder with mdEditor JSON files).
What if our mdEditor metadata is incomplete? The tools handle missing fields gracefully — records still appear, just with less detail. The Metadata Quality Report helps you find and fix gaps systematically.
Does this replace ServCat? No. ServCat remains FWS's authoritative repository for published references. These tools complement it by surfacing unpublished project data and works in progress, and they link to ServCat records where they exist.
How current is the Catalog? The Catalog builds its index when it starts. If new projects are added to the drive, restart the Catalog to pick them up. A production deployment would refresh automatically.
What does this cost? Nothing in software. The prototype uses only free, open-source tools (Python, FastAPI, Flask). A production Drupal deployment would use existing FWS web infrastructure with no licensing fees.
Contact the project maintainer for help with this repository.
Contact the project maintainer for information about contributing to this repository. Submit a GitHub Issue to report a bug or request a feature or enhancement.
Built by FWS Alaska Region · Prototype developed 2026 Code: https://github.com/USFWS/RDR_Catalog · Contact: Alaska Region Data Management