diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index b5de748b5..a5583e10b 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -29,6 +29,15 @@ jobs: name: bogota-artifacts-${{ github.sha }} path: data/Bogota retention-days: 7 + # The build re-pins artifacts.json to the names it just built, so the + # committed manifest is stale for dated artifacts (otp graphs, gtfs, + # elevation). Downstream jobs need this copy, not the checked-out one. + - name: Upload artifact manifest + uses: actions/upload-artifact@v5 + with: + name: bogota-manifest-${{ github.sha }} + path: builds/Bogota/artifacts.json + retention-days: 7 build_images: name: Build service images runs-on: ubuntu-latest @@ -126,6 +135,11 @@ jobs: with: name: bogota-artifacts-${{ github.sha }} path: data/Bogota + - name: Download artifact manifest + uses: actions/download-artifact@v6 + with: + name: bogota-manifest-${{ github.sha }} + path: builds/Bogota - name: Download Docker images uses: actions/download-artifact@v6 with: @@ -164,6 +178,11 @@ jobs: with: name: bogota-artifacts-${{ github.sha }} path: data/Bogota + - name: Download artifact manifest + uses: actions/download-artifact@v6 + with: + name: bogota-manifest-${{ github.sha }} + path: builds/Bogota - name: Download Docker images uses: actions/download-artifact@v6 with: @@ -171,10 +190,6 @@ jobs: path: . - name: Load Docker images run: gunzip -c docker-images.tar.gz | docker load - # The build writes dated transit artifacts; link the latest to the names - # docker compose serves them from (as run-integration-tests.sh does). - - name: Link transit artifacts - run: TRANSIT_DATA_ROOT=./data bin/link-latest-transit builds/Bogota - name: Start services run: bin/start-services --no-follow-logs builds/Bogota - name: Wait for services diff --git a/BUILD.md b/BUILD.md index a9077692a..e3f546e3c 100644 --- a/BUILD.md +++ b/BUILD.md @@ -44,14 +44,12 @@ This approach will download all the mapping data you need automatically, but onl 1. Pick a metro area from the list above, like "Amsterdam" or "Denver". These values are case-sensitive. In all the examples, replace "Amsterdam" with your metro area of choice. 2. Configuration is managed per build directory in `builds/`. Copy a template build directory: `cp -r builds/Bogota builds/Amsterdam`, review and edit `builds/Amsterdam/.env`. Bogota is configured for transit routing, so unless you're setting that up too (step 4), delete the copied `builds/Amsterdam/transit` directory and unset `HEADWAY_ENABLE_TRANSIT_ROUTING`. 3. Execute `bin/build builds/Amsterdam` to build data artifacts -4. (Optional) Set up transit routing. Note: This dramatically increases hardware requirements for large metro areas. +4. (Optional) Set up transit routing. Note: This increases hosting requirements for large metro areas - you'll want at least 4GB RAM extra for a medium sized city's transit service. 1. Find nearby transit schedules by running `bin/export-nearby-transit-feeds builds/Amsterdam` - 2. Examine `builds/Amsterdam/transit/gtfs-feeds/Amsterdam.gtfs_feeds.csv` and manually edit it if necessary to curate GTFS feeds. Some may have errors, and many may be useless for your purposes. + 2. Examine `builds/Amsterdam/transit/gtfs-feeds/amsterdam.gtfs_feeds.csv` and manually edit it if necessary to curate GTFS feeds. Some may have errors, and many may be useless for your purposes. 3. Build transit routing with `bin/build-transit builds/Amsterdam` - 4. Transit artifacts are built with dated names, so link the ones to serve: `TRANSIT_DATA_ROOT=./data bin/link-latest-transit builds/Amsterdam`. Re-run this after each `bin/build-transit`. -5. Start services from the build directory by running: `cd builds/Amsterdam && docker compose -f ../../docker-compose.yaml up -d`. With transit routing, use `../../docker-compose-with-transit.yaml` instead - or `bin/start-services builds/Amsterdam`, which picks the compose file based on `HEADWAY_ENABLE_TRANSIT_ROUTING`. -6. This will bring up the headway stack with a web frontend on port 8080. -7. (For https and non-default port use only) reverse-proxy traffic to port 8080. +5. Run `bin/start-services builds/Amsterdam`. This will bring up the Headway stack with a web frontend on port 8080. + 1. (Optional for https and non-default port use only) reverse-proxy traffic to port 8080. That's it! @@ -65,9 +63,17 @@ The process is largely the same as above. After downloading your OSM extract, mo ## Docker-compose restarts -Because Headway's docker-compose configuration uses init containers to populate a docker volume containing internal data, rebuilding the data for a metro area won't update existing containers. It's necessary to run `docker compose down --volumes` to re-initialize the data in the init containers. +Rebuilding the data for a metro area won't update existing containers. -When running docker compose commands, make sure to run them from the build directory (e.g., `cd builds/Amsterdam && docker compose -f ../../docker-compose.yaml down --volumes`) so they use the correct `.env` file, or use the integration test helper: `tests/integration/stop-and-remove-services builds/Amsterdam` +``` +# delete all existing docker data volumes and containers +bin/stop-and-remove-services builds/Amsterdam +# start the services again, which will pull in fresh data +bin/start-services builds/Amsterdam + +# or both in a single command +bin/reset-services builds/Amsterdam +``` This is necessary whenever you rebuild the data for a metro area, or change which area you're serving data for in the `builds//.env` file. diff --git a/bin/_compose-file.sh b/bin/_compose-file.sh index 838898277..2e48cfe47 100644 --- a/bin/_compose-file.sh +++ b/bin/_compose-file.sh @@ -15,3 +15,6 @@ if [ -z "${COMPOSE_FILE:-}" ]; then COMPOSE_FILE=docker-compose.yaml fi fi + +export HEADWAY_OTP_GRAPH_FILE=$(bin/artifacts otp-graph "$CONFIG_DIR") +export HEADWAY_ELEVATION_FILE=$(bin/artifacts elevation "$CONFIG_DIR") diff --git a/bin/artifacts b/bin/artifacts new file mode 100755 index 000000000..f780e4b1c --- /dev/null +++ b/bin/artifacts @@ -0,0 +1,180 @@ +#!/bin/bash + +set -e + +APP_ROOT=$(git rev-parse --show-toplevel) +cd "$APP_ROOT" + +function usage() { + cat << EOF +Pin and look up build artifacts. This runs on the build machine: --pin reads +data/, and the lookups exist for the local docker compose stack. Anything +that generates deploy config reads builds//artifacts.json directly. + +usage: $0 + $0 + $0 --pin + $0 --verify + +examples: # pin the newest artifacts to builds//artifacts.json + \$0 --pin builds/Amsterdam + # check every pinned artifact is present in data/ + \$0 --verify builds/Amsterdam + # what the local compose stack mounts + \$0 otp-graph builds/planet +EOF +} + +COMMAND="${1:-}" +CONFIG_DIR="${2:-}" +if [ -z "$COMMAND" ] || [ -z "$CONFIG_DIR" ]; then + usage + exit 1 +fi + +if [ ! -d "$CONFIG_DIR" ]; then + echo "Error: config dir \"$CONFIG_DIR\" doesn't exist" >&2 + exit 1 +fi + +source bin/_source-env.sh "$CONFIG_DIR" + +set -o nounset + +MANIFEST="${CONFIG_DIR}/artifacts.json" + +if ! command -v jq > /dev/null; then + echo "Error: jq is required to read ${MANIFEST}" >&2 + exit 1 +fi + +case "$COMMAND" in + --pin) + # Generate artifacts.json from what's in data/, pinning the newest of each dated artifact. + AREA_DIR="data/${HEADWAY_AREA}" + TRANSIT_DIR="${AREA_DIR}/transit" + MANIFEST="${CONFIG_DIR}/artifacts.json" + + if [ ! -d "$AREA_DIR" ]; then + echo "Error: nothing built at ${AREA_DIR}" >&2 + exit 1 + fi + + manifest="{}" + + function add_if_present() { + local key="$1" file="$2" + if [ -f "${AREA_DIR}/${file}" ]; then + manifest=$(jq --arg key "$key" --arg file "$file" '.[$key] = $file' <<< "$manifest") + fi + } + + add_if_present valhalla "${HEADWAY_AREA}.valhalla.tar.zst" + add_if_present elasticsearch "${HEADWAY_AREA}.elasticsearch.tar.zst" + add_if_present placeholder "${HEADWAY_AREA}.placeholder.tar.zst" + add_if_present pmtiles "${HEADWAY_AREA}.pmtiles" + add_if_present osm "${HEADWAY_AREA}.osm.pbf" + + shared="{}" + for file in terrain.mbtiles landcover.mbtiles; do + if [ -f "${AREA_DIR}/${file}" ]; then + shared=$(jq --arg key "${file%.mbtiles}" --arg file "$file" '.[$key] = $file' <<< "$shared") + fi + done + if [ "$shared" != "{}" ]; then + manifest=$(jq --argjson shared "$shared" '.shared = $shared' <<< "$manifest") + fi + + # Dated names sort oldest to newest, so the last write wins. + function add_dated_by_zone() { + local key="$1" suffix="$2" + local found="{}" path file rest + + for path in "${TRANSIT_DIR}/${HEADWAY_AREA}-"*"$suffix"; do + [ -e "$path" ] || continue + file=$(basename "$path") + + rest=${file#"${HEADWAY_AREA}-"} + rest=${rest%"$suffix"} + if [[ ! $rest =~ ^(.+)-([0-9]{4}-[0-9]{2}-[0-9]{2})$ ]]; then + echo "Error: unexpected ${suffix} name: ${file}" >&2 + exit 1 + fi + found=$(jq --arg zone "${BASH_REMATCH[1]}" --arg file "transit/$file" '.[$zone] = $file' <<< "$found") + done + + if [ "$found" != "{}" ]; then + manifest=$(jq --arg key "$key" --argjson found "$found" '.[$key] = $found' <<< "$manifest") + fi + } + + add_dated_by_zone otp-graphs .graph.obj.zst + add_dated_by_zone gtfs .gtfs.tar.zst + + elevation="" + for path in "${TRANSIT_DIR}/${HEADWAY_AREA}-"*.elevation-tifs.tar.zst; do + [ -e "$path" ] || continue + elevation=$(basename "$path") + done + if [ -n "$elevation" ]; then + manifest=$(jq --arg file "transit/$elevation" '.elevation = $file' <<< "$manifest") + fi + + jq -S . <<< "$manifest" > "$MANIFEST" + + echo "Wrote ${MANIFEST}:" >&2 + cat "$MANIFEST" >&2 + ;; + --verify) + # Ensure all pinned assets are present locally + AREA_DIR="data/${HEADWAY_AREA}" + + if [ ! -f "$MANIFEST" ]; then + echo "Error: no ${MANIFEST}; run bin/build ${CONFIG_DIR}" >&2 + exit 1 + fi + + missing=() + while IFS= read -r file; do + [ -n "$file" ] || continue + [ -f "${AREA_DIR}/${file}" ] || missing+=("$file") + done < <(jq -r '.. | strings' "$MANIFEST") + + if [ ${#missing[@]} -gt 0 ]; then + { + echo "Error: ${MANIFEST} pins ${#missing[@]} artifact(s) missing from ${AREA_DIR}:" + printf ' %s\n' "${missing[@]}" + echo + echo "The pinned names are stale, or the build didn't finish. Either rebuild" + echo "(bin/build ${CONFIG_DIR}), or re-pin what's already built" + echo "(bin/artifacts --pin ${CONFIG_DIR})." + } >&2 + exit 1 + fi + + echo "All artifacts pinned by ${MANIFEST} are present in ${AREA_DIR}." + ;; + otp-graph) + [ -f "$MANIFEST" ] || exit 0 + # docker compose runs a single OTP, so serve the first zone + jq -r 'first(."otp-graphs"[]?) // empty' "$MANIFEST" + ;; + elevation) + [ -f "$MANIFEST" ] || exit 0 + jq -r '.elevation // empty' "$MANIFEST" + ;; + files) + # All artifacts for the deployment except `shared`, + # which is excluded because those publish to a different prefix. + [ -f "$MANIFEST" ] || exit 0 + jq -r 'del(.shared) | .. | strings' "$MANIFEST" + ;; + shared-files) + [ -f "$MANIFEST" ] || exit 0 + jq -r '.shared // {} | .[]' "$MANIFEST" + ;; + *) + usage + exit 1 + ;; +esac diff --git a/bin/build b/bin/build index 43774a6d3..cc62907af 100755 --- a/bin/build +++ b/bin/build @@ -45,4 +45,4 @@ if [ -d "$TRANSIT_CONFIG_DIR" ]; then bin/build-transit "$CONFIG_DIR" fi -echo "Next, run: git co . && ./bin/k8s-regenerate-all" +echo "Next, run: ./bin/revert-fetch-urls && ./bin/k8s-regenerate-all" diff --git a/bin/build-transit b/bin/build-transit index af6f4e094..e47c3f21d 100755 --- a/bin/build-transit +++ b/bin/build-transit @@ -42,6 +42,32 @@ fi TRANSIT_CONFIG_DIR="$CONFIG_DIR/transit" +# Each file in gtfs-feeds/ is built into a zone named after the file, and that +# name is used verbatim from here on: it keys artifacts.json and it names the OTP +# deployment, service and config map in k8s. So it has to be a valid k8s object +# name - cheap to check, expensive to discover after a multi-hour build. +function lint_zone_names() { + set +x + local found=0 zone file + for file in "${TRANSIT_CONFIG_DIR}"/gtfs-feeds/*.gtfs_feeds.csv; do + [ -e "$file" ] || continue + found=1 + zone=$(basename "$file" .gtfs_feeds.csv) + # 63 chars is the k8s limit, minus room for the "opentripplanner-" prefix. + if [[ ! $zone =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ ]] || [ ${#zone} -gt 40 ]; then + echo "Error: ${file}: zone \"${zone}\" is not a valid k8s object name; rename it to lowercase letters, digits and dashes" >&2 + exit 1 + fi + done + + if [ "$found" -eq 0 ]; then + echo "Error: no *.gtfs_feeds.csv in ${TRANSIT_CONFIG_DIR}/gtfs-feeds - nothing to build" >&2 + exit 1 + fi + set -x +} +lint_zone_names + USE_LOCAL_PBF="" INPUT_PBF="${HEADWAY_AREA}.osm.pbf" if [[ -f $INPUT_PBF ]]; then @@ -50,14 +76,17 @@ fi dagger -c "with-area ${HEADWAY_AREA} ${USE_LOCAL_PBF} | build-transit ${TRANSIT_CONFIG_DIR} ${GTFS_API_KEYS} | export 'data/${HEADWAY_AREA}/transit'" +bin/artifacts --pin "$CONFIG_DIR" + set +x cat <&2 +} + if [ -z "$HEADWAY_DATA_TAG" ]; then source "bin/_headway_version.sh" fi @@ -40,12 +44,19 @@ set -a source ".bin-env" set +a +MANIFEST="${CONFIG_DIR}/artifacts.json" +if [ ! -f "$MANIFEST" ]; then + log "missing ${MANIFEST}" + log 'run `bin/artifacts --pin` on the build machine' + exit 1 +fi + EXISTING_DEPLOYMENT_CONFIG="${OUTPUT_DIR}/deployment-config.yaml" PELIAS_CONFIG_FILE="data/${HEADWAY_AREA}/${HEADWAY_AREA}.pelias.json" if [ -f "$PELIAS_CONFIG_FILE" ]; then PELIAS_CONFIG_JSON=$(cat $PELIAS_CONFIG_FILE) elif [ -f "$EXISTING_DEPLOYMENT_CONFIG" ]; then - echo "Warning: ${PELIAS_CONFIG_FILE} not found; extracting pelias config from existing ${EXISTING_DEPLOYMENT_CONFIG}" + log "Warning: ${PELIAS_CONFIG_FILE} not found; extracting pelias config from existing ${EXISTING_DEPLOYMENT_CONFIG}" # Grab the `pelias-config-json: |` block scalar and strip its 4 space indent. PELIAS_CONFIG_JSON=$(awk ' in_block && NF && !/^ / { exit } @@ -53,7 +64,7 @@ elif [ -f "$EXISTING_DEPLOYMENT_CONFIG" ]; then /^ pelias-config-json: \|$/ { in_block = 1 } ' "$EXISTING_DEPLOYMENT_CONFIG") else - echo "Missing pelias config file: ${PELIAS_CONFIG_FILE} and no existing rendered output to fall back on" + log "Missing pelias config file: ${PELIAS_CONFIG_FILE} and no existing rendered output to fall back on" exit 1 fi @@ -61,62 +72,57 @@ indent4() { while IFS= read -r line; do printf ' %s\n' "$line"; done; } export PELIAS_CONFIG_JSON_YAML="| $(echo "$PELIAS_CONFIG_JSON" | indent4)" +# Render to a staging dir and swap once everything succeeds +STAGING_DIR=$(mktemp -d) +trap 'rm -rf "$STAGING_DIR"' EXIT + function render_template() { local template_file=$1 local output="${2:-$(basename $template_file | sed 's/\.tpl$//')}" - envsubst < "$template_file" > "${OUTPUT_DIR}/${output}" + envsubst < "$template_file" > "${STAGING_DIR}/${output}" } set -o nounset -mkdir -p "$OUTPUT_DIR" -rm -f "${OUTPUT_DIR}"/* +export ELEVATION_ARTIFACT_URL="" +elevation_path=$(jq -r '.elevation // empty' "$MANIFEST") +if [ -n "$elevation_path" ]; then + ELEVATION_ARTIFACT_URL="${HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT}/${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}/${elevation_path}" +fi function generate_transit() { local otp_endpoints="" local separator=", " - for headway_transit_area in $HEADWAY_TRANSIT_AREAS; do - local otp_endpoint_name=$(generate_otp $headway_transit_area); + local transit_zone graph_path otp_endpoint_name + while IFS=$'\t' read -r transit_zone graph_path; do + otp_endpoint_name=$(generate_otp "$transit_zone" "$graph_path") otp_endpoints+="${separator}\"http://${otp_endpoint_name}:8000\"" - done + done < <(jq -r '."otp-graphs" // {} | to_entries[] | [.key, .value] | @tsv' "$MANIFEST") export OTP_ENDPOINTS_JSON="[\"http://valhalla:8002\"${otp_endpoints}]" render_template "k8s/_template/travelmux-deployment.yaml.tpl" } -function log() { - echo $@ 1>&2 -} - function generate_otp() { - local headway_transit_area=$1 - if [ -z "$headway_transit_area" ]; then - echo "must specify headway_transit_area to $0" - exit 1 - fi + export TRANSIT_ZONE=$1 + local graph_path=$2 - # Lowercase transit area for idiomatic k8s names - export TRANSIT_AREA=$(echo $headway_transit_area | tr '[:upper:]' '[:lower:]') - export OTP_GRAPH_URL="${HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT}/${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}/${headway_transit_area}.graph.obj.zst" - export OTP_ENDPOINT_NAME="opentripplanner-${TRANSIT_AREA}" + export OTP_GRAPH_URL="${HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT}/${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}/${graph_path}" + export OTP_ENDPOINT_NAME="opentripplanner-${TRANSIT_ZONE}" - local router_config_file="${CONFIG_DIR}/transit/${headway_transit_area}-router-config.json" + local router_config_file="${CONFIG_DIR}/transit/${TRANSIT_ZONE}-router-config.json" if [ -f "$router_config_file" ]; then + router_config_json=$(cat "$router_config_file") export OTP_ROUTER_CONFIG_JSON_YAML="| -$(indent4 < "$router_config_file")" +$(indent4 <<< "$router_config_json")" else - # blank keys or "null" keys don't appear in the config map - # which causes an error in pod creation export OTP_ROUTER_CONFIG_JSON_YAML='""' fi - export OTP_CONFIG_JSON_YAML="| -$(indent4 < "services/otp/otp-config.json")" - for file in k8s/_template/opentripplanner*.tpl; do - output=$(basename "$file" | sed 's/\.tpl$//' | sed "s/AREA/${TRANSIT_AREA}/") + output=$(basename "$file" | sed 's/\.tpl$//' | sed "s/ZONE/${TRANSIT_ZONE}/") render_template "$file" "$output" done echo "$OTP_ENDPOINT_NAME" @@ -140,3 +146,8 @@ do render_template $file done +# Everything rendered - now it's safe to replace the previous output. +mkdir -p "$OUTPUT_DIR" +rm -f "${OUTPUT_DIR}"/* +cp "${STAGING_DIR}"/* "${OUTPUT_DIR}/" + diff --git a/bin/link-latest-transit b/bin/link-latest-transit deleted file mode 100755 index 6830d9cec..000000000 --- a/bin/link-latest-transit +++ /dev/null @@ -1,150 +0,0 @@ -#!/bin/bash -e - -function usage() { - cat << EOF -Link latest transit files in web root -usage: $0 -examples: - $0 builds/planet - -Use a version prefix other than what's in bin/_headway_version.sh - HEADWAY_DATA_TAG=dev $0 builds/Seattle - -For local hosting - TRANSIT_DATA_ROOT="./data" $0 builds/Seattle -EOF -} - -CONFIG_DIR="$1" -if [ -z "$CONFIG_DIR" ]; then - usage - exit 1 -fi - -REPO_ROOT=$(git rev-parse --show-toplevel) - -source bin/_source-env.sh "$CONFIG_DIR" - -if [ -z "$TRANSIT_DATA_ROOT" ]; then - if [ -z "$HEADWAY_DATA_TAG" ]; then - source "${REPO_ROOT}/bin/_headway_version.sh" - fi - OUTPUT_ROOT="/pool1/www/data.maps.earth" - TRANSIT_DATA_DIR="${OUTPUT_ROOT}/${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}" -else - TRANSIT_DATA_DIR="${TRANSIT_DATA_ROOT}/${HEADWAY_AREA}" -fi - -snake_to_upper_camel() { - local input="$1" - local result="" - - # Split the input string by underscores - IFS='_' read -ra words <<< "$input" - - # Capitalize the first letter of each word and remove underscores - for word in "${words[@]}"; do - # Capitalize the first letter of the word - capitalized=$(tr '[:lower:]' '[:upper:]' <<< "${word:0:1}") - # Remove the first letter and append to the result - result+="$capitalized${word:1}" - done - - echo "$result" -} - - -transit_cities_duped() { - # Iterate over the files (check for `-` to exclude the final symlinks) - find . -name \*graph.obj.zst -type f | while read file; do - local regex=".*-([a-zA-Z_]+)-([0-9]{4}-[0-9]{2}-[0-9]{2}).graph.obj.zst$" - if [[ $file =~ $regex ]]; then - city_name="${BASH_REMATCH[1]}" - echo $city_name - else - echo "unexpected graph name: $file" >&2 - exit 1 - fi - done -} - -elevation_duped() { - for file in $(all_elevation_files); do - local regex="\./(.+)-([0-9]{4}-[0-9]{2}-[0-9]{2}).elevation-tifs.tar.zst$" - if [[ $file =~ $regex ]]; then - archive_name="${BASH_REMATCH[1]}" - echo $archive_name - else - echo "unexpected elevation name: $file" >&2 - exit 1 - fi - done -} - -all_transit_graphs() { - local city=$1 - if [ -z "$city" ]; then - echo "city was blank" - exit 1 - fi - find . -name \*.graph.obj.zst -type f | grep "$city" -} - -most_recent_transit_graph() { - all_transit_graphs $1 | sort | tail -n 1 -} - -all_elevation_files() { - find . -name \*elevation-tifs.tar.zst -type f -} - -most_recent_elevation_file() { - all_elevation_files | sort | tail -n 1 -} - -indent() { - local spaces="${1:-2}" - local prefix=$(printf "%*s" "$spaces" "") - sed "s/^/$prefix/" - } - -main() { - cd $TRANSIT_DATA_DIR - echo "PWD: $(pwd)" - - # Handle transit graphs - local transit_cities=$(transit_cities_duped | uniq) - for city in $transit_cities; do - echo "Transit city: $city" - printf "all graphs:\n$(all_transit_graphs "$city" | indent)\n" | indent - - local most_recent=$(most_recent_transit_graph "$city") - echo "most recent graph: $most_recent" | indent - - local link_name=$(snake_to_upper_camel "$city").graph.obj.zst - echo "link_name: $link_name" | indent - - ln -sf "$most_recent" "$link_name" - done - - # Handle elevation files - local elevation_archives=$(elevation_duped | uniq) - for archive in $elevation_archives; do - echo "Elevation archive: $archive" - printf "all elevation files:\n$(all_elevation_files | indent)\n" | indent - - local most_recent_elevation=$(most_recent_elevation_file) - if [ -n "$most_recent_elevation" ]; then - echo "most recent elevation: $most_recent_elevation" | indent - - local elevation_link_name=$(basename "${archive}.elevation-tifs.tar.zst") - echo "elevation link_name: $elevation_link_name" | indent - - ln -sf "$most_recent_elevation" "$elevation_link_name" - else - echo "no elevation files found for $archive" | indent - fi - done -} - -main diff --git a/bin/pre-commit b/bin/pre-commit index 659231311..f31dc3603 100755 --- a/bin/pre-commit +++ b/bin/pre-commit @@ -37,7 +37,7 @@ should_run() { # Run checks (all in parallel, or only the named one) should_run "dagger" && run_job "dagger fmt" bash -c "cd dagger && go fmt" -should_run "frontend" && run_job "frontend checks" bash -c "cd services/frontend/www-app && yarn install && yarn fmt && yarn lint && yarn tsc && yarn test" +should_run "frontend" && run_job "frontend checks" bash -c "cd services/frontend/www-app && yarn install && yarn quasar prepare && yarn fmt && yarn lint && yarn tsc && yarn test" should_run "travelmux" && run_job "travelmux checks" bash -c "cd services/travelmux && cargo fmt && cargo clippy --all-features -- -D warnings && cargo test" should_run "gtfout" && run_job "gtfout checks" bash -c "cd services/gtfs/gtfout && cargo fmt --all && cargo clippy --all-targets --all-features -- -D warnings && cargo test --all-targets --all-features" should_run "pelias" && run_job "pelias config checks" bash -c "cd services/pelias/generate_config && yarn install && yarn fmt && yarn lint && yarn build" diff --git a/bin/publish-data b/bin/publish-data index a9c2382a0..f4db91eef 100755 --- a/bin/publish-data +++ b/bin/publish-data @@ -48,58 +48,56 @@ function kill_uploads() { trap kill_uploads EXIT function upload() { - local input_glob=$1 + local file=$1 local output_prefix=$2 - local input_path_glob="${DATA_ROOT}/${HEADWAY_AREA}/${input_glob}" - if ! ls $input_path_glob 1> /dev/null 2>&1; then - echo "target file doesn't exist: '${input_path_glob}'" + local input_path="${DATA_ROOT}/${HEADWAY_AREA}/${file}" + if [ ! -f "$input_path" ]; then + echo "pinned in ${MANIFEST} but not built here: ${file}" exit 1 fi - for input_path in $input_path_glob; do - local resource="${output_prefix}/$(basename "$input_path")" - local output_path="${OUTPUT_DIR}/${resource}" - - set -o nounset - if [ -z "$HOST" ]; then - echo "Copying ${input_path} -> ${output_path}" - mkdir -p "$(dirname $output_path)" - cp "$input_path" "$output_path" - else - echo "Sycning ${input_path} -> ${HOST}:${output_path}" - ssh "$HOST" mkdir -p "$(dirname $output_path)" - rsync -av --progress -e ssh "$input_path" "${HOST}:${output_path}" - fi - done + # The asset host mirrors data/, so a manifest value is the path here, + # under /bootstrap in compose, and in the URLs bin/k8s-generate renders. + local resource="${output_prefix}/${file}" + local output_path="${OUTPUT_DIR}/${resource}" + + set -o nounset + if [ -z "$HOST" ]; then + echo "Copying ${input_path} -> ${output_path}" + mkdir -p "$(dirname $output_path)" + cp "$input_path" "$output_path" + else + echo "Sycning ${input_path} -> ${HOST}:${output_path}" + ssh "$HOST" mkdir -p "$(dirname $output_path)" + rsync -av --progress -e ssh "$input_path" "${HOST}:${output_path}" + fi } source bin/_source-env.sh "$CONFIG_DIR" if [ -z "$HEADWAY_DATA_TAG" ]; then - explicit_data_tag=0 source "${REPO_ROOT}/bin/_headway_version.sh" -else - explicit_data_tag=1 fi export HEADWAY_DATA_TAG set -o nounset -upload "${HEADWAY_AREA}.valhalla.tar.zst" "${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}" -upload "${HEADWAY_AREA}.elasticsearch.tar.zst" "${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}" -upload "${HEADWAY_AREA}.pmtiles" "${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}" -upload "${HEADWAY_AREA}.placeholder.tar.zst" "${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}" -upload "${HEADWAY_AREA}.osm.pbf" "${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}" +MANIFEST="${CONFIG_DIR}/artifacts.json" +if [ ! -f "$MANIFEST" ]; then + echo "No pinned manifest at ${MANIFEST}" + echo "Run bin/artifacts --pin ${CONFIG_DIR}" + exit 1 +fi -# Upload transit -upload "transit/${HEADWAY_AREA}*.graph.obj.zst" "${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}" -upload "transit/${HEADWAY_AREA}*.gtfs.tar.zst" "${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}" -upload "transit/${HEADWAY_AREA}*.elevation-tifs.tar.zst" "${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}" +for file in $("${REPO_ROOT}/bin/artifacts" files "$CONFIG_DIR"); do + upload "$file" "${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}" +done -# These files are generic across all areas -upload "terrain.mbtiles" "${HEADWAY_DATA_TAG}" -upload "landcover.mbtiles" "${HEADWAY_DATA_TAG}" +# Generic across all areas, so they sit above the area directory. +for file in $("${REPO_ROOT}/bin/artifacts" shared-files "$CONFIG_DIR"); do + upload "$file" "${HEADWAY_DATA_TAG}" +done rets=() for pid in ${upload_pids[*]}; do @@ -113,11 +111,3 @@ for pid in ${upload_pids[*]}; do done echo "Return codes: ${rets[*]}" - -echo "Next, on $HOST..." -# Todo: just do it automatically -if [ "$explicit_data_tag" -eq 1 ]; then -echo "Do you want to 'HEADWAY_DATA_TAG=${HEADWAY_DATA_TAG} bin/link-latest-transit ${CONFIG_DIR}' next?" -else -echo "Do you want to 'bin/link-latest-transit ${CONFIG_DIR}' next?" -fi diff --git a/bin/revert-fetch-urls b/bin/revert-fetch-urls new file mode 100755 index 000000000..787c14dfa --- /dev/null +++ b/bin/revert-fetch-urls @@ -0,0 +1,6 @@ +#!/bin/bash +# Undo bin/update-fetch-urls, putting the placeholder artifact root back. +set -e + +APP_ROOT=$(git rev-parse --show-toplevel) +exec "$APP_ROOT/bin/update-fetch-urls" --revert diff --git a/bin/start-services b/bin/start-services index 3e67f0b33..ee8e0ec69 100755 --- a/bin/start-services +++ b/bin/start-services @@ -51,13 +51,14 @@ echo "Area: ${HEADWAY_AREA}" echo "Data directory: data/${HEADWAY_AREA}" echo "" -# Check if data directory exists if [ ! -d "data/${HEADWAY_AREA}" ]; then echo "Error: Data directory data/${HEADWAY_AREA} not found" echo "Please run: bin/build $CONFIG_DIR" exit 1 fi +bin/artifacts --verify "$CONFIG_DIR" + # Start services - cd to build dir so docker-compose picks up the .env cd "$CONFIG_DIR" echo "Starting docker compose services from $CONFIG_DIR..." diff --git a/bin/update-fetch-urls b/bin/update-fetch-urls index f0b86d668..4f6da21b3 100755 --- a/bin/update-fetch-urls +++ b/bin/update-fetch-urls @@ -1,21 +1,47 @@ #!/bin/bash +# Point the committed artifact URLs at the real asset host, so a build's +# rendered configs and feed URLs fetch from somewhere that exists. +# +# usage: $0 [--revert] +# +# This leaves the tree dirty. bin/revert-fetch-urls puts it back. + +set -e APP_ROOT=$(git rev-parse --show-toplevel) +REVERT=0 +if [ "${1:-}" = "--revert" ]; then + REVERT=1 +fi + +if [ ! -f "$APP_ROOT/.bin-env" ]; then + echo "Error: $APP_ROOT/.bin-env not found - see .bin-env-example" >&2 + exit 1 +fi + set -a -source $APP_ROOT/.bin-env +source "$APP_ROOT/.bin-env" set +a if [ -z "$HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT" ]; then - echo "set HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT in .bin-env" + echo "set HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT in .bin-env" >&2 exit 1 fi if [ -z "$DEPLOYMENT_ARTIFACT_ROOT" ]; then - echo "set DEPLOYMENT_ARTIFACT_ROOT in .bin-env" + echo "set DEPLOYMENT_ARTIFACT_ROOT in .bin-env" >&2 exit 1 fi +if [ "$REVERT" -eq 1 ]; then + FROM="$DEPLOYMENT_ARTIFACT_ROOT" + TO="$HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT" +else + FROM="$HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT" + TO="$DEPLOYMENT_ARTIFACT_ROOT" +fi + # Detect OS and set sed command if [[ "$OSTYPE" == "darwin"* ]]; then SED_CMD="sed -i ''" @@ -23,7 +49,6 @@ else SED_CMD="sed -i" fi -# Use bash -c to execute sed with the proper syntax -find $APP_ROOT/k8s/configs -type f -exec bash -c "$SED_CMD 's,${HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT},${DEPLOYMENT_ARTIFACT_ROOT},' \"\$1\"" _ {} \; -find $APP_ROOT/builds/*/transit -type f -exec bash -c "$SED_CMD 's,${HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT},${DEPLOYMENT_ARTIFACT_ROOT},' \"\$1\"" _ {} \; -bash -c "$SED_CMD 's,${HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT},${DEPLOYMENT_ARTIFACT_ROOT},' services/pelias/generate_config/src/Area.ts" +find "$APP_ROOT"/k8s/configs -type f -exec bash -c "$SED_CMD 's,${FROM},${TO},' \"\$1\"" _ {} \; +find "$APP_ROOT"/builds/*/transit -type f -exec bash -c "$SED_CMD 's,${FROM},${TO},' \"\$1\"" _ {} \; +bash -c "$SED_CMD 's,${FROM},${TO},' \"${APP_ROOT}/services/pelias/generate_config/src/Area.ts\"" diff --git a/builds/Bogota/.env b/builds/Bogota/.env index 18202e346..e1086533a 100644 --- a/builds/Bogota/.env +++ b/builds/Bogota/.env @@ -4,7 +4,6 @@ HEADWAY_COUNTRIES="CO" HEADWAY_PUBLIC_URL=http://localhost:8080 HEADWAY_SERVICE_PORT=30401 HEADWAY_ENABLE_TRANSIT_ROUTING=1 -HEADWAY_TRANSIT_AREAS=Bogota HEADWAY_BBOX="-74.2155 4.5199 -74.0078 4.7711" PELIAS_ELASTICSEARCH_MEMORY_REQUEST=750Mi VALHALLA_MEMORY_REQUEST=250Mi diff --git a/builds/Bogota/artifacts.json b/builds/Bogota/artifacts.json new file mode 100644 index 000000000..1f085818b --- /dev/null +++ b/builds/Bogota/artifacts.json @@ -0,0 +1,18 @@ +{ + "elasticsearch": "Bogota.elasticsearch.tar.zst", + "elevation": "transit/Bogota-2026-08-14.elevation-tifs.tar.zst", + "gtfs": { + "bogota": "transit/Bogota-bogota-2026-08-14.gtfs.tar.zst" + }, + "osm": "Bogota.osm.pbf", + "otp-graphs": { + "bogota": "transit/Bogota-bogota-2026-08-14.graph.obj.zst" + }, + "placeholder": "Bogota.placeholder.tar.zst", + "pmtiles": "Bogota.pmtiles", + "shared": { + "landcover": "landcover.mbtiles", + "terrain": "terrain.mbtiles" + }, + "valhalla": "Bogota.valhalla.tar.zst" +} diff --git a/builds/Bogota/transit/Bogota.gtfs_feeds.csv b/builds/Bogota/transit/Bogota.gtfs_feeds.csv deleted file mode 100644 index b44ea0457..000000000 --- a/builds/Bogota/transit/Bogota.gtfs_feeds.csv +++ /dev/null @@ -1,2 +0,0 @@ -mdb_source_id,data_type,entity_type,location.country_code,location.subdivision_name,location.municipality,provider,is_official,name,note,feed_contact_email,static_reference,urls.direct_download,urls.authentication_type,urls.authentication_info,urls.api_key_parameter_name,urls.latest,urls.license,location.bounding_box.minimum_latitude,location.bounding_box.maximum_latitude,location.bounding_box.minimum_longitude,location.bounding_box.maximum_longitude,location.bounding_box.extracted_on,status,features,redirect.id,redirect.comment -3358,gtfs,,CO,Distrito Capital de Bogotá,,SIMUR Bogota,True,,,,,https://storage.googleapis.com/gtfs-estaticos/GTFS-2026-04-29.zip,0,,,https://storage.googleapis.com/storage/v1/b/mdb-latest/o/co-distrito-capital-de-bogota-simur-bogota-gtfs-3358.zip?alt=media,http://www.simur.gov.co/gtfs,4.398140426179031,4.818429737670479,-74.21512003523206,-74.0058612942782,2026-07-10T16:14:04+00:00,,,, diff --git a/builds/Bogota/transit/gtfs-feeds/Bogota.gtfs_feeds.csv b/builds/Bogota/transit/gtfs-feeds/bogota.gtfs_feeds.csv similarity index 100% rename from builds/Bogota/transit/gtfs-feeds/Bogota.gtfs_feeds.csv rename to builds/Bogota/transit/gtfs-feeds/bogota.gtfs_feeds.csv diff --git a/builds/Seattle/.env b/builds/Seattle/.env index b272b1531..5987969dd 100644 --- a/builds/Seattle/.env +++ b/builds/Seattle/.env @@ -4,7 +4,6 @@ HEADWAY_COUNTRIES="US" HEADWAY_PUBLIC_URL=https://seattle.maps.earth HEADWAY_SERVICE_PORT=30401 HEADWAY_ENABLE_TRANSIT_ROUTING=1 -HEADWAY_TRANSIT_AREAS=Seattle HEADWAY_BBOX="-122.462 47.394 -122.005 47.831" PELIAS_ELASTICSEARCH_MEMORY_REQUEST=750Mi VALHALLA_MEMORY_REQUEST=250Mi diff --git a/builds/Seattle/artifacts.json b/builds/Seattle/artifacts.json new file mode 100644 index 000000000..d358da8e1 --- /dev/null +++ b/builds/Seattle/artifacts.json @@ -0,0 +1,18 @@ +{ + "elasticsearch": "Seattle.elasticsearch.tar.zst", + "elevation": "transit/Seattle-2026-08-14.elevation-tifs.tar.zst", + "gtfs": { + "seattle": "transit/Seattle-seattle-2026-08-14.gtfs.tar.zst" + }, + "osm": "Seattle.osm.pbf", + "otp-graphs": { + "seattle": "transit/Seattle-seattle-2026-08-14.graph.obj.zst" + }, + "placeholder": "Seattle.placeholder.tar.zst", + "pmtiles": "Seattle.pmtiles", + "shared": { + "landcover": "landcover.mbtiles", + "terrain": "terrain.mbtiles" + }, + "valhalla": "Seattle.valhalla.tar.zst" +} diff --git a/builds/Seattle/transit/gtfs-feeds/Seattle.gtfs_feeds.csv b/builds/Seattle/transit/gtfs-feeds/seattle.gtfs_feeds.csv similarity index 100% rename from builds/Seattle/transit/gtfs-feeds/Seattle.gtfs_feeds.csv rename to builds/Seattle/transit/gtfs-feeds/seattle.gtfs_feeds.csv diff --git a/builds/Seattle/transit/Seattle-router-config.json b/builds/Seattle/transit/seattle-router-config.json similarity index 100% rename from builds/Seattle/transit/Seattle-router-config.json rename to builds/Seattle/transit/seattle-router-config.json diff --git a/builds/planet/.env b/builds/planet/.env index cb98d4a55..551442ab3 100644 --- a/builds/planet/.env +++ b/builds/planet/.env @@ -7,7 +7,6 @@ HEADWAY_COUNTRIES="ALL" HEADWAY_PUBLIC_URL=https://maps.earth HEADWAY_SERVICE_PORT=30400 HEADWAY_ENABLE_TRANSIT_ROUTING=1 -HEADWAY_TRANSIT_AREAS="Barcelona LosAngeles PugetSound SfBayArea" PELIAS_ELASTICSEARCH_MEMORY_REQUEST=8Gi VALHALLA_MEMORY_REQUEST=4Gi HEADWAY_ABOUT_URL="https://about.maps.earth" diff --git a/builds/planet/artifacts.json b/builds/planet/artifacts.json new file mode 100644 index 000000000..d062ef8d4 --- /dev/null +++ b/builds/planet/artifacts.json @@ -0,0 +1,24 @@ +{ + "elasticsearch": "maps-earth-planet-v1.260727.elasticsearch.tar.zst", + "elevation": "transit/maps-earth-planet-v1.260727-2026-08-14.elevation-tifs.tar.zst", + "gtfs": { + "barcelona": "transit/maps-earth-planet-v1.260727-barcelona-2026-08-14.gtfs.tar.zst", + "los-angeles": "transit/maps-earth-planet-v1.260727-los-angeles-2026-08-14.gtfs.tar.zst", + "puget-sound": "transit/maps-earth-planet-v1.260727-puget-sound-2026-08-14.gtfs.tar.zst", + "sf-bay-area": "transit/maps-earth-planet-v1.260727-sf-bay-area-2026-08-14.gtfs.tar.zst" + }, + "osm": "maps-earth-planet-v1.260727.osm.pbf", + "otp-graphs": { + "barcelona": "transit/maps-earth-planet-v1.260727-barcelona-2026-08-14.graph.obj.zst", + "los-angeles": "transit/maps-earth-planet-v1.260727-los-angeles-2026-08-14.graph.obj.zst", + "puget-sound": "transit/maps-earth-planet-v1.260727-puget-sound-2026-08-14.graph.obj.zst", + "sf-bay-area": "transit/maps-earth-planet-v1.260727-sf-bay-area-2026-08-14.graph.obj.zst" + }, + "placeholder": "maps-earth-planet-v1.260727.placeholder.tar.zst", + "pmtiles": "maps-earth-planet-v1.260727.pmtiles", + "shared": { + "landcover": "landcover.mbtiles", + "terrain": "terrain.mbtiles" + }, + "valhalla": "maps-earth-planet-v1.260727.valhalla.tar.zst" +} diff --git a/builds/planet/transit/gtfs-feeds/los_angeles.gtfs_feeds.csv b/builds/planet/transit/gtfs-feeds/los-angeles.gtfs_feeds.csv similarity index 100% rename from builds/planet/transit/gtfs-feeds/los_angeles.gtfs_feeds.csv rename to builds/planet/transit/gtfs-feeds/los-angeles.gtfs_feeds.csv diff --git a/builds/planet/transit/gtfs-feeds/puget_sound.gtfs_feeds.csv b/builds/planet/transit/gtfs-feeds/puget-sound.gtfs_feeds.csv similarity index 100% rename from builds/planet/transit/gtfs-feeds/puget_sound.gtfs_feeds.csv rename to builds/planet/transit/gtfs-feeds/puget-sound.gtfs_feeds.csv diff --git a/builds/planet/transit/gtfs-feeds/sf_bay_area.gtfs_feeds.csv b/builds/planet/transit/gtfs-feeds/sf-bay-area.gtfs_feeds.csv similarity index 95% rename from builds/planet/transit/gtfs-feeds/sf_bay_area.gtfs_feeds.csv rename to builds/planet/transit/gtfs-feeds/sf-bay-area.gtfs_feeds.csv index 7e525e29f..c26e333c1 100644 --- a/builds/planet/transit/gtfs-feeds/sf_bay_area.gtfs_feeds.csv +++ b/builds/planet/transit/gtfs-feeds/sf-bay-area.gtfs_feeds.csv @@ -18,7 +18,6 @@ mdb_source_id,data_type,entity_type,location.country_code,location.subdivision_n 1288,gtfs,,US,California,Napa,VINE Transit,,,,https://data.trilliumtransit.com/gtfs/vinetransit-ca-us/vinetransit-ca-us.zip,0,,,https://storage.googleapis.com/storage/v1/b/mdb-latest/o/us-california-vine-transit-gtfs-1288.zip?alt=media,https://vinetransit.com/open-data-portal/,37.925323,38.583539,-122.579754889011,-122.041518688202,2022-05-23T16:46:03+00:00,, 2224,gtfs,,US,California,Emeryville,Emery Go-Round,,,,https://emerygoround.com/data/emerygoround-ca-us.zip,0,,,https://storage.googleapis.com/storage/v1/b/mdb-latest/o/us-california-emery-go-round-gtfs-2224.zip?alt=media,,37.829424,37.852317,-122.313403,-122.266983,2024-11-26T19:53:24+00:00,, 2421,gtfs,,US,California,,County Connection (CCCTA),,,,https://countyconnection.com/GTFS/google_transit.zip,,,,https://storage.googleapis.com/storage/v1/b/mdb-latest/o/us-california-county-connection-gtfs-2421.zip?alt=media,,,,,,,, -2455,gtfs,,US,California,Oakland,Alameda-Contra Costa Transit District (AC Transit),,,,https://api.actransit.org/transit/gtfs/download,1,https://api.actransit.org/transit,token,https://storage.googleapis.com/storage/v1/b/mdb-latest/o/us-california-alameda-contra-costa-transit-district-ac-transit-gtfs-2455.zip?alt=media,https://www.actransit.org/data-terms-and-conditions/,37.409768,38.017813,-122.421947,-121.886829,2022-03-23T23:19:13+00:00,, 2880,gtfs,,US,California,Mountain View,Mountain View Transportation Management Association (MVgo),,,,https://gtfs.mvgo.org/gtfs.zip,0,,,https://storage.googleapis.com/storage/v1/b/mdb-latest/o/us-california-mountain-view-transportation-management-association-mvgo-gtfs-2880.zip?alt=media,https://mvgo.org/developers/,37.387655512462,37.4314288965063,-122.111623478335,-122.047760594783,2024-05-14T17:15:22+00:00,, 2886,gtfs,,US,California,San Francisco,San Francisco Municipal Transportation Agency (SFMTA - Muni),,,,https://muni-gtfs.apps.sfmta.com/data/muni_gtfs-current.zip,0,,,https://storage.googleapis.com/storage/v1/b/mdb-latest/o/us-california-san-francisco-municipal-transportation-agency-sfmta-gtfs-2886.zip?alt=media,https://www.sfmta.com/reports/gtfs-transit-data,,,,,,, 2937,gtfs,,US,California,San Francisco,Marin Transit,,,,https://marintransit.gov/data/google_transit.zip,0,,,https://storage.googleapis.com/storage/v1/b/mdb-latest/o/us-california-marin-transit-gtfs-2937.zip?alt=media,,,,,,,,fares-v1 diff --git a/builds/planet/transit/PugetSound-router-config.json b/builds/planet/transit/puget-sound-router-config.json similarity index 100% rename from builds/planet/transit/PugetSound-router-config.json rename to builds/planet/transit/puget-sound-router-config.json diff --git a/builds/planet/transit/SfBayArea-router-config.json b/builds/planet/transit/sf-bay-area-router-config.json similarity index 100% rename from builds/planet/transit/SfBayArea-router-config.json rename to builds/planet/transit/sf-bay-area-router-config.json diff --git a/dagger/transit.go b/dagger/transit.go index dc606cd27..3c0e49f87 100644 --- a/dagger/transit.go +++ b/dagger/transit.go @@ -27,11 +27,6 @@ type TransitZone struct { // scheme, which is what keeps the two honest with each other: the feeds are // fetched at most once per day, and the date in the name is by construction // the day they were fetched. - // - // Published builds accumulate side by side as e.g. - // planet-puget_sound-2026-07-28.graph.obj.zst, and bin/link-latest-transit - // parses the date back out to symlink the newest one to - // PugetSound.graph.obj.zst. BuildDate string TransitFeeds *dagger.File GTFSDir *dagger.Directory diff --git a/docker-compose-with-transit.yaml b/docker-compose-with-transit.yaml index dd0cbca54..00be4ca53 100644 --- a/docker-compose-with-transit.yaml +++ b/docker-compose-with-transit.yaml @@ -31,7 +31,7 @@ services: image: ghcr.io/headwaymaps/opentripplanner-init:latest env_file: builds/${HEADWAY_AREA:?}/.env environment: - OTP_ARTIFACT_SOURCE_PATH: /bootstrap/${HEADWAY_AREA}.graph.obj.zst + OTP_ARTIFACT_SOURCE_PATH: /bootstrap/${HEADWAY_OTP_GRAPH_FILE} volumes: - "./data/${HEADWAY_AREA}/:/bootstrap/:ro" - "opentripplanner_data:/data/:rw" @@ -56,7 +56,7 @@ services: image: ghcr.io/headwaymaps/travelmux-init:latest env_file: builds/${HEADWAY_AREA:?}/.env environment: - TRAVELMUX_ELEVATION_ARTIFACT_SOURCE_PATH: /bootstrap/${HEADWAY_AREA}.elevation-tifs.tar.zst + TRAVELMUX_ELEVATION_ARTIFACT_SOURCE_PATH: /bootstrap/${HEADWAY_ELEVATION_FILE} volumes: - "./data/${HEADWAY_AREA}/:/bootstrap/:ro" - "travelmux_data:/data/:rw" diff --git a/docker-compose.yaml b/docker-compose.yaml index 7a3a103e4..8f9871684 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -31,7 +31,7 @@ services: image: ghcr.io/headwaymaps/travelmux-init:latest env_file: builds/${HEADWAY_AREA:?}/.env environment: - TRAVELMUX_ELEVATION_ARTIFACT_SOURCE_PATH: /bootstrap/transit/${HEADWAY_AREA}.elevation-tifs.tar.zst + TRAVELMUX_ELEVATION_ARTIFACT_SOURCE_PATH: /bootstrap/${HEADWAY_ELEVATION_FILE} volumes: - "./data/${HEADWAY_AREA}/:/bootstrap/:ro" - "travelmux_data:/data/:rw" diff --git a/k8s/_template/deployment-config.yaml.tpl b/k8s/_template/deployment-config.yaml.tpl index 950515535..57422747b 100644 --- a/k8s/_template/deployment-config.yaml.tpl +++ b/k8s/_template/deployment-config.yaml.tpl @@ -15,6 +15,6 @@ data: areamap-source-url: ${HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT}/${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}/${HEADWAY_AREA}.pmtiles valhalla-artifact-url: ${HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT}/${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}/${HEADWAY_AREA}.valhalla.tar.zst placeholder-artifact-url: ${HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT}/${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}/${HEADWAY_AREA}.placeholder.tar.zst - elasticsearch-artifact-url: ${HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT}/${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}/${HEADWAY_AREA}.elasticsearch.tar.zst${OTP_GRAPHS_YAML} - elevation-artifact-url: ${HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT}/${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}/${HEADWAY_AREA}.elevation-tifs.tar.zst + elasticsearch-artifact-url: ${HEADWAY_K8S_TEMPLATE_ARTIFACT_ROOT}/${HEADWAY_DATA_TAG}/${HEADWAY_AREA_TAG}/${HEADWAY_AREA}.elasticsearch.tar.zst + elevation-artifact-url: "${ELEVATION_ARTIFACT_URL}" pelias-config-json: ${PELIAS_CONFIG_JSON_YAML} diff --git a/k8s/_template/opentripplanner-AREA-config.yaml.tpl b/k8s/_template/opentripplanner-ZONE-config.yaml.tpl similarity index 62% rename from k8s/_template/opentripplanner-AREA-config.yaml.tpl rename to k8s/_template/opentripplanner-ZONE-config.yaml.tpl index 603024311..b8d85cf9a 100644 --- a/k8s/_template/opentripplanner-AREA-config.yaml.tpl +++ b/k8s/_template/opentripplanner-ZONE-config.yaml.tpl @@ -1,9 +1,8 @@ apiVersion: v1 kind: ConfigMap metadata: - name: otp-${TRANSIT_AREA}-config + name: otp-${TRANSIT_ZONE}-config data: graph-url: ${OTP_GRAPH_URL} - otp-config-json: ${OTP_CONFIG_JSON_YAML} router-config-json: ${OTP_ROUTER_CONFIG_JSON_YAML} diff --git a/k8s/_template/opentripplanner-AREA-deployment.yaml.tpl b/k8s/_template/opentripplanner-ZONE-deployment.yaml.tpl similarity index 88% rename from k8s/_template/opentripplanner-AREA-deployment.yaml.tpl rename to k8s/_template/opentripplanner-ZONE-deployment.yaml.tpl index 43b3104a7..8e4997f1b 100644 --- a/k8s/_template/opentripplanner-AREA-deployment.yaml.tpl +++ b/k8s/_template/opentripplanner-ZONE-deployment.yaml.tpl @@ -23,17 +23,12 @@ spec: - name: OTP_ARTIFACT_URL valueFrom: configMapKeyRef: - name: otp-${TRANSIT_AREA}-config + name: otp-${TRANSIT_ZONE}-config key: graph-url - - name: OTP_CONFIG_JSON - valueFrom: - configMapKeyRef: - name: otp-${TRANSIT_AREA}-config - key: otp-config-json - name: OTP_ROUTER_CONFIG_JSON valueFrom: configMapKeyRef: - name: otp-${TRANSIT_AREA}-config + name: otp-${TRANSIT_ZONE}-config key: router-config-json resources: limits: diff --git a/k8s/_template/opentripplanner-AREA-service.yaml.tpl b/k8s/_template/opentripplanner-ZONE-service.yaml.tpl similarity index 100% rename from k8s/_template/opentripplanner-AREA-service.yaml.tpl rename to k8s/_template/opentripplanner-ZONE-service.yaml.tpl diff --git a/k8s/configs/planet-dev/deployment-config.yaml b/k8s/configs/planet-dev/deployment-config.yaml index 6993b3d8b..4ad49a971 100644 --- a/k8s/configs/planet-dev/deployment-config.yaml +++ b/k8s/configs/planet-dev/deployment-config.yaml @@ -16,7 +16,7 @@ data: valhalla-artifact-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/maps-earth-planet-v1.260727.valhalla.tar.zst placeholder-artifact-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/maps-earth-planet-v1.260727.placeholder.tar.zst elasticsearch-artifact-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/maps-earth-planet-v1.260727.elasticsearch.tar.zst - elevation-artifact-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/maps-earth-planet-v1.260727.elevation-tifs.tar.zst + elevation-artifact-url: "https://data.example.com/0.12.0/maps-earth-planet-v1.260727/transit/maps-earth-planet-v1.260727-2026-08-14.elevation-tifs.tar.zst" pelias-config-json: | { "logger": { diff --git a/k8s/configs/planet-dev/opentripplanner-barcelona-config.yaml b/k8s/configs/planet-dev/opentripplanner-barcelona-config.yaml index aeac9df68..40345aada 100644 --- a/k8s/configs/planet-dev/opentripplanner-barcelona-config.yaml +++ b/k8s/configs/planet-dev/opentripplanner-barcelona-config.yaml @@ -3,8 +3,6 @@ kind: ConfigMap metadata: name: otp-barcelona-config data: - graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/Barcelona.graph.obj.zst - otp-config-json: | - {} + graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/transit/maps-earth-planet-v1.260727-barcelona-2026-08-14.graph.obj.zst router-config-json: "" diff --git a/k8s/configs/planet-dev/opentripplanner-barcelona-deployment.yaml b/k8s/configs/planet-dev/opentripplanner-barcelona-deployment.yaml index ccdd3ed97..311d9c2f4 100644 --- a/k8s/configs/planet-dev/opentripplanner-barcelona-deployment.yaml +++ b/k8s/configs/planet-dev/opentripplanner-barcelona-deployment.yaml @@ -25,11 +25,6 @@ spec: configMapKeyRef: name: otp-barcelona-config key: graph-url - - name: OTP_CONFIG_JSON - valueFrom: - configMapKeyRef: - name: otp-barcelona-config - key: otp-config-json - name: OTP_ROUTER_CONFIG_JSON valueFrom: configMapKeyRef: diff --git a/k8s/configs/planet-dev/opentripplanner-losangeles-config.yaml b/k8s/configs/planet-dev/opentripplanner-los-angeles-config.yaml similarity index 54% rename from k8s/configs/planet-dev/opentripplanner-losangeles-config.yaml rename to k8s/configs/planet-dev/opentripplanner-los-angeles-config.yaml index ba4f6b5bc..0f6fd404e 100644 --- a/k8s/configs/planet-dev/opentripplanner-losangeles-config.yaml +++ b/k8s/configs/planet-dev/opentripplanner-los-angeles-config.yaml @@ -1,10 +1,8 @@ apiVersion: v1 kind: ConfigMap metadata: - name: otp-losangeles-config + name: otp-los-angeles-config data: - graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/LosAngeles.graph.obj.zst - otp-config-json: | - {} + graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/transit/maps-earth-planet-v1.260727-los-angeles-2026-08-14.graph.obj.zst router-config-json: "" diff --git a/k8s/configs/planet-dev/opentripplanner-sfbayarea-deployment.yaml b/k8s/configs/planet-dev/opentripplanner-los-angeles-deployment.yaml similarity index 84% rename from k8s/configs/planet-dev/opentripplanner-sfbayarea-deployment.yaml rename to k8s/configs/planet-dev/opentripplanner-los-angeles-deployment.yaml index 7a7549a09..d1c7dedc3 100644 --- a/k8s/configs/planet-dev/opentripplanner-sfbayarea-deployment.yaml +++ b/k8s/configs/planet-dev/opentripplanner-los-angeles-deployment.yaml @@ -1,16 +1,16 @@ apiVersion: apps/v1 kind: Deployment metadata: - name: opentripplanner-sfbayarea + name: opentripplanner-los-angeles spec: selector: matchLabels: - app: opentripplanner-sfbayarea + app: opentripplanner-los-angeles replicas: 1 template: metadata: labels: - app: opentripplanner-sfbayarea + app: opentripplanner-los-angeles spec: initContainers: - name: init @@ -23,17 +23,12 @@ spec: - name: OTP_ARTIFACT_URL valueFrom: configMapKeyRef: - name: otp-sfbayarea-config + name: otp-los-angeles-config key: graph-url - - name: OTP_CONFIG_JSON - valueFrom: - configMapKeyRef: - name: otp-sfbayarea-config - key: otp-config-json - name: OTP_ROUTER_CONFIG_JSON valueFrom: configMapKeyRef: - name: otp-sfbayarea-config + name: otp-los-angeles-config key: router-config-json resources: limits: diff --git a/k8s/configs/planet-dev/opentripplanner-sfbayarea-service.yaml b/k8s/configs/planet-dev/opentripplanner-los-angeles-service.yaml similarity index 62% rename from k8s/configs/planet-dev/opentripplanner-sfbayarea-service.yaml rename to k8s/configs/planet-dev/opentripplanner-los-angeles-service.yaml index 9ee8894ef..4c856b0ee 100644 --- a/k8s/configs/planet-dev/opentripplanner-sfbayarea-service.yaml +++ b/k8s/configs/planet-dev/opentripplanner-los-angeles-service.yaml @@ -1,10 +1,10 @@ apiVersion: v1 kind: Service metadata: - name: opentripplanner-sfbayarea + name: opentripplanner-los-angeles spec: selector: - app: opentripplanner-sfbayarea + app: opentripplanner-los-angeles ports: - protocol: TCP name: service diff --git a/k8s/configs/planet-dev/opentripplanner-pugetsound-config.yaml b/k8s/configs/planet-dev/opentripplanner-puget-sound-config.yaml similarity index 83% rename from k8s/configs/planet-dev/opentripplanner-pugetsound-config.yaml rename to k8s/configs/planet-dev/opentripplanner-puget-sound-config.yaml index 733966498..560d7d379 100644 --- a/k8s/configs/planet-dev/opentripplanner-pugetsound-config.yaml +++ b/k8s/configs/planet-dev/opentripplanner-puget-sound-config.yaml @@ -1,11 +1,9 @@ apiVersion: v1 kind: ConfigMap metadata: - name: otp-pugetsound-config + name: otp-puget-sound-config data: - graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/PugetSound.graph.obj.zst - otp-config-json: | - {} + graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/transit/maps-earth-planet-v1.260727-puget-sound-2026-08-14.graph.obj.zst router-config-json: | { "updaters": [ diff --git a/k8s/configs/planet-dev/opentripplanner-losangeles-deployment.yaml b/k8s/configs/planet-dev/opentripplanner-puget-sound-deployment.yaml similarity index 84% rename from k8s/configs/planet-dev/opentripplanner-losangeles-deployment.yaml rename to k8s/configs/planet-dev/opentripplanner-puget-sound-deployment.yaml index 419c8e171..2011d2cb8 100644 --- a/k8s/configs/planet-dev/opentripplanner-losangeles-deployment.yaml +++ b/k8s/configs/planet-dev/opentripplanner-puget-sound-deployment.yaml @@ -1,16 +1,16 @@ apiVersion: apps/v1 kind: Deployment metadata: - name: opentripplanner-losangeles + name: opentripplanner-puget-sound spec: selector: matchLabels: - app: opentripplanner-losangeles + app: opentripplanner-puget-sound replicas: 1 template: metadata: labels: - app: opentripplanner-losangeles + app: opentripplanner-puget-sound spec: initContainers: - name: init @@ -23,17 +23,12 @@ spec: - name: OTP_ARTIFACT_URL valueFrom: configMapKeyRef: - name: otp-losangeles-config + name: otp-puget-sound-config key: graph-url - - name: OTP_CONFIG_JSON - valueFrom: - configMapKeyRef: - name: otp-losangeles-config - key: otp-config-json - name: OTP_ROUTER_CONFIG_JSON valueFrom: configMapKeyRef: - name: otp-losangeles-config + name: otp-puget-sound-config key: router-config-json resources: limits: diff --git a/k8s/configs/planet-dev/opentripplanner-losangeles-service.yaml b/k8s/configs/planet-dev/opentripplanner-puget-sound-service.yaml similarity index 62% rename from k8s/configs/planet-dev/opentripplanner-losangeles-service.yaml rename to k8s/configs/planet-dev/opentripplanner-puget-sound-service.yaml index e9a9a3816..399f77748 100644 --- a/k8s/configs/planet-dev/opentripplanner-losangeles-service.yaml +++ b/k8s/configs/planet-dev/opentripplanner-puget-sound-service.yaml @@ -1,10 +1,10 @@ apiVersion: v1 kind: Service metadata: - name: opentripplanner-losangeles + name: opentripplanner-puget-sound spec: selector: - app: opentripplanner-losangeles + app: opentripplanner-puget-sound ports: - protocol: TCP name: service diff --git a/k8s/configs/planet-dev/opentripplanner-sfbayarea-config.yaml b/k8s/configs/planet-dev/opentripplanner-sf-bay-area-config.yaml similarity index 82% rename from k8s/configs/planet-dev/opentripplanner-sfbayarea-config.yaml rename to k8s/configs/planet-dev/opentripplanner-sf-bay-area-config.yaml index 9b4d04c67..c27c570a8 100644 --- a/k8s/configs/planet-dev/opentripplanner-sfbayarea-config.yaml +++ b/k8s/configs/planet-dev/opentripplanner-sf-bay-area-config.yaml @@ -1,11 +1,9 @@ apiVersion: v1 kind: ConfigMap metadata: - name: otp-sfbayarea-config + name: otp-sf-bay-area-config data: - graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/SfBayArea.graph.obj.zst - otp-config-json: | - {} + graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/transit/maps-earth-planet-v1.260727-sf-bay-area-2026-08-14.graph.obj.zst router-config-json: | { "updaters": [ diff --git a/k8s/configs/planet-dev/opentripplanner-pugetsound-deployment.yaml b/k8s/configs/planet-dev/opentripplanner-sf-bay-area-deployment.yaml similarity index 84% rename from k8s/configs/planet-dev/opentripplanner-pugetsound-deployment.yaml rename to k8s/configs/planet-dev/opentripplanner-sf-bay-area-deployment.yaml index 92bc51827..48115f261 100644 --- a/k8s/configs/planet-dev/opentripplanner-pugetsound-deployment.yaml +++ b/k8s/configs/planet-dev/opentripplanner-sf-bay-area-deployment.yaml @@ -1,16 +1,16 @@ apiVersion: apps/v1 kind: Deployment metadata: - name: opentripplanner-pugetsound + name: opentripplanner-sf-bay-area spec: selector: matchLabels: - app: opentripplanner-pugetsound + app: opentripplanner-sf-bay-area replicas: 1 template: metadata: labels: - app: opentripplanner-pugetsound + app: opentripplanner-sf-bay-area spec: initContainers: - name: init @@ -23,17 +23,12 @@ spec: - name: OTP_ARTIFACT_URL valueFrom: configMapKeyRef: - name: otp-pugetsound-config + name: otp-sf-bay-area-config key: graph-url - - name: OTP_CONFIG_JSON - valueFrom: - configMapKeyRef: - name: otp-pugetsound-config - key: otp-config-json - name: OTP_ROUTER_CONFIG_JSON valueFrom: configMapKeyRef: - name: otp-pugetsound-config + name: otp-sf-bay-area-config key: router-config-json resources: limits: diff --git a/k8s/configs/planet-dev/opentripplanner-pugetsound-service.yaml b/k8s/configs/planet-dev/opentripplanner-sf-bay-area-service.yaml similarity index 62% rename from k8s/configs/planet-dev/opentripplanner-pugetsound-service.yaml rename to k8s/configs/planet-dev/opentripplanner-sf-bay-area-service.yaml index 18fcb62a0..0ea3321e8 100644 --- a/k8s/configs/planet-dev/opentripplanner-pugetsound-service.yaml +++ b/k8s/configs/planet-dev/opentripplanner-sf-bay-area-service.yaml @@ -1,10 +1,10 @@ apiVersion: v1 kind: Service metadata: - name: opentripplanner-pugetsound + name: opentripplanner-sf-bay-area spec: selector: - app: opentripplanner-pugetsound + app: opentripplanner-sf-bay-area ports: - protocol: TCP name: service diff --git a/k8s/configs/planet-dev/travelmux-deployment.yaml b/k8s/configs/planet-dev/travelmux-deployment.yaml index c426ea61f..7c645d152 100644 --- a/k8s/configs/planet-dev/travelmux-deployment.yaml +++ b/k8s/configs/planet-dev/travelmux-deployment.yaml @@ -34,7 +34,7 @@ spec: - name: main image: ghcr.io/headwaymaps/travelmux:maps-earth-dev imagePullPolicy: Always - args: ["http://valhalla:8002", "http://opentripplanner-barcelona:8000", "http://opentripplanner-losangeles:8000", "http://opentripplanner-pugetsound:8000", "http://opentripplanner-sfbayarea:8000"] + args: ["http://valhalla:8002", "http://opentripplanner-barcelona:8000", "http://opentripplanner-los-angeles:8000", "http://opentripplanner-puget-sound:8000", "http://opentripplanner-sf-bay-area:8000"] env: - name: ELEVATION_TIFS_DIR value: /data/elevation-tifs diff --git a/k8s/configs/planet/deployment-config.yaml b/k8s/configs/planet/deployment-config.yaml index 6993b3d8b..4ad49a971 100644 --- a/k8s/configs/planet/deployment-config.yaml +++ b/k8s/configs/planet/deployment-config.yaml @@ -16,7 +16,7 @@ data: valhalla-artifact-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/maps-earth-planet-v1.260727.valhalla.tar.zst placeholder-artifact-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/maps-earth-planet-v1.260727.placeholder.tar.zst elasticsearch-artifact-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/maps-earth-planet-v1.260727.elasticsearch.tar.zst - elevation-artifact-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/maps-earth-planet-v1.260727.elevation-tifs.tar.zst + elevation-artifact-url: "https://data.example.com/0.12.0/maps-earth-planet-v1.260727/transit/maps-earth-planet-v1.260727-2026-08-14.elevation-tifs.tar.zst" pelias-config-json: | { "logger": { diff --git a/k8s/configs/planet/opentripplanner-barcelona-config.yaml b/k8s/configs/planet/opentripplanner-barcelona-config.yaml index aeac9df68..40345aada 100644 --- a/k8s/configs/planet/opentripplanner-barcelona-config.yaml +++ b/k8s/configs/planet/opentripplanner-barcelona-config.yaml @@ -3,8 +3,6 @@ kind: ConfigMap metadata: name: otp-barcelona-config data: - graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/Barcelona.graph.obj.zst - otp-config-json: | - {} + graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/transit/maps-earth-planet-v1.260727-barcelona-2026-08-14.graph.obj.zst router-config-json: "" diff --git a/k8s/configs/planet/opentripplanner-barcelona-deployment.yaml b/k8s/configs/planet/opentripplanner-barcelona-deployment.yaml index 074a20aee..c73728896 100644 --- a/k8s/configs/planet/opentripplanner-barcelona-deployment.yaml +++ b/k8s/configs/planet/opentripplanner-barcelona-deployment.yaml @@ -25,11 +25,6 @@ spec: configMapKeyRef: name: otp-barcelona-config key: graph-url - - name: OTP_CONFIG_JSON - valueFrom: - configMapKeyRef: - name: otp-barcelona-config - key: otp-config-json - name: OTP_ROUTER_CONFIG_JSON valueFrom: configMapKeyRef: diff --git a/k8s/configs/planet/opentripplanner-losangeles-config.yaml b/k8s/configs/planet/opentripplanner-los-angeles-config.yaml similarity index 54% rename from k8s/configs/planet/opentripplanner-losangeles-config.yaml rename to k8s/configs/planet/opentripplanner-los-angeles-config.yaml index ba4f6b5bc..0f6fd404e 100644 --- a/k8s/configs/planet/opentripplanner-losangeles-config.yaml +++ b/k8s/configs/planet/opentripplanner-los-angeles-config.yaml @@ -1,10 +1,8 @@ apiVersion: v1 kind: ConfigMap metadata: - name: otp-losangeles-config + name: otp-los-angeles-config data: - graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/LosAngeles.graph.obj.zst - otp-config-json: | - {} + graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/transit/maps-earth-planet-v1.260727-los-angeles-2026-08-14.graph.obj.zst router-config-json: "" diff --git a/k8s/configs/planet/opentripplanner-sfbayarea-deployment.yaml b/k8s/configs/planet/opentripplanner-los-angeles-deployment.yaml similarity index 84% rename from k8s/configs/planet/opentripplanner-sfbayarea-deployment.yaml rename to k8s/configs/planet/opentripplanner-los-angeles-deployment.yaml index ac50fd035..13fac1431 100644 --- a/k8s/configs/planet/opentripplanner-sfbayarea-deployment.yaml +++ b/k8s/configs/planet/opentripplanner-los-angeles-deployment.yaml @@ -1,16 +1,16 @@ apiVersion: apps/v1 kind: Deployment metadata: - name: opentripplanner-sfbayarea + name: opentripplanner-los-angeles spec: selector: matchLabels: - app: opentripplanner-sfbayarea + app: opentripplanner-los-angeles replicas: 1 template: metadata: labels: - app: opentripplanner-sfbayarea + app: opentripplanner-los-angeles spec: initContainers: - name: init @@ -23,17 +23,12 @@ spec: - name: OTP_ARTIFACT_URL valueFrom: configMapKeyRef: - name: otp-sfbayarea-config + name: otp-los-angeles-config key: graph-url - - name: OTP_CONFIG_JSON - valueFrom: - configMapKeyRef: - name: otp-sfbayarea-config - key: otp-config-json - name: OTP_ROUTER_CONFIG_JSON valueFrom: configMapKeyRef: - name: otp-sfbayarea-config + name: otp-los-angeles-config key: router-config-json resources: limits: diff --git a/k8s/configs/planet/opentripplanner-losangeles-service.yaml b/k8s/configs/planet/opentripplanner-los-angeles-service.yaml similarity index 62% rename from k8s/configs/planet/opentripplanner-losangeles-service.yaml rename to k8s/configs/planet/opentripplanner-los-angeles-service.yaml index e9a9a3816..4c856b0ee 100644 --- a/k8s/configs/planet/opentripplanner-losangeles-service.yaml +++ b/k8s/configs/planet/opentripplanner-los-angeles-service.yaml @@ -1,10 +1,10 @@ apiVersion: v1 kind: Service metadata: - name: opentripplanner-losangeles + name: opentripplanner-los-angeles spec: selector: - app: opentripplanner-losangeles + app: opentripplanner-los-angeles ports: - protocol: TCP name: service diff --git a/k8s/configs/planet/opentripplanner-pugetsound-config.yaml b/k8s/configs/planet/opentripplanner-puget-sound-config.yaml similarity index 83% rename from k8s/configs/planet/opentripplanner-pugetsound-config.yaml rename to k8s/configs/planet/opentripplanner-puget-sound-config.yaml index 733966498..560d7d379 100644 --- a/k8s/configs/planet/opentripplanner-pugetsound-config.yaml +++ b/k8s/configs/planet/opentripplanner-puget-sound-config.yaml @@ -1,11 +1,9 @@ apiVersion: v1 kind: ConfigMap metadata: - name: otp-pugetsound-config + name: otp-puget-sound-config data: - graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/PugetSound.graph.obj.zst - otp-config-json: | - {} + graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/transit/maps-earth-planet-v1.260727-puget-sound-2026-08-14.graph.obj.zst router-config-json: | { "updaters": [ diff --git a/k8s/configs/planet/opentripplanner-losangeles-deployment.yaml b/k8s/configs/planet/opentripplanner-puget-sound-deployment.yaml similarity index 84% rename from k8s/configs/planet/opentripplanner-losangeles-deployment.yaml rename to k8s/configs/planet/opentripplanner-puget-sound-deployment.yaml index a244af0d1..91b29f480 100644 --- a/k8s/configs/planet/opentripplanner-losangeles-deployment.yaml +++ b/k8s/configs/planet/opentripplanner-puget-sound-deployment.yaml @@ -1,16 +1,16 @@ apiVersion: apps/v1 kind: Deployment metadata: - name: opentripplanner-losangeles + name: opentripplanner-puget-sound spec: selector: matchLabels: - app: opentripplanner-losangeles + app: opentripplanner-puget-sound replicas: 1 template: metadata: labels: - app: opentripplanner-losangeles + app: opentripplanner-puget-sound spec: initContainers: - name: init @@ -23,17 +23,12 @@ spec: - name: OTP_ARTIFACT_URL valueFrom: configMapKeyRef: - name: otp-losangeles-config + name: otp-puget-sound-config key: graph-url - - name: OTP_CONFIG_JSON - valueFrom: - configMapKeyRef: - name: otp-losangeles-config - key: otp-config-json - name: OTP_ROUTER_CONFIG_JSON valueFrom: configMapKeyRef: - name: otp-losangeles-config + name: otp-puget-sound-config key: router-config-json resources: limits: diff --git a/k8s/configs/planet/opentripplanner-puget-sound-service.yaml b/k8s/configs/planet/opentripplanner-puget-sound-service.yaml new file mode 100644 index 000000000..399f77748 --- /dev/null +++ b/k8s/configs/planet/opentripplanner-puget-sound-service.yaml @@ -0,0 +1,11 @@ +apiVersion: v1 +kind: Service +metadata: + name: opentripplanner-puget-sound +spec: + selector: + app: opentripplanner-puget-sound + ports: + - protocol: TCP + name: service + port: 8000 diff --git a/k8s/configs/planet/opentripplanner-pugetsound-service.yaml b/k8s/configs/planet/opentripplanner-pugetsound-service.yaml deleted file mode 100644 index 18fcb62a0..000000000 --- a/k8s/configs/planet/opentripplanner-pugetsound-service.yaml +++ /dev/null @@ -1,11 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - name: opentripplanner-pugetsound -spec: - selector: - app: opentripplanner-pugetsound - ports: - - protocol: TCP - name: service - port: 8000 diff --git a/k8s/configs/planet/opentripplanner-sfbayarea-config.yaml b/k8s/configs/planet/opentripplanner-sf-bay-area-config.yaml similarity index 82% rename from k8s/configs/planet/opentripplanner-sfbayarea-config.yaml rename to k8s/configs/planet/opentripplanner-sf-bay-area-config.yaml index 9b4d04c67..c27c570a8 100644 --- a/k8s/configs/planet/opentripplanner-sfbayarea-config.yaml +++ b/k8s/configs/planet/opentripplanner-sf-bay-area-config.yaml @@ -1,11 +1,9 @@ apiVersion: v1 kind: ConfigMap metadata: - name: otp-sfbayarea-config + name: otp-sf-bay-area-config data: - graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/SfBayArea.graph.obj.zst - otp-config-json: | - {} + graph-url: https://data.example.com/0.12.0/maps-earth-planet-v1.260727/transit/maps-earth-planet-v1.260727-sf-bay-area-2026-08-14.graph.obj.zst router-config-json: | { "updaters": [ diff --git a/k8s/configs/planet/opentripplanner-pugetsound-deployment.yaml b/k8s/configs/planet/opentripplanner-sf-bay-area-deployment.yaml similarity index 84% rename from k8s/configs/planet/opentripplanner-pugetsound-deployment.yaml rename to k8s/configs/planet/opentripplanner-sf-bay-area-deployment.yaml index 95c3ae20d..cd3ee3b6e 100644 --- a/k8s/configs/planet/opentripplanner-pugetsound-deployment.yaml +++ b/k8s/configs/planet/opentripplanner-sf-bay-area-deployment.yaml @@ -1,16 +1,16 @@ apiVersion: apps/v1 kind: Deployment metadata: - name: opentripplanner-pugetsound + name: opentripplanner-sf-bay-area spec: selector: matchLabels: - app: opentripplanner-pugetsound + app: opentripplanner-sf-bay-area replicas: 1 template: metadata: labels: - app: opentripplanner-pugetsound + app: opentripplanner-sf-bay-area spec: initContainers: - name: init @@ -23,17 +23,12 @@ spec: - name: OTP_ARTIFACT_URL valueFrom: configMapKeyRef: - name: otp-pugetsound-config + name: otp-sf-bay-area-config key: graph-url - - name: OTP_CONFIG_JSON - valueFrom: - configMapKeyRef: - name: otp-pugetsound-config - key: otp-config-json - name: OTP_ROUTER_CONFIG_JSON valueFrom: configMapKeyRef: - name: otp-pugetsound-config + name: otp-sf-bay-area-config key: router-config-json resources: limits: diff --git a/k8s/configs/planet/opentripplanner-sf-bay-area-service.yaml b/k8s/configs/planet/opentripplanner-sf-bay-area-service.yaml new file mode 100644 index 000000000..0ea3321e8 --- /dev/null +++ b/k8s/configs/planet/opentripplanner-sf-bay-area-service.yaml @@ -0,0 +1,11 @@ +apiVersion: v1 +kind: Service +metadata: + name: opentripplanner-sf-bay-area +spec: + selector: + app: opentripplanner-sf-bay-area + ports: + - protocol: TCP + name: service + port: 8000 diff --git a/k8s/configs/planet/opentripplanner-sfbayarea-service.yaml b/k8s/configs/planet/opentripplanner-sfbayarea-service.yaml deleted file mode 100644 index 9ee8894ef..000000000 --- a/k8s/configs/planet/opentripplanner-sfbayarea-service.yaml +++ /dev/null @@ -1,11 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - name: opentripplanner-sfbayarea -spec: - selector: - app: opentripplanner-sfbayarea - ports: - - protocol: TCP - name: service - port: 8000 diff --git a/k8s/configs/planet/travelmux-deployment.yaml b/k8s/configs/planet/travelmux-deployment.yaml index c5c4a6e03..4c0b41b7e 100644 --- a/k8s/configs/planet/travelmux-deployment.yaml +++ b/k8s/configs/planet/travelmux-deployment.yaml @@ -34,7 +34,7 @@ spec: - name: main image: ghcr.io/headwaymaps/travelmux:maps-earth-latest imagePullPolicy: Always - args: ["http://valhalla:8002", "http://opentripplanner-barcelona:8000", "http://opentripplanner-losangeles:8000", "http://opentripplanner-pugetsound:8000", "http://opentripplanner-sfbayarea:8000"] + args: ["http://valhalla:8002", "http://opentripplanner-barcelona:8000", "http://opentripplanner-los-angeles:8000", "http://opentripplanner-puget-sound:8000", "http://opentripplanner-sf-bay-area:8000"] env: - name: ELEVATION_TIFS_DIR value: /data/elevation-tifs diff --git a/k8s/configs/seattle-dev/deployment-config.yaml b/k8s/configs/seattle-dev/deployment-config.yaml index 49c4f7274..d49703c57 100644 --- a/k8s/configs/seattle-dev/deployment-config.yaml +++ b/k8s/configs/seattle-dev/deployment-config.yaml @@ -16,7 +16,7 @@ data: valhalla-artifact-url: https://data.example.com/dev/seattle-latest/Seattle.valhalla.tar.zst placeholder-artifact-url: https://data.example.com/dev/seattle-latest/Seattle.placeholder.tar.zst elasticsearch-artifact-url: https://data.example.com/dev/seattle-latest/Seattle.elasticsearch.tar.zst - elevation-artifact-url: https://data.example.com/dev/seattle-latest/Seattle.elevation-tifs.tar.zst + elevation-artifact-url: "https://data.example.com/dev/seattle-latest/transit/Seattle-2026-08-14.elevation-tifs.tar.zst" pelias-config-json: | { "logger": { diff --git a/k8s/configs/seattle-dev/opentripplanner-seattle-config.yaml b/k8s/configs/seattle-dev/opentripplanner-seattle-config.yaml index 9fc4c15ad..8bacc5cdf 100644 --- a/k8s/configs/seattle-dev/opentripplanner-seattle-config.yaml +++ b/k8s/configs/seattle-dev/opentripplanner-seattle-config.yaml @@ -3,9 +3,7 @@ kind: ConfigMap metadata: name: otp-seattle-config data: - graph-url: https://data.example.com/dev/seattle-latest/Seattle.graph.obj.zst - otp-config-json: | - {} + graph-url: https://data.example.com/dev/seattle-latest/transit/Seattle-seattle-2026-08-14.graph.obj.zst router-config-json: | { "updaters": [ diff --git a/k8s/configs/seattle-dev/opentripplanner-seattle-deployment.yaml b/k8s/configs/seattle-dev/opentripplanner-seattle-deployment.yaml index 84a95ef00..6556e7db3 100644 --- a/k8s/configs/seattle-dev/opentripplanner-seattle-deployment.yaml +++ b/k8s/configs/seattle-dev/opentripplanner-seattle-deployment.yaml @@ -25,11 +25,6 @@ spec: configMapKeyRef: name: otp-seattle-config key: graph-url - - name: OTP_CONFIG_JSON - valueFrom: - configMapKeyRef: - name: otp-seattle-config - key: otp-config-json - name: OTP_ROUTER_CONFIG_JSON valueFrom: configMapKeyRef: diff --git a/services/otp/init.sh b/services/otp/init.sh index e42946c69..25bfbc796 100755 --- a/services/otp/init.sh +++ b/services/otp/init.sh @@ -20,7 +20,3 @@ fi if [ -n "$OTP_ROUTER_CONFIG_JSON" ]; then echo "$OTP_ROUTER_CONFIG_JSON" > /data/router-config.json fi - -if [ -n "$OTP_CONFIG_JSON" ]; then - echo "$OTP_CONFIG_JSON" > /data/otp-config.json -fi diff --git a/services/otp/otp-config.json b/services/otp/otp-config.json deleted file mode 100644 index 0967ef424..000000000 --- a/services/otp/otp-config.json +++ /dev/null @@ -1 +0,0 @@ -{} diff --git a/tests/browser/README.md b/tests/browser/README.md index 64b23973f..62f7b2e39 100644 --- a/tests/browser/README.md +++ b/tests/browser/README.md @@ -17,7 +17,6 @@ the fixtures are Bogota landmarks and the transit test assumes Start a stack first, exactly as for the integration tests: ```bash -TRANSIT_DATA_ROOT=./data bin/link-latest-transit builds/Bogota # after a fresh bin/build-transit bin/start-services --no-follow-logs builds/Bogota bin/wait-for-services ``` diff --git a/tests/integration/README.md b/tests/integration/README.md index 0858275ef..69d615de2 100644 --- a/tests/integration/README.md +++ b/tests/integration/README.md @@ -20,22 +20,19 @@ This is what CI uses. Services are automatically cleaned up when tests complete. When iterating on tests, keep services running between test runs: ```bash -# 1. Link the latest transit artifacts (only needed after a fresh bin/build-transit) -TRANSIT_DATA_ROOT=./data bin/link-latest-transit builds/Bogota - -# 2. Start services once +# 1. Start services once bin/start-services --no-follow-logs builds/Bogota -# 3. Wait for services to be ready +# 2. Wait for services to be ready bin/wait-for-services -# 4. Run tests (can repeat this step) +# 3. Run tests (can repeat this step) ./tests/integration/run-tests.sh # Edit test scripts and re-run as needed... ./tests/integration/run-tests.sh -# 5. Stop services when done +# 4. Stop services when done # BEWARE: This will destroy your headway docker volumes. bin/stop-and-remove-services builds/Bogota ``` diff --git a/tests/integration/run-integration-tests.sh b/tests/integration/run-integration-tests.sh index d67fa07d6..452cd320d 100755 --- a/tests/integration/run-integration-tests.sh +++ b/tests/integration/run-integration-tests.sh @@ -16,12 +16,6 @@ source bin/_source-env.sh "$CONFIG_DIR" FRONTEND_URL="${FRONTEND_URL:-http://localhost:8080}" -if [ "${HEADWAY_ENABLE_TRANSIT_ROUTING:-0}" != 0 ]; then - # The build writes dated transit artifacts - link the latest ones to the names - # docker compose serves them from. - TRANSIT_DATA_ROOT=./data "$APP_ROOT/bin/link-latest-transit" "$CONFIG_DIR" -fi - echo "========================================" echo "Headway Integration Test Runner" echo "========================================"