AWS location is append twice to some uploads

[sambles]

Inputs tarball re-uploaded under a doubled storage prefix (oasis/files/oasis/files/)

When a model run adds extra input files, generate_losses_output writes the inputs tarball back to storage under the wrong key. With S3 storage and AWS_LOCATION set, the storage location prefix ends up in the key twice, so the object lands at <AWS_LOCATION>/<AWS_LOCATION>/<name>. The original object it was meant to replace is left unchanged.

Seen in a tenant bucket as:
s3://<bucket>/oasis/files/oasis/files/... with OASIS_AWS_LOCATION=oasis/files

Affected versions

• OasisPlatform 2.5.8 (worker), with oasis-data-manager==0.2.3
• Needs STORAGE_TYPE=S3 and a non-empty AWS_LOCATION
• Only hit when a run adds extra input files to input-data, e.g. occurrence.bin

Root cause

oasis-data-manager 158b037 ("Fix OASIS_AWS_LOCATION var", #43) changed AwsS3Storage.put() in two ways:

• it now adds location in front of the key it writes;
• it returns the bucket-relative key including that prefix (oasis/files/<name>).

That behaviour is intended: the server's is_in_bucket() / CopyObject and file_storage_link() all work with raw bucket keys.

src/model_execution_worker/distributed_tasks.py (generate_losses_output) takes a reference to the stored tarball and passes it straight back into put():

if input_files_added:
    filestore.put(res['oasis_files_dir'], filename=res['input_location_storage'])

input_location_storage is one of two things:

1. A storage key that already includes the prefix. In generate-and-run it is the output_location returned by the input-generation put(), e.g. oasis/files/.tar.gz. put() adds location again → oasis/files/oasis/files/.tar.gz.
2. A presigned download URL. This is the case when the server sends download links rather than bucket keys (AWS_SHARED_BUCKET unset). The whole URL is used as the filename → key oasis/files/https://oasis/files/?X-Amz-....

Either way the existing inputs tarball isn't overwritten. It keeps its old contents, and a stray object is written somewhere else in the bucket.

Impact

• Stray objects pile up under oasis/files/oasis/files/ (or under keys containing https:).
• The inputs tarball linked to the analysis lacks the input files added during the run. Not checked end-to-end: the bucket objects couldn't be listed.

Fix

Branch fix/aws-location-issue:

• src/common/filestore/filestore.py: new strip_storage_location(filestore, reference), which removes a leading location/ from a key and leaves anything else unchanged.

• src/model_execution_worker/distributed_tasks.me(), used for the re-upload:

  • URL → the object's file name (last part of
  • key → key with the location prefix removed.

  put() then adds the prefix exactly once and o. Shared-filesystem storage is unaffected.

Tests

• src/common/tests/test_filestore.py: unit testagainst a real
  AwsS3Storage(location='oasis/files'), includitrip.
• src/model_execution_worker/tests/test_distribilename: key, presigned URL, shared-fs, and acheck that put() with the resolved name returns the original key.
  All worker tests pass (89) and flake8 is clean. t covered by this fixServer file conversion (files/v1_api/tasks.pyres the location-prefixed key fromget_storage_url() in converted_file, so DjangION a second time when reading it. This can'thappen today: run_file_conversion fails earlier, because get_filestore(settings) is given Django settings, which have no .get(). The converter package also isn't instif conversion is turned back on.
• Azure: AzureABFSStorage uses location as root_dir (keys relative to location), while S3 keys are relative to the bucket. file_storage_link() adds storage.location foray happen on Azure reads when AZURE_LOCATION isset. Not verified.
• Existing stray objects under oasis/files/oasip separately.

[codecov]

Codecov Report

❌ Patch coverage is 97.77778% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.22%. Comparing base (fedbdd7) to head (bbfab5d).

Files with missing lines	Patch %	Lines
src/common/tests/test_storage_location.py	96.55%	1 Missing ⚠️
src/model_execution_worker/distributed_tasks.py	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1453      +/-   ##
==========================================
+ Coverage   77.02%   77.22%   +0.19%     
==========================================
  Files         219      221       +2     
  Lines       14853    14941      +88     
==========================================
+ Hits        11441    11538      +97     
+ Misses       3412     3403       -9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.