Skip to content

fix: aws filter fallback and chunks sorting - #2342

Open
anesson-cs wants to merge 5 commits into
developfrom
aws-objects-filter
Open

fix: aws filter fallback and chunks sorting#2342
anesson-cs wants to merge 5 commits into
developfrom
aws-objects-filter

Conversation

@anesson-cs

@anesson-cs anesson-cs commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Description

While downloading COP-DEM products with creodias_s3, the exception NoMatchingCollection("No product found to download.") was raised. This PR fixes this by handling mypy_boto3_s3.service_resource.BucketObjectsCollection.filter() in any case (with a prefix ending with a slash or not) with a fallback mechanism. The first try is by passing in parameter to filter() the prefix without a slash. If filter() return an empty list, the slash is added to the parameter and the list will not be empty anymore.

Then, a process allow to keep only chunks which are:

  • exact object at the product prefix which does not have any child object, even if it has zero size attribute
  • exact object at the product prefix which has children objects, but it must have a non-zero size attribute
  • actual product files (child below the product folder), even if they have zero size attribute.

This allow to ignore S3 directory markers and sibling files.

Additional information

Furthermore, a lot of linter 'ruff' warnings are removed and some AWS classes from library mypy_boto3_s3 are used for typing.

@github-actions

github-actions Bot commented Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Benchmark comparison

Compared baseline: origin/develop
Compared candidate: df6f69b


------------------------------- benchmark: 9 tests, 2 sources -------------------------------
Name (time in us)                                 baseline Median  candidate Median   ΔMedian
---------------------------------------------------------------------------------------------
test_benchmark_eoproduct_assets_population                12.1590           12.5470     +3.2%
test_benchmark_eoproduct_instantiation                    27.5420           27.7410     +0.7%
test_benchmark_stac_reader_fetch_recursive             3,511.1960        3,502.1065     -0.3%
test_benchmark_cli_version                            14,275.3375       14,634.0380     +2.5%
test_benchmark_cli_without_args_subprocess           128,119.9900      128,880.3080     +0.6%
test_benchmark_core_search_with_provider             139,157.2910      141,697.2730     +1.8%
test_benchmark_core_search_with_count                187,017.2220      192,366.8535     +2.9%
test_benchmark_cli_list_collections                  251,760.5190      253,014.7580     +0.5%
test_benchmark_eodag_instantiation_subprocess        964,266.0605      966,165.5935     +0.2%
---------------------------------------------------------------------------------------------

Legend:
  Cyan: reference source for comparison. Green: improvement, Red: regression.
  Δ: percentage change from reference source.

@github-actions

github-actions Bot commented Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Test Results

    4 files  ±  0      4 suites  ±0   3m 59s ⏱️ -15s
1 003 tests + 32  1 003 ✅ + 32  0 💤 ±0  0 ❌ ±0 
4 094 runs  +132  4 092 ✅ +132  2 💤 ±0  0 ❌ ±0 

Results for commit 4a02a64. ± Comparison against base commit 086ad68.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

badge

Code Coverage (Ubuntu)

Details 
Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     19       2  89.47%   61, 70
cli.py                                         259      11  95.75%   99-110, 376, 658
config.py                                      409      39  90.46%   84, 203, 227-242, 264, 269, 303-305, 308, 311, 314, 318, 322, 326-328, 822-824, 875, 1010-1012, 1036-1040, 1059, 1067, 1095-1100, 1102
crunch.py                                        2       0  100.00%
api/__init__.py                                  0       0  100.00%
api/collection.py                              154       7  95.45%   193, 229, 339, 378, 381, 399, 402
api/core.py                                    754      26  96.55%   157, 272, 284-295, 540, 584, 621, 661, 679, 750, 847-851, 873, 1026, 1031, 1293, 1364, 1664, 1889, 2046, 2050, 2159
api/provider.py                                414      44  89.37%   141-145, 212, 216-217, 354-375, 443, 559, 563-564, 569-572, 583, 644-645, 701-711, 896-897, 955, 962, 976, 999-1002, 1036-1043, 1047-1048
api/search_result.py                           138      12  91.30%   111, 123, 133, 154, 208, 266, 278, 406, 461-464
api/product/__init__.py                         26       4  84.62%   51-54
api/product/_assets.py                          78       5  93.59%   141, 231-232, 244-248
api/product/_product.py                        356      26  92.70%   229, 387, 413-416, 453, 472-473, 582, 611, 618, 658, 708, 714, 851, 894-897, 906-909, 963, 1038, 1050, 1118
api/product/metadata_mapping.py                887      57  93.57%   136-138, 259-264, 288, 347-348, 428, 448-449, 454, 517-518, 555, 576-579, 602, 614-615, 656, 679, 704-707, 709-714, 793, 799, 815, 823, 1054, 1207, 1229, 1238-1242, 1259-1264, 1397, 1420, 1429, 1451, 1456, 1523, 1620, 1641, 1667, 1681, 1779, 1874, 1970, 2036, 2080
api/product/drivers/__init__.py                 11       0  100.00%
api/product/drivers/base.py                     32       0  100.00%
api/product/drivers/generic.py                  11       0  100.00%
api/product/drivers/sentinel1.py                33       0  100.00%
api/product/drivers/sentinel2.py                33       0  100.00%
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 38       3  92.11%   50, 84-85
plugins/manager.py                             198      13  93.43%   104-109, 139, 263, 267, 294, 333-334, 428-431
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             8       0  100.00%
plugins/apis/ecmwf.py                           95      10  89.47%   163-165, 213-214, 240-242, 273-274
plugins/apis/usgs.py                           175      25  85.71%   158, 246, 279, 322-324, 329, 357-358, 363, 393-400, 411-416, 438-444
plugins/authentication/__init__.py               6       1  83.33%   32
plugins/authentication/aws_auth.py             127      28  77.95%   69-70, 142-149, 193-212, 235, 267-271, 288, 312, 328-329
plugins/authentication/base.py                  26       4  84.62%   61, 74, 97, 111
plugins/authentication/eoiam.py                 98       1  98.98%   170
plugins/authentication/generic.py               16       3  81.25%   50, 55, 65
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       4  91.30%   154, 177-182
plugins/authentication/openid_connect.py       243      21  91.36%   101-102, 110-128, 175, 183, 197, 217, 363-366, 392, 433, 623-626
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              58       3  94.83%   68, 89, 135
plugins/authentication/token.py                141       9  93.62%   192, 229, 309-310, 358-362
plugins/authentication/token_exchange.py        36       3  91.67%   75, 114-115
plugins/crunch/__init__.py                       6       0  100.00%
plugins/crunch/base.py                          12       0  100.00%
plugins/crunch/filter_date.py                   47       0  100.00%
plugins/crunch/filter_latest_intersect.py       52       6  88.46%   87-88, 95-96, 98-102
plugins/crunch/filter_latest_tpl_name.py        35       0  100.00%
plugins/crunch/filter_overlap.py                66      10  84.85%   118-121, 137-163
plugins/crunch/filter_property.py               30       2  93.33%   81-84
plugins/download/__init__.py                     4       0  100.00%
plugins/download/aws.py                        406      43  89.41%   275, 309, 358-361, 391-392, 400-404, 482-485, 525-527, 531, 561-562, 568-572, 602, 669-681, 767, 784, 799, 831-832, 895, 921, 966-968, 1020
plugins/download/base.py                       297      24  91.92%   148, 178, 394-395, 439, 473, 550-554, 584, 619-620, 645-654, 713, 734, 756, 764
plugins/download/http.py                       582      73  87.46%   257, 299-302, 364-367, 370, 377-382, 414-415, 432, 447, 508, 543, 557, 571, 584-588, 604-609, 620, 639, 675-678, 699, 709, 716, 880, 912, 944-953, 989, 1034-1039, 1048, 1063-1065, 1069, 1072, 1087-1088, 1104, 1179, 1231, 1274-1275, 1287, 1297, 1353-1354, 1384, 1411, 1431, 1511-1512
plugins/search/__init__.py                      25       0  100.00%
plugins/search/base.py                         220      20  90.91%   119, 123, 147-153, 162, 228-231, 264, 314, 346, 367, 500, 585-588, 597
plugins/search/build_search_result.py          555      41  92.61%   258, 430-438, 520-521, 570, 582-584, 608, 610, 675, 683-687, 707, 717, 743, 792, 817, 879-884, 945, 972, 975, 1092, 1228, 1243, 1246, 1271, 1445-1446, 1490, 1499-1501, 1558
plugins/search/cop_ghsl.py                     410      27  93.41%   73-74, 109-110, 174, 176, 218, 298, 314, 360, 384, 406, 427, 461, 477, 512, 541-542, 556-558, 569-571, 613, 650, 654
plugins/search/cop_marine.py                   297      34  88.55%   56, 64-66, 82, 104, 106, 109, 175-176, 194, 218, 224, 228, 250-251, 259, 303, 313, 326, 334, 340-344, 361, 393-396, 411, 481, 493, 537-538
plugins/search/creodias_s3.py                   29       1  96.55%   59
plugins/search/csw.py                          118      12  89.83%   121, 165, 203, 214-225, 230
plugins/search/eumetsat_ds.py                   24       0  100.00%
plugins/search/geodes.py                        66       0  100.00%
plugins/search/oar.py                           82      13  84.15%   50, 57-60, 172-180, 186-188, 195
plugins/search/qssearch.py                     896      84  90.62%   381, 543-544, 567-568, 580-584, 880, 946, 980, 1058, 1079, 1082-1083, 1101, 1110-1111, 1129, 1200, 1209, 1221-1231, 1240, 1255, 1264, 1277, 1300, 1341, 1414, 1452, 1525-1526, 1528-1530, 1532-1533, 1805, 1808, 1812-1813, 1832-1835, 1847, 1869-1886, 1894-1900, 1958-1964, 1971, 2009, 2055, 2060-2061, 2091, 2211, 2215, 2226, 2250, 2263, 2271-2281, 2307
plugins/search/stac_list_assets.py              25       0  100.00%
plugins/search/static_stac_search.py            84      17  79.76%   99-127, 166-169, 182
resources/__init__.py                            0       0  100.00%
resources/shp/__init__.py                        0       0  100.00%
types/__init__.py                              180      43  76.11%   58, 71-75, 86-98, 126-128, 135-140, 155, 165, 247, 280, 290-306, 311, 313, 335, 340, 348, 358
types/bbox.py                                   39       0  100.00%
types/download_args.py                          10       0  100.00%
types/queryables.py                            113       0  100.00%
types/search_args.py                            71      18  74.65%   67-71, 78-95, 110
types/stac_extensions.py                       162       4  97.53%   455-461, 641
types/stac_metadata.py                         159      22  86.16%   98, 120, 153, 186-188, 202-216, 225-232, 259-262
utils/__init__.py                              658      21  96.81%   57, 216, 223, 239, 274, 351, 603-606, 646-647, 676, 1060-1063, 1133-1134, 1166, 1373, 1481, 1613
utils/cache.py                                  22       0  100.00%
utils/dates.py                                 191       1  99.48%   182
utils/deserialize.py                            74      10  86.49%   53, 55, 103, 108-109, 148, 162, 184-185, 191
utils/env.py                                     3       0  100.00%
utils/exceptions.py                             58       0  100.00%
utils/free_text_search.py                       65       2  96.92%   83, 91
utils/import_system.py                          28       6  78.57%   65-73
utils/logging.py                                42       3  92.86%   42-43, 61
utils/notebook.py                               44      13  70.45%   27, 41, 58-62, 72-78, 86-87
utils/repr.py                                   38       0  100.00%
utils/requests.py                               55       6  89.09%   94, 110, 118-120, 128
utils/s3.py                                    271      11  95.94%   205-206, 247, 263, 285, 507, 590-591, 655, 730, 756
utils/stac_reader.py                           113      42  62.83%   63-85, 101, 138, 154-159, 206-216, 226-256
utils/streamresponse.py                         91       4  95.60%   106, 166, 175, 187
utils/yaml.py                                   42       7  83.33%   49, 65, 70-74
TOTAL                                        11877     982  91.73%

Diff against develop

Filename                              Stmts    Miss  Cover
----------------------------------  -------  ------  -------
plugins/authentication/aws_auth.py       +3      -4  +3.76%
utils/s3.py                              +5       0  +0.08%
TOTAL                                    +8      -4  +0.04%

Results for commit: 4a02a64

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@github-actions

github-actions Bot commented Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

badge

Code Coverage (Windows)

Details 
Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     19       2  89.47%   61, 70
cli.py                                         259      11  95.75%   99-110, 376, 658
config.py                                      409      39  90.46%   84, 203, 227-242, 264, 269, 303-305, 308, 311, 314, 318, 322, 326-328, 822-824, 875, 1010-1012, 1036-1040, 1059, 1067, 1095-1100, 1102
crunch.py                                        2       0  100.00%
api/__init__.py                                  0       0  100.00%
api/collection.py                              154       7  95.45%   193, 229, 339, 378, 381, 399, 402
api/core.py                                    754      26  96.55%   157, 272, 284-295, 540, 584, 621, 661, 679, 750, 847-851, 873, 1026, 1031, 1293, 1364, 1664, 1889, 2046, 2050, 2159
api/provider.py                                414      44  89.37%   141-145, 212, 216-217, 354-375, 443, 559, 563-564, 569-572, 583, 644-645, 701-711, 896-897, 955, 962, 976, 999-1002, 1036-1043, 1047-1048
api/search_result.py                           138      12  91.30%   111, 123, 133, 154, 208, 266, 278, 406, 461-464
api/product/__init__.py                         26       4  84.62%   51-54
api/product/_assets.py                          78       5  93.59%   141, 231-232, 244-248
api/product/_product.py                        356      26  92.70%   229, 387, 413-416, 453, 472-473, 582, 611, 618, 658, 708, 714, 851, 894-897, 906-909, 963, 1038, 1050, 1118
api/product/metadata_mapping.py                887      57  93.57%   136-138, 259-264, 288, 347-348, 428, 448-449, 454, 517-518, 555, 576-579, 602, 614-615, 656, 679, 704-707, 709-714, 793, 799, 815, 823, 1054, 1207, 1229, 1238-1242, 1259-1264, 1397, 1420, 1429, 1451, 1456, 1523, 1620, 1641, 1667, 1681, 1779, 1874, 1970, 2036, 2080
api/product/drivers/__init__.py                 11       0  100.00%
api/product/drivers/base.py                     32       0  100.00%
api/product/drivers/generic.py                  11       0  100.00%
api/product/drivers/sentinel1.py                33       0  100.00%
api/product/drivers/sentinel2.py                33       0  100.00%
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 38       3  92.11%   50, 84-85
plugins/manager.py                             198      13  93.43%   104-109, 139, 263, 267, 294, 333-334, 428-431
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             8       0  100.00%
plugins/apis/ecmwf.py                           95      10  89.47%   163-165, 213-214, 240-242, 273-274
plugins/apis/usgs.py                           175      25  85.71%   158, 246, 279, 322-324, 329, 357-358, 363, 393-400, 411-416, 438-444
plugins/authentication/__init__.py               6       1  83.33%   32
plugins/authentication/aws_auth.py             127      28  77.95%   69-70, 142-149, 193-212, 235, 267-271, 288, 312, 328-329
plugins/authentication/base.py                  26       4  84.62%   61, 74, 97, 111
plugins/authentication/eoiam.py                 98       1  98.98%   170
plugins/authentication/generic.py               16       3  81.25%   50, 55, 65
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       4  91.30%   154, 177-182
plugins/authentication/openid_connect.py       243      21  91.36%   101-102, 110-128, 175, 183, 197, 217, 363-366, 392, 433, 623-626
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              58       3  94.83%   68, 89, 135
plugins/authentication/token.py                141       9  93.62%   192, 229, 309-310, 358-362
plugins/authentication/token_exchange.py        36       3  91.67%   75, 114-115
plugins/crunch/__init__.py                       6       0  100.00%
plugins/crunch/base.py                          12       0  100.00%
plugins/crunch/filter_date.py                   47       0  100.00%
plugins/crunch/filter_latest_intersect.py       52       6  88.46%   87-88, 95-96, 98-102
plugins/crunch/filter_latest_tpl_name.py        35       0  100.00%
plugins/crunch/filter_overlap.py                66      10  84.85%   118-121, 137-163
plugins/crunch/filter_property.py               30       2  93.33%   81-84
plugins/download/__init__.py                     4       0  100.00%
plugins/download/aws.py                        406      43  89.41%   275, 309, 358-361, 391-392, 400-404, 482-485, 525-527, 531, 561-562, 568-572, 602, 669-681, 767, 784, 799, 831-832, 895, 921, 966-968, 1020
plugins/download/base.py                       297      26  91.25%   148, 178, 253-255, 394-395, 439, 473, 550-554, 584, 619-620, 645-654, 713, 734, 756, 764
plugins/download/http.py                       582      73  87.46%   257, 299-302, 364-367, 370, 377-382, 414-415, 432, 447, 508, 543, 557, 571, 584-588, 604-609, 620, 639, 675-678, 699, 709, 716, 880, 912, 944-953, 989, 1034-1039, 1048, 1063-1065, 1069, 1072, 1087-1088, 1104, 1179, 1231, 1274-1275, 1287, 1297, 1353-1354, 1384, 1411, 1431, 1511-1512
plugins/search/__init__.py                      25       0  100.00%
plugins/search/base.py                         220      20  90.91%   119, 123, 147-153, 162, 228-231, 264, 314, 346, 367, 500, 585-588, 597
plugins/search/build_search_result.py          555      41  92.61%   258, 430-438, 520-521, 570, 582-584, 608, 610, 675, 683-687, 707, 717, 743, 792, 817, 879-884, 945, 972, 975, 1092, 1228, 1243, 1246, 1271, 1445-1446, 1490, 1499-1501, 1558
plugins/search/cop_ghsl.py                     410      27  93.41%   73-74, 109-110, 174, 176, 218, 298, 314, 360, 384, 406, 427, 461, 477, 512, 541-542, 556-558, 569-571, 613, 650, 654
plugins/search/cop_marine.py                   297      34  88.55%   56, 64-66, 82, 104, 106, 109, 175-176, 194, 218, 224, 228, 250-251, 259, 303, 313, 326, 334, 340-344, 361, 393-396, 411, 481, 493, 537-538
plugins/search/creodias_s3.py                   29       1  96.55%   59
plugins/search/csw.py                          118      12  89.83%   121, 165, 203, 214-225, 230
plugins/search/eumetsat_ds.py                   24       0  100.00%
plugins/search/geodes.py                        66       0  100.00%
plugins/search/oar.py                           82      13  84.15%   50, 57-60, 172-180, 186-188, 195
plugins/search/qssearch.py                     896      84  90.62%   381, 543-544, 567-568, 580-584, 880, 946, 980, 1058, 1079, 1082-1083, 1101, 1110-1111, 1129, 1200, 1209, 1221-1231, 1240, 1255, 1264, 1277, 1300, 1341, 1414, 1452, 1525-1526, 1528-1530, 1532-1533, 1805, 1808, 1812-1813, 1832-1835, 1847, 1869-1886, 1894-1900, 1958-1964, 1971, 2009, 2055, 2060-2061, 2091, 2211, 2215, 2226, 2250, 2263, 2271-2281, 2307
plugins/search/stac_list_assets.py              25       0  100.00%
plugins/search/static_stac_search.py            84      17  79.76%   99-127, 166-169, 182
resources/__init__.py                            0       0  100.00%
resources/shp/__init__.py                        0       0  100.00%
types/__init__.py                              180      43  76.11%   58, 71-75, 86-98, 126-128, 135-140, 155, 165, 247, 280, 290-306, 311, 313, 335, 340, 348, 358
types/bbox.py                                   39       0  100.00%
types/download_args.py                          10       0  100.00%
types/queryables.py                            113       0  100.00%
types/search_args.py                            71      18  74.65%   67-71, 78-95, 110
types/stac_extensions.py                       162       4  97.53%   455-461, 641
types/stac_metadata.py                         159      22  86.16%   98, 120, 153, 186-188, 202-216, 225-232, 259-262
utils/__init__.py                              658      21  96.81%   57, 216, 223, 239, 274, 351, 603-606, 646-647, 676, 1060-1063, 1133-1134, 1166, 1373, 1481, 1613
utils/cache.py                                  22       0  100.00%
utils/dates.py                                 191       1  99.48%   182
utils/deserialize.py                            74      10  86.49%   53, 55, 103, 108-109, 148, 162, 184-185, 191
utils/env.py                                     3       0  100.00%
utils/exceptions.py                             58       0  100.00%
utils/free_text_search.py                       65       2  96.92%   83, 91
utils/import_system.py                          28       6  78.57%   65-73
utils/logging.py                                42       3  92.86%   42-43, 61
utils/notebook.py                               44      13  70.45%   27, 41, 58-62, 72-78, 86-87
utils/repr.py                                   38       0  100.00%
utils/requests.py                               55       6  89.09%   94, 110, 118-120, 128
utils/s3.py                                    271      11  95.94%   205-206, 247, 263, 285, 507, 590-591, 655, 730, 756
utils/stac_reader.py                           113      42  62.83%   63-85, 101, 138, 154-159, 206-216, 226-256
utils/streamresponse.py                         91       4  95.60%   106, 166, 175, 187
utils/yaml.py                                   42       7  83.33%   49, 65, 70-74
TOTAL                                        11877     984  91.72%

Diff against develop

Filename                              Stmts    Miss  Cover
----------------------------------  -------  ------  -------
plugins/authentication/aws_auth.py       +3      -4  +3.76%
utils/s3.py                              +5       0  +0.08%
TOTAL                                    +8      -4  +0.04%

Results for commit: 4a02a64

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@anesson-cs
fix: aws filter fallback and chunks sorting
8dc41dc 
remove a lot of linter 'ruff' warnings

use 'mypy_boto3_s3' classes for typing
@anesson-cs anesson-cs changed the title fix: aws filter fallback and chunk sorting fix: aws filter fallback and chunks sorting Sep 3, 2026
@anesson-cs
anesson-cs requested a review from sbrunato September 3, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sbrunato sbrunato Awaiting requested review from sbrunato

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@anesson-cs @sbrunato