Skip to content

fix(core): declare the dataset fields that core paradigms read - #891

Open
akshita317 wants to merge 1 commit into
kubeedge:mainfrom
akshita317:fix/dataset-register-paradigm-keys
Open

fix(core): declare the dataset fields that core paradigms read#891
akshita317 wants to merge 1 commit into
kubeedge:mainfrom
akshita317:fix/dataset-register-paradigm-keys

Conversation

@akshita317

Copy link
Copy Markdown

What type of PR is this?

/kind bug

What this PR does / why we need it:

Dataset._parse_config (core/testenvmanager/dataset/dataset.py:71-73) assigns only keys that already exist on the instance:

for attr, value in config.items():
    if attr in self.__dict__:
        self.__dict__[attr] = value

Four fields are read off the Dataset instance by core paradigm code, but are not declared in Dataset.__init__. Any value a testenv.yaml supplies for them is therefore discarded without warning, and the later attribute access raises:

field read by
val_url core/.../singletask_learning_tta.py:77
known_dataset_url core/.../singletask_learning_active_boost.py:61
unknown_dataset_url core/.../singletask_learning_active_boost.py:62
image_folder_url core/.../singletask_learning_active_boost.py:49,63 and singletask_learning_tta.py:46,76

Reproduced on main against the real classes, using examples/yaoba/singletask_learning_boost/testenv/testenv.yaml unmodified:

declared in YAML : ['image_folder_url', 'known_dataset_url', 'test_url',
                    'train_url', 'unknown_dataset_url', 'val_url']
present on object: ['label', 'test_data', 'test_data_info', 'test_index',
                    'test_url', 'train_data', 'train_data_info',
                    'train_index', 'train_url']

dataset.image_folder_url    -> AttributeError: 'Dataset' object has no attribute 'image_folder_url'
dataset.known_dataset_url   -> AttributeError: 'Dataset' object has no attribute 'known_dataset_url'
dataset.unknown_dataset_url -> AttributeError: 'Dataset' object has no attribute 'unknown_dataset_url'

git log --all -S image_folder_url -- core/testenvmanager/dataset/dataset.py returns nothing, so these were never declared. The paradigms that read them and the example that configures them both arrived in 7111198, without the matching registration.

The change is four attribute declarations defaulting to "", consistent with the surrounding fields. It is purely additive — examples that do not set these keys keep the same empty-string default and behave exactly as before.

Verification

check result
the attribute reads above resolve to the configured paths (3/3 → 0/3 failing)
Dataset constructed for every examples/**/testenv*.yaml 48 ok, 0 failed
Rank constructed for every rank: block 43 ok, 0 failed
pylint --max-positional-arguments=10 core/ 10.00/10, unchanged from main

Not verified: a full ianvs -f run of examples/yaoba. That example pins mmcv-full 1.7.1 and mmdet 2.28.2 and needs the YaoBa image corpus, which has no public download, so I could not execute it end to end and am not claiming it passes.

Which issue(s) this PR fixes:

Related to #742 and #823.

Special notes for your reviewer:

This does not on its own make examples/yaoba run — the paradigm dispatch registration in #824 (@31groot) is also needed, and the two changes are independent and do not conflict.

What it does do is make the path corrections in #796 (@hibaa23) and in #824 effective. Both correct image_folder_url to a repo-relative path, and at present that value cannot reach core because the key is dropped here. @hibaa23 confirmed on #796 that they are happy to defer the config-line edit to whichever PR carries the core registration; this PR deliberately touches no example file, so it should not conflict with either.

@Nikhil-VS1811, who is assigned to #742, and I split the work on that issue: this registration is the piece I picked up, and they are looking at the broader silent-drop pattern across the other example families.

Scope is deliberately narrow. Auditing all 48 examples/**/testenv*.yaml against the fields Dataset.__init__ declares, 12 configs declare at least one dropped key, but cross-checking every self.dataset.<attr> read in core/ against those declarations closes the set at exactly these four:

undeclared but read: ['image_folder_url', 'known_dataset_url', 'unknown_dataset_url', 'val_url']

The dropped keys in the other 10 configs (cifar100, federated-llm, PIPL, industrialEI) have no consumer anywhere in core/ or examples/, so they are dead config or human-readable metadata rather than this defect, and registering them would be speculative. Full audit and script in #742.

Does this PR introduce a user-facing change?:

Dataset configuration keys read by the single-task learning TTA and active-boost paradigms (val_url, known_dataset_url, unknown_dataset_url, image_folder_url) are now honoured instead of being silently discarded.

Dataset._parse_config assigns only keys that already exist on the instance:

    for attr, value in config.items():
        if attr in self.__dict__:
            self.__dict__[attr] = value

Four fields are read off the Dataset instance by core paradigm code but are
not declared in Dataset.__init__, so any value a testenv.yaml supplies for
them is silently discarded and the later attribute access raises:

    val_url             core/.../singletask_learning_tta.py:77
    known_dataset_url   core/.../singletask_learning_active_boost.py:61
    unknown_dataset_url core/.../singletask_learning_active_boost.py:62
    image_folder_url    core/.../singletask_learning_active_boost.py:49,63
                        core/.../singletask_learning_tta.py:46,76

Reproduced against the real classes on main, using the example config
unmodified:

    dataset.image_folder_url    -> AttributeError: 'Dataset' object has no
                                   attribute 'image_folder_url'
    dataset.known_dataset_url   -> AttributeError
    dataset.unknown_dataset_url -> AttributeError

git log -S image_folder_url over core/testenvmanager/dataset/dataset.py
returns nothing, so these were never declared; the paradigms that read them
and the example that configures them arrived together in 7111198 without the
corresponding registration.

The change is four attribute declarations defaulting to the empty string,
matching the surrounding fields. It is additive: examples that do not set
these keys keep the same empty-string default and behave exactly as before.

Verified after the change:
  - the three attribute reads above resolve to the configured paths
  - Dataset constructs for all 48 examples/**/testenv*.yaml   (48 ok, 0 failed)
  - Rank constructs for all 43 rank blocks                     (43 ok, 0 failed)
  - pylint --max-positional-arguments=10 core/  stays at 10.00/10

This does not by itself make examples/yaoba run; the paradigm dispatch work in
kubeedge#824 is also required, and the two are independent. It does make
the path corrections in kubeedge#796 and kubeedge#824 take effect,
which they currently cannot.

Signed-off-by: Akshita <110122283+akshita317@users.noreply.github.com>
@kubeedge-bot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: akshita317
To complete the pull request process, please assign moorezheng after the PR has been reviewed.
You can assign the PR to them by writing /assign @moorezheng in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubeedge-bot kubeedge-bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Aug 23, 2026
@akshita317

Copy link
Copy Markdown
Author

/assign @MooreZheng
/cc @hsj576

Small Core change (10 lines, additive) unblocking the yaoba path fixes in #796 and #824. Verified against all 48 example testenv configs and all 43 rank blocks with no regression; details in the description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Categorizes issue or PR as related to a bug. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants