Refactor experiment status traceability

[ntorqulu]

Closes #2049, #2137

Things done

• Extended available experiment status in experiment_status table. Possible experiment status are: NOT_RUNNING, RUNNING, ARCHIVED and DELETED.
• Experiments created (expid and create) are marked as NOT_RUNNING. Implementation left open if we want to add more states in the future.
• Keep traceability of deleted experiments in both autosubmit.db and as_times.db. Once the experiment is deleted, it's kept in the database with status DELETED (tombstone).
• Added column last_heartbeat to table experiment_status. Each experiment creates a daemon thread (autosubmit-heartbeat-{expid}) that updates the last_heartbeat timestamp every 2 min while the experiment is running.
• Enforce unique indx on name in experiment_status table, kept most recent one.

Check List

• I have read CONTRIBUTING.md.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to pyproject.toml.
• Tests are included (or explain why tests are not needed).
• Changelog entry included in CHANGELOG.md if this is a change that can affect users.
• Documentation updated.
• If this is a bug fix, PR should include a link to the issue (e.g. Closes #1234).

[ntorqulu]

PROBLEMS/OBSERVATIONS:

1. experiment_status.db allows repeated experiment ids as rows

What I mean by that: you can have multiple experiements with the same expid in the db. It should be enforced uniqueness of the name column too.

2. AS already creates the experiment_status db. This step does not depend on the api.

So, I will be investigating more and add some tests for this case, but #2641 is solved

3. Enforce single source of truth

I'll enforce that only AS backend writes into the experiment_status.db. API should not be able to write into the db, only read the info stored

[ntorqulu]

Which should be the possible states for an experiment?
Current RunningStatus:

• RUNNING
• NOT_RUNNING

Should we track when:

• an experiment is paused -> not running or paused?
• an experiment has failed -> not running or failed?
• experiment archived and experiment deleted -> yes based on specifications with one state for each

[codecov-commenter]

Codecov Report

❌ Patch coverage is 94.55446% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.98%. Comparing base (018eb63) to head (c9b86d4).

Files with missing lines	Patch %	Lines
autosubmit/autosubmit.py	66.66%	7 Missing and 3 partials ⚠️
autosubmit/experiment/experiment_common.py	87.50%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2980      +/-   ##
==========================================
+ Coverage   77.75%   77.98%   +0.23%     
==========================================
  Files          91       91              
  Lines       20185    20327     +142     
  Branches     3879     3889      +10     
==========================================
+ Hits        15694    15852     +158     
+ Misses       3548     3531      -17     
- Partials      943      944       +1     

Flag	Coverage Δ
fast-tests	77.98% <94.55%> (+0.23%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ntorqulu]

@LuiggiTenorioK I've a few questions about how to handle the statuses in the experient_status database.

I'm thinking about future maintainability of the databases (both experiment_status and experiment tables) and I'm not sure if it's correct to store the DELETED experiments... Why the GUI needs to show the deleted experiments? Is not possible that if a experiment is deleted it's trace is completely lost?
I understand that they want to visualize the archived ones, but why would they want to visualize the deleted ones?

[kinow]

I can answer on the deleted part, @ntorqulu . Not sure about the part of showing the experiment, but we want to keep the register of a deleted experiment so we know it was deleted, and so that we cannot re-use the expid.

Keeping deleted entries in a DB is sometimes called a tombstone entry -- https://en.wikipedia.org/wiki/Tombstone_(data_store). We can purge deleted entries after some time if needed, but it could be useful for traceability.

At the moment, the main problem we have with the deleted experiments is this one: #1072

[ntorqulu]

Great, thanks for the explanation @kinow! It makes sense to delete them in batch once the current_time - modified_time is greater than a specified threshold if necessary.

About the issue with the deleted experiments, yes, I've been facing the same problem. If you:

• expid & create & run an experiment
• delete the experiment -> then this experiment is deleted from the experiment table but not from the experiment_status table.
• expid & create & run -> it will assign the same expid that was previously deleted, because it's no longer present in the experiment table. And will then create a duplication of expid in the experiment_status table because in this table the experiment row was not deleted...

For now, I'll enforce uniqueness in expid column and do not delete experiment rows in neither table. The only thing that will be updated is the status of the experiment to DELETED. For the old ones, I'll keep the newest row with the same expid.

[LuiggiTenorioK]

@ntorqulu is just as Bruno mentioned above. A logical delete is almost always a must if we want to keep traceability, especially if we don't want to reuse expids.

In the GUI, it is also nice to know if the experiment has been deleted to keep the user informed (probably with a message like this experiment has been deleted on 20XX-XX-XX at XX:XX.

Also, take into account that Autosubmit is not the only one that modifies the experiment_status data. The API does it as well every 5 minutes, meaning that your changes to change the status to DELETED will be quickly overwritten by the API, which will set it as NOT RUNNING. So, in every case, we have to modify the API to better orchestrate the status management.

[ntorqulu]

Yes, I've this in mind! I was just checking first the backend because it's what I'm currently more familiar.
The idea would be that the API only overwrites statuses for RUNNING experiments when the heartbeat is outdated (should define what outdated means, 15 min maybe? I'm not sure about this).

How do you want to handle the API modifications @LuiggiTenorioK? I open an issue for you to handle it or I assign it to myself to continue the refactor in the API?

[LuiggiTenorioK]

The idea would be that the API only overwrites statuses for RUNNING experiments when the heartbeat is outdated (should define what outdated means, 15 min maybe? I'm not sure about this).

Filtering by status seems like a good idea. I'll probably include the NOT RUNNING as well to do a sanity check in case Autosubmit failed to set it up as RUNNING.

How do you want to handle the API modifications @LuiggiTenorioK? I open an issue for you to handle it or I assign it to myself to continue the refactor in the API?

First, I think it's better if we have an issue to discuss the modifications to be made in the API. I'll create it and tag you to continue the thread there.

[ntorqulu]

Okay! I'll take a look @LuiggiTenorioK . Thanks!
About the state diagram of an experiment, I've a few doubts about the granularity of the detail for the states of the experiments and if we should track also experiments that are created but not run yet.

1. Currently RUNNING experiments can be archived, but RUNNING experiments cannot be deleted. Is it correct? Should only NOT_RUNNING experiments be archived?

2. When creating the exeperiment (EXPID and CREATE), the experiment is not stored in the experiment_status table. Only when the experiment has been archived, deleted or run it appears as a new entry. Should we enforce the existance in the table once the experiment is created?

3. NOT_RUNNING is a very broad status. It captures paused experiments, failed ones and just the ones that have finished the execution. Is it correct to keep it as NOT_RUNNING?

[LuiggiTenorioK]

1. Currently RUNNING experiments can be archived, but RUNNING experiments cannot be deleted. Is it correct? Should only NOT_RUNNING experiments be archived?

RUNNING experiments shouldn't be archived. This could potentially cause experiments to crash and generate corrupted data.

2. When creating the exeperiment (EXPID and CREATE), the experiment is not stored in the experiment_status table. Only when the experiment has been archived, deleted or run it appears as a new entry. Should we enforce the existance in the table once the experiment is created?

To keep data consistent, we should

3. NOT_RUNNING is a very broad status. It captures paused experiments, failed ones and just the ones that have finished the execution. Is it correct to keep it as NOT_RUNNING?

It is not incorrect to have it like that.

RUNNING and NOT_RUNNING experiments can have failed jobs, so splitting from there will diversify all states because of mutual exclusivity. IMO, experiments with failed jobs are more an observation than a state.

Then, "paused" and "completed" (or better said, "stopped" (because there is no pause command, just stop or kill the process) and "finished" (because there might be pending tasks to do, like restart failed jobs)) with an additional "created" state could be valid states to split the NOT_RUNNING state. However, I think there is enough complexity in this issue to add those. I suggest keeping the same 4 states for now, while leaving the solution extensible for this change in the future.

[kinow]

+1 to what @LuiggiTenorioK said. I think @ntorqulu 's state diagram comes from the code. I don't think the archive command checks for the lock file.

[ntorqulu]

Yes, the current state diagram is from the code, I'll update it with @LuiggiTenorioK's suggestions before changing anything in the code. This way we can have it as reference.

• commands expid and create -> set experiment as NOT RUNNING
• from RUNNING cannot transition to ARCHIVE
• from NOT RUNNING it can transition to ARCHIVE and to DELETE
• status STOPPED, FAILED, COMPLETED will be mapped to NOT RUNNING for now, taking into account maintainability and scalability for future states

[ntorqulu]

[ntorqulu]

QUESTION: What to do with old experiments and its last_heartbeat value. Leave as NULL or prefill with modified time?

[LuiggiTenorioK]

QUESTION: What to do with old experiments and its last_heartbeat value. Leave as NULL or prefill with modified time?

last_heartbeat won't always be equal to modified time? I think last_heartbeat might be redundant in this case.

[ntorqulu]

From what I'm understanding, modified is more like a timestamp for the table, that keeps track of when the last modifications were done in the row (change of status, change of the last_heartbeat etc...).
And last_heartbeat acts like an i'm-alive signal while the experiment is running, updating its time while there are active jobs in the experiment.
For instance, if an experiment transition from NOT RUNNING to ARCHIVED currently status and modified columns get updated, but not last_heartbeat. last_heartbeat keeps the last value it had while the status was RUNNING.
If we use modified as the single column to store the heartbeat too, we will lose the timestamp of the last run when the experiment transitions to NOT RUNNING.

[ntorqulu]

Also is it usefull to have the timestamp when the experiment was created?

[LuiggiTenorioK]

From what I'm understanding, modified is more like a timestamp for the table, that keeps track of when the last modifications were done in the row (change of status, change of the last_heartbeat etc...). And last_heartbeat acts like an i'm-alive signal while the experiment is running, updating its time while there are active jobs in the experiment. For instance, if an experiment transition from NOT RUNNING to ARCHIVED currently status and modified columns get updated, but not last_heartbeat. last_heartbeat keeps the last value it had while the status was RUNNING.

Ok, it makes sense like that. This heartbeat will serve to determine if the experiment stopped abruptly and didn't achieve the change in status to NOT_RUNNING.

If we use modified as the single column to store the heartbeat too, we will lose the timestamp of the last run when the experiment transitions to NOT RUNNING.

That timestamp should also be available in the experiment_run table. Do you think that modified timestamp in the experiment_run table could work as the heartbeat, or is it better to keep it separated?

Also is it usefull to have the timestamp when the experiment was created?

Yes, this is another issue, because we currently use the creation time of the dir as the experiment creation, and we don't want to rely on the file system. More likely, we can have it as another column in the experiment table.

[ntorqulu]

Ok, it makes sense like that. This heartbeat will serve to determine if the experiment stopped abruptly and didn't achieve the change in status to NOT_RUNNING.

Yes, this way we can filter even more the experiments the API can modify. For now I'll leave last_heartbeat as NULL, only updated when the experiment is RUNNING. Not pre-filled with modified for old experiments.
This way we don't introduce false data, the API will handle that.

That timestamp should also be available in the experiment_run table. Do you think that modified timestamp in the experiment_run table could work as the heartbeat, or is it better to keep it separated?

I've been debugging where modified is updated for this table in run_experiment method.
Around line 2324 it calls

exp_history = Autosubmit.process_historical_data_iteration(job_list, job_changes_tracker,
                                                                                       expid)

The problem is that only when there are job changes the modified column gets updated. So we are in the same situation as before, modified acts more as a time-of-edit of the row rather than a keep-alive signal of the experiment.
If it's necessary to have a keep-alive signal in table experiment_run, I would add the last_heartbeat there too.
Could happen that an experiment is RUNNING but it's job statuses remain the same during a long period of time. The last_heartbeat would keep being updated (because the job is running), but the modified time would be fixed for this period.

Yes, this is another issue, because we currently use the creation time of the dir as the experiment creation, and we don't want to rely on the file system. More likely, we can have it as another column in the experiment table.

Is it ok to leave it for another pr? Or better implement it in this one too?

[LuiggiTenorioK]

Is it ok to leave it for another pr? Or better implement it in this one too?

It seems unrelated to this PR. I would leave it for another one.

[ntorqulu]

Is it ok to leave it for another pr? Or better implement it in this one too?

It seems unrelated to this PR. I would leave it for another one.

Okay! I'll open a new issue to not forget this.

[ntorqulu]

QUESTION:

1. Currently autosubmit install creates only database autosubmit.db with tables experiment and db_version.
   I'm not sure why:

• table details from autosubmit.db is not created
• database as_times.db with table experiment_status is not initialized
  I would initialize them all together with the install command.

2. With both databases empty, when executing create:

• autosubmit create <EXPID with a configuration folder stored> -> Incorrect. Does not raise error and does not insert the experiment into autosubmit.db and neither in as_times.db. I'm not sure if it's better to raise an error or insert the experiment in the database in this case. Probably insert it to be more robust to fallbacks, because the configuration is correctly stored in its experiment folder.

autosubmit create <EXPID not created yet> -> raise error. Correct.

3. With both databases empty, when executing expid:

• currently it autofills the experiment table with consecutive experiment names, with consecutive id's and all of them with description = No description and autosubmit_version = 3.14.0 until it reaches the first expid that does not have a configuration folder assigned (the first one, not the last one!). For this one it correctly assigns the description given in the command and autosubmit version 4.1.17.
  Incorrect. I'm not sure about this one. Iterate through all the configuration folders to insert again the stored experiments and assign always the highest expid to a new experiment (not the first not assigned yet!)?

[LuiggiTenorioK]

QUESTION:

1. Currently autosubmit install creates only database autosubmit.db with tables experiment and db_version.
   I'm not sure why:

• table details from autosubmit.db is not created

The details table was only maintained by the API before, and it was also its responsibility to create it. However, now that Autosubmit also makes changes in that table, it should create the table as well. I think we just forgot to add that create statement in the implementation.

• database as_times.db with table experiment_status is not initialized

Same as the details table, but take into account that the API machine and the Autosubmit one can be different Linux users. We should chmod it to allow write for the group of users.

I would initialize them all together with the install command.

Agree +1

2. With both databases empty, when executing create:

• autosubmit create <EXPID with a configuration folder stored> -> Incorrect. Does not raise error and does not insert the experiment into autosubmit.db and neither in as_times.db. I'm not sure if it's better to raise an error or insert the experiment in the database in this case. Probably insert it to be more robust to fallbacks, because the configuration is correctly stored in its experiment folder.

autosubmit create <EXPID not created yet> -> raise error. Correct.

Currently, neither table is essential for running an experiment. So, probably it is ok to keep it like that because it is worse if the user cannot run the experiment than having that traceability.

3. With both databases empty, when executing expid:

• currently it autofills the experiment table with consecutive experiment names, with consecutive id's and all of them with description = No description and autosubmit_version = 3.14.0 until it reaches the first expid that does not have a configuration folder assigned (the first one, not the last one!). For this one it correctly assigns the description given in the command and autosubmit version 4.1.17.
  Incorrect. I'm not sure about this one. Iterate through all the configuration folders to insert again the stored experiments and assign always the highest expid to a new experiment (not the first not assigned yet!)?

I saw that happening before, and it is annoying. For me, it shouldn't do that. However, I am not 100% sure if removing that process can affect other things; maybe @kinow knows better.

[ntorqulu]

Same as the details table, but take into account that the API machine and the Autosubmit one can be different Linux users. We should chmod it to allow write for the group of users.

Okay thanks! I was not thinking about this. I see that install is not applying chmod in autosubmit.db neither. I'll apply it to both (autosubmit.db and as_times.db).

Currently, neither table is essential for running an experiment. So, probably it is ok to keep it like that because it is worse if the user cannot run the experiment than having that traceability.

Mmm okay. I was thinking that when user executes create, before inserting the status into the experiment_status table:

• check if the experiment exists in autosubmit.db
• if it does not exist (db empty but config files exists) automatically insert it into autosubmit.db?
• if registration into autosubmit.db fails, log warning but do not fail, continue with the execution

But I don't know if it's in the scope of this pr. It's an edge case when the exp has config files but is not registered into autosubmit.db.

[LuiggiTenorioK]

Mmm okay. I was thinking that when user executes create, before inserting the status into the experiment_status table:

• check if the experiment exists in autosubmit.db
• if it does not exist (db empty but config files exists) automatically insert it into autosubmit.db?
• if registration into autosubmit.db fails, log warning but do not fail, continue with the execution

But I don't know if it's in the scope of this pr. It's an edge case when the exp has config files but is not registered into autosubmit.db.

Yes, I think it goes a little bit outside the scope. It is an edge case of another issue. In this PR, it is better to assume there is a correct common setup.

Also, the creation of the tables and permissions can be handled in a different PR. There is also an existing separated issue related to that: #2641

[ntorqulu]

Databases section of the documentation should be updated. Currently only changed the part related to the as_times.db to avoid making this pr bigger.
But the rst is also referencing some status.db and a test folder I don't find anywhere.

[LuiggiTenorioK]

autosubmit/metadata/test/status.db used to exist to measure the IO latency of /esarchive at BSC. This was one of the first things I removed from the old API since we wanted Autosubmit to be a generalized tool rather than one made for BSC.