Clear pool id if volume allocation fails by anniejili · Pull Request #8202 · apache/cloudstack

anniejili · 2023-11-08T18:40:12Z

Description

When there is outage in storage API coincides with a VM start, the volume could be stuck in "Allocated" state but with a pool Id assigned in DB. It blocked the ability to start the VM during planDeployment process, in the method storagePoolCompatibleWithVolumePool() shown below when validating a volume with a storage pool, if the volume's pool Id is not null and state is Allocated, it would always return false for any storage pools. As a result, the allocator cannot find a suitable pool and it blocks the VM start process.

@Override
    public boolean storagePoolCompatibleWithVolumePool(StoragePool pool, Volume volume) {
        if (pool == null || volume == null) {
            return false;
        }

        if (volume.getPoolId() == null) {
            // Volume is not allocated to any pool. Not possible to check compatibility with other pool, let it try
            return true;
        }

        StoragePool volumePool = _storagePoolDao.findById(volume.getPoolId());
        if (volumePool == null) {
            // Volume pool doesn't exist. Not possible to check compatibility with other pool, let it try
            return true;
        }

        if (volume.getState() == Volume.State.Ready) {
            if (volumePool.getPoolType() == Storage.StoragePoolType.PowerFlex && pool.getPoolType() != Storage.StoragePoolType.PowerFlex) {
                return false;
            } else if (volumePool.getPoolType() != Storage.StoragePoolType.PowerFlex && pool.getPoolType() == Storage.StoragePoolType.PowerFlex) {
                return false;
            }
        } else {
            return false;
        }

        return true;
    }

This PR fixed the issue above by validating volumes's state and pool id before the deployment in DeploymentPlanningManagerImpl.java's findSuitablePoolsForVolumes() method. Before DeploymentPlanningManager is trying to find suitable storage pools for volumes, if any volume is in Allocated state with a pool ID, the pool id will be cleared.
Testing
For a stopped VM, manually updated its root volume state to "Allocated" with a pool ID. Then restart the VM.

Before the change

2023-08-22 23:40:00,948 INFO [o.a.c.a.c.u.v.StartVMCmd] (API-Job-Executor-70:ctx-xxxx job-xxxx ctx-xxxx) (logid:xxxx) No destination found for a deployment for VM instance {id: "xxxx", name: "xxxx, uuid: "xxxx", type="User"}

After the change

The VM is started successfully.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

The change is tested via added junit test cases. The change is also tested and verified in Cloud Stack development environment.

…located with a pool id assigned

boring-cyborg · 2023-11-08T18:40:16Z

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

In case of a new feature add useful documentation (raise doc PR at https://github.com/apache/cloudstack-documentation)
Be patient and persistent. It might take some time to get a review or get the final approval from the committers.
Pay attention to the quality of your code, ensure tests are passing and your PR doesn't have conflicts.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Issues, Mailing list and Slack.
Be sure to read the CloudStack Coding Conventions.
Apache CloudStack is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@cloudstack.apache.org (https://cloudstack.apache.org/mailing-lists.html)
Slack: https://apachecloudstack.slack.com/

codecov · 2023-11-09T06:09:28Z

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (1a2dbeb) 13.10% compared to head (91896cd) 13.12%.

Files	Patch %	Lines
...om/cloud/deploy/DeploymentPlanningManagerImpl.java	42.85%	1 Missing and 3 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.18    #8202      +/-   ##
============================================
+ Coverage     13.10%   13.12%   +0.01%     
- Complexity     9122     9135      +13     
============================================
  Files          2720     2720              
  Lines        257638   257645       +7     
  Branches      40168    40171       +3     
============================================
+ Hits          33753    33805      +52     
+ Misses       219619   219548      -71     
- Partials       4266     4292      +26

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

yadvr · 2023-11-09T06:32:22Z

@blueorangutan package

blueorangutan · 2023-11-09T06:44:03Z

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-11-09T07:46:08Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7690

harikrishna-patnala · 2023-11-09T09:12:17Z

@blueorangutan test

blueorangutan · 2023-11-09T09:14:03Z

@harikrishna-patnala a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

harikrishna-patnala

LGTM, Tested both root and data disks, VM started fine.

DaanHoogland

clgtm, one worry though; @anniejili Do we need to check if the volume is actually present on the pool? I.E. if the volume has been allocated before it might actually exist (even though it has the state of Allocated)

harikrishna-patnala · 2023-11-10T05:25:11Z

clgtm, one worry though; @anniejili Do we need to check if the volume is actually present on the pool? I.E. if the volume has been allocated before it might actually exist (even though it has the state of Allocated)

@DaanHoogland according to the volume state machine, volume will be in allocated state either as the starting state or when the operation is failed. So I dont think volume will actually present. I feel this code check is good enough.

DaanHoogland · 2023-11-10T08:05:07Z

clgtm, one worry though; @anniejili Do we need to check if the volume is actually present on the pool? I.E. if the volume has been allocated before it might actually exist (even though it has the state of Allocated)

@DaanHoogland according to the volume state machine, volume will be in allocated state either as the starting state or when the operation is failed. So I dont think volume will actually present. I feel this code check is good enough.

ok @harikrishna-patnala.

DaanHoogland

CLGTM, simpel principle.

However I accept @harikrishna-patnala 's argument, it won't hurt to try if they can break it, as per the discussion here: #8202 (comment)

DaanHoogland · 2023-11-14T08:22:06Z

+            if (toBeCreated.getState() == Volume.State.Allocated && toBeCreated.getPoolId() != null) {
+                toBeCreated.setPoolId(null);
+                if (!_volsDao.update(toBeCreated.getId(), toBeCreated)) {
+                    throw new CloudRuntimeException(String.format("Error updating volume [%s] to clear pool Id.", toBeCreated.getId()));
+                }
+                if (s_logger.isDebugEnabled()) {
+                    String msg = String.format("Setting pool_id to NULL for volume id=%s as it is in Allocated state", toBeCreated.getId());
+                    s_logger.debug(msg);
+                }
+            }


this can be moved to a separate method and tested (separately)

yadvr · 2023-11-14T17:23:29Z

@blueorangutan package

blueorangutan · 2023-11-14T17:24:04Z

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-11-14T18:25:13Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7755

DaanHoogland · 2023-11-17T10:06:50Z

@blueorangutan test

blueorangutan · 2023-11-17T10:08:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

…me_allocation_fails

anniejili · 2023-11-17T19:46:58Z

Rebased with the latest 4.18 branch.

blueorangutan · 2023-11-18T01:14:34Z

[SF] Trillian test result (tid-8352)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43413 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8202-t8352-kvm-centos7.zip
Smoke tests completed. 109 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland · 2023-11-18T11:24:30Z

@blueorangutan package

blueorangutan · 2023-11-18T11:26:04Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-11-18T12:25:55Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7794

DaanHoogland · 2023-11-18T17:37:18Z

@blueorangutan test

blueorangutan · 2023-11-18T17:38:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-11-19T07:12:53Z

[SF] Trillian test result (tid-8373)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47393 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8202-t8373-kvm-centos7.zip
Smoke tests completed. 106 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_upgrade_kubernetes_cluster	`Failure`	560.57	test_kubernetes_clusters.py
test_08_migrate_vm	`Error`	43.88	test_vm_life_cycle.py
test_02_redundant_VPC_default_routes	`Failure`	370.05	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Failure`	572.56	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	572.58	test_vpc_redundant.py

boring-cyborg · 2023-11-21T10:11:09Z

Awesome work, congrats on your first merged pull request!

anniejili · 2023-11-28T18:13:09Z

Thanks everyone!

Annie Li added 2 commits November 6, 2023 13:29

clear pool id if volume allocation fails and leave volume state as Al…

832b924

…located with a pool id assigned

clear_pool_id_if_volume_allocation_fails

a8a930f

yadvr requested review from DaanHoogland, GabrielBrascher, harikrishna-patnala, mlsorensen and shwstppr November 9, 2023 06:32

harikrishna-patnala approved these changes Nov 9, 2023

View reviewed changes

DaanHoogland reviewed Nov 9, 2023

View reviewed changes

yadvr added this to the 4.18.2.0 milestone Nov 9, 2023

harikrishna-patnala requested review from DaanHoogland, nvazquez and weizhouapache and removed request for shwstppr November 14, 2023 03:58

DaanHoogland approved these changes Nov 14, 2023

View reviewed changes

DaanHoogland closed this Nov 14, 2023

DaanHoogland reopened this Nov 14, 2023

Merge remote-tracking branch 'origin/4.18' into clear_pool_id_if_volu…

91896cd

…me_allocation_fails

harikrishna-patnala merged commit 3c7c75b into apache:4.18 Nov 21, 2023

anniejili deleted the clear_pool_id_if_volume_allocation_fails branch November 28, 2023 18:13

Uh oh!

Conversation

anniejili commented Nov 8, 2023

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

boring-cyborg Bot commented Nov 8, 2023

Uh oh!

codecov Bot commented Nov 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yadvr commented Nov 9, 2023

Uh oh!

blueorangutan commented Nov 9, 2023

Uh oh!

blueorangutan commented Nov 9, 2023

Uh oh!

harikrishna-patnala commented Nov 9, 2023

Uh oh!

blueorangutan commented Nov 9, 2023

Uh oh!

harikrishna-patnala left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harikrishna-patnala commented Nov 10, 2023

Uh oh!

DaanHoogland commented Nov 10, 2023

Uh oh!

DaanHoogland left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DaanHoogland Nov 14, 2023

Choose a reason for hiding this comment

Uh oh!

yadvr commented Nov 14, 2023

Uh oh!

blueorangutan commented Nov 14, 2023

Uh oh!

blueorangutan commented Nov 14, 2023

Uh oh!

DaanHoogland commented Nov 17, 2023

Uh oh!

blueorangutan commented Nov 17, 2023

Uh oh!

anniejili commented Nov 17, 2023

Uh oh!

blueorangutan commented Nov 18, 2023

Uh oh!

DaanHoogland commented Nov 18, 2023

Uh oh!

blueorangutan commented Nov 18, 2023

Uh oh!

blueorangutan commented Nov 18, 2023

Uh oh!

DaanHoogland commented Nov 18, 2023

Uh oh!

blueorangutan commented Nov 18, 2023

Uh oh!

blueorangutan commented Nov 19, 2023

Uh oh!

boring-cyborg Bot commented Nov 21, 2023

Uh oh!

anniejili commented Nov 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

codecov Bot commented Nov 9, 2023 •

edited

Loading

DaanHoogland left a comment •

edited

Loading

DaanHoogland left a comment •

edited

Loading