Skip to content
This repository was archived by the owner on Apr 8, 2025. It is now read-only.

[ISSUE-1051] Handle Kubelet's Wrong CSI Call Inconsistent with Real Volume Status for OBS Release 1.3 - #1053

Open
CraneShiEMC wants to merge 18 commits into
rel/objectscale-1.3from
bugfix-handle-kubelet-wrong-call-obs-1.3
Open

[ISSUE-1051] Handle Kubelet's Wrong CSI Call Inconsistent with Real Volume Status for OBS Release 1.3#1053
CraneShiEMC wants to merge 18 commits into
rel/objectscale-1.3from
bugfix-handle-kubelet-wrong-call-obs-1.3

Conversation

@CraneShiEMC

@CraneShiEMC CraneShiEMC commented Aug 27, 2023 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Purpose

Resolves #1051

  1. Handle Kubelet's Wrong CSI Call Inconsistent with Real Volume Status
  2. Proceed Kubelet's CSI call also on Failed Volume

PR checklist

  • Add link to the issue
  • Choose Project
  • Choose PR label
  • New unit tests added
  • Modified code has meaningful comments
  • All TODOs are linked with the issues
  • All comments are resolved

Testing

I've similuated the scenario of volume's k8s global device mountpoint missing in my standalone test. This CSI defensive enhancement can work well as expected in the test.

custom ci passed: https://asd-ecs-jenkins.isus.emc.com/job/csi-custom-ci/1562/

custom-acceptance passed:
Atlantic (rke2): https://asd-ecs-jenkins.isus.emc.com/job/csi-custom-acceptance-tar_b_ona/39/
Openshift: https://asd-ecs-jenkins.isus.emc.com/job/csi-custom-acceptance-oil_bd/255/

yimingwangdell and others added 17 commits July 18, 2023 14:10
@yimingwangdell
[ISSUE-1038] 1.3.0 branched out. Change version to 1.4.0 for new rele…
b5ffc6c 
…ase. (#1039)

Signed-off-by: yimingwangdell <yiming.wang@dell.com>
@yimingwangdell
[ISSUE-1040] fix pip3 install PyYAML failed. (#1041)
57095d3 
Signed-off-by: yimingwangdell <yiming.wang@dell.com>
@CraneShiEMC
[ISSUE-1018] Refine Storage Group Feature (#1031)
8085a0a 
* add StorageGroupStatus

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* support StorageGroupStatus in current workflows

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine log

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine error handling

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* trigger storage group resync if applicable in drive removal

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* A drive whose Usage is REMOVED will not be selected in any storage group and its existing sg label takes no effect

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* sg feature will not apply to drive physically removed

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* handle the drive removal case of drive with manual sg label

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix go lint error

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add UT case for drive-removal-triggered sg sync

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* improve UT coverage

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine sg annotation for drive removal

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* handle case of invalid sg for drive removal

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* also exclude removing sg for trigger sg resync in drive removal

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine sg removal status handling

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* Revert "refine error handling"

This reverts commit 06607e7.

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine log and some code logic

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* try to add immutability validation rule to storagegroup spec

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* upgrade controller-gen version to v0.9.2

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add storagegroupcontroller UT initial

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* Revert "add storagegroupcontroller UT initial"

This reverts commit 1ea8660.

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refactor and add UT of storagegroupcontroller

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix storagegroupcontroller UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine the logic of sg deletion

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix bug

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix go-lint err

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix go-lint error

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add drive IsClean support, decrease k8s api call, remove manual sg labeling support

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine corner case handling

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine and add UT to storagegroupcontroller

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine storagegroupcontroller and add UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* make controller svc's k8scache also sync sg and lvg objs'

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* use k8s cache, re-support sg label manual change and refine in sg ctrl

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix lint err

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* storagegroup controller will not reconcile on drive delete event

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* not support Health, Status, Usage and IsClean as DriveSelector's MatchFields

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* in storagegroupcontroller's reconcile, only sync drive when reqName is uuid

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine the logic to avoid nil pointer error

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* revert the usage of k8scache

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add custom storage group proposal draft

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine custom storage group proposal

Signed-off-by: Shi, Crane <crane.shi@emc.com>

---------

Signed-off-by: Shi, Crane <crane.shi@emc.com>
@CraneShiEMC
[ISSUE-1045] Fake Attach Feature for Non-LVM-Block-Mode Volumes (#1046)
4c3fd07 
* fix pr validation startup failure

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* directly use generated file for fake attach block mode

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* still use loopback device wrap

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine code for creating fake device

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix go lint error

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add mock func implementation

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix go lint

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine func

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* support fake attach block mode with fake device, add removeLoopDevice,

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix go lint

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* support clean fake device in fake-attach block-mode

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* support non-existing current fake device case

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* change fake device dir on host

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine log

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* support the case of get fake device info failure

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* clean fake device also in removal of fake-attach block-mode vol

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* if fake-device ann is invalid, re-create the fake device and update ann

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* enhance

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add comment

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* check loop device err shouldn't block subsequent op; clean fake device should also check loop device first

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* update fake-attach doc accordingly

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix typo

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine doc

Signed-off-by: Shi, Crane <crane.shi@emc.com>

---------

Signed-off-by: Shi, Crane <crane.shi@emc.com>
Shi, Crane
for NodeUnstage/NodePublish/NodeUnpublish, also need to retry on Fail…
5f071d6 
…ed volume

Signed-off-by: Shi, Crane <crane.shi@emc.com>
Shi, Crane
fix lint error
2e6ab7f 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
Shi, Crane
fix UT
fdc5970 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
Shi, Crane
add UT
a50a7f1 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
Shi, Crane
add UT
bfb1d98 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
Shi, Crane
Redo NodeStageVolume if srcPath is not mounted
1a77d43 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
Shi, Crane
correct stagingTargetPath
d36e785 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
Shi, Crane
add warning log for redo NodeStageVolume
4376b10 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
Shi, Crane
fix that, in multi-pod case, successful unpublish volume cannot resum…
c835b96 
…e failed volume status

Signed-off-by: Shi, Crane <crane.shi@emc.com>
Shi, Crane
unlock volume to redo NodeStageVolume if applicable in NodePublishVolume
a3abfb9 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
Shi, Crane
fix issue of outdated volume
af81d9a 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
Shi, Crane
Revert "[ISSUE-1038] 1.3.0 branched out. Change version to 1.4.0 for …
aee7d03 
…new release. (#1039)"

This reverts commit b5ffc6c.

Signed-off-by: Shi, Crane <crane.shi@emc.com>
Shi, Crane
Merge branch 'rel/objectscale-1.3' of https://github.com/dell/csi-bar…
e65eb59 
…emetal into bugfix-handle-kubelet-wrong-call-obs-1.3

Signed-off-by: Shi, Crane <crane.shi@emc.com>
libzhang
libzhang approved these changes Aug 27, 2023
View reviewed changes

@libzhang libzhang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please fix UT failure in PR validation before merge.

@CraneShiEMC

CraneShiEMC commented Aug 27, 2023

Copy link
Copy Markdown
Collaborator Author

please fix UT failure in PR validation before merge.

OK

Shi, Crane
fix and add UT
11d8f7e 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@yanxiucai yanxiucai Awaiting requested review from yanxiucai
@yimingwangdell yimingwangdell Awaiting requested review from yimingwangdell
@aiqiEMC aiqiEMC Awaiting requested review from aiqiEMC
2 more reviewers
@libzhang libzhang libzhang approved these changes
@lordjun lordjun lordjun approved these changes
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@CraneShiEMC @libzhang @lordjun @yimingwangdell