Virtual-kubelet skipping sync pod

[lmq1999]

I have install VK and run it, but it keep skipping pod like this

root@controller:~# virtual-kubelet 
ERRO[0000] TLS certificates not provided, not setting up pod http server  certPath= keyPath= node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=
INFO[0000] Initialized                                   node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=
INFO[0000] Pod cache in-sync                             node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=
INFO[0000] starting workers                              node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=
INFO[0000] started workers                               node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=
INFO[0000] Updated pod in provider                       key=kube-system/kube-proxy-zv842 method=createOrUpdatePod name=kube-proxy-zv842 namespace=kube-system node=virtual-kubelet operatingSystem=Linux phase=Pending pod=kube-proxy-zv842 provider=openstack reason= uid=20cafefa-3f14-45d3-98dc-43cf31a3d94a watchedNamespace= workerId=0
WARN[0005] skipping sync of pod "kube-system/kube-proxy-zv842" in "Failed" phase  key=kube-system/kube-proxy-zv842 method=syncPodInProvider name=kube-proxy-zv842 namespace=kube-system node=virtual-kubelet operatingSystem=Linux phase=Failed provider=openstack reason= uid=20cafefa-3f14-45d3-98dc-43cf31a3d94a watchedNamespace= workerId=9
INFO[0007] Created pod in provider                       key=kube-system/kube-proxy-zb9xz method=createOrUpdatePod name=kube-proxy-zb9xz namespace=kube-system node=virtual-kubelet operatingSystem=Linux phase=Pending pod=kube-proxy-zb9xz provider=openstack reason= uid=2146469d-5c85-4e20-b2b9-c58dc00b1926 watchedNamespace= workerId=2
INFO[0010] Updated pod in provider                       key=kube-system/kube-proxy-zb9xz method=createOrUpdatePod name=kube-proxy-zb9xz namespace=kube-system node=virtual-kubelet operatingSystem=Linux phase=Pending pod=kube-proxy-zb9xz provider=openstack reason= uid=2146469d-5c85-4e20-b2b9-c58dc00b1926 watchedNamespace= workerId=3
WARN[0020] skipping sync of pod "kube-system/kube-proxy-zb9xz" in "Failed" phase  key=kube-system/kube-proxy-zb9xz method=syncPodInProvider name=kube-proxy-zb9xz namespace=kube-system node=virtual-kubelet operatingSystem=Linux phase=Failed provider=openstack reason= uid=2146469d-5c85-4e20-b2b9-c58dc00b1926 watchedNamespace= workerId=6
INFO[0024] Created pod in provider                       key=kube-system/kube-proxy-k6tvn method=createOrUpdatePod name=kube-proxy-k6tvn namespace=kube-system node=virtual-kubelet operatingSystem=Linux phase=Pending pod=kube-proxy-k6tvn provider=openstack reason= uid=2e46dde1-289f-4497-9764-aaa6e8ad084e watchedNamespace= workerId=8
INFO[0025] Updated pod in provider                       key=kube-system/kube-proxy-k6tvn method=createOrUpdatePod name=kube-proxy-k6tvn namespace=kube-system node=virtual-kubelet operatingSystem=Linux phase=Pending pod=kube-proxy-k6tvn provider=openstack reason= uid=2e46dde1-289f-4497-9764-aaa6e8ad084e watchedNamespace= workerId=0
WARN[0035] skipping sync of pod "kube-system/kube-proxy-k6tvn" in "Failed" phase  key=kube-system/kube-proxy-k6tvn method=syncPodInProvider name=kube-proxy-k6tvn namespace=kube-system node=virtual-kubelet operatingSystem=Linux phase=Failed provider=openstack reason= uid=2e46dde1-289f-4497-9764-aaa6e8ad084e watchedNamespace= workerId=9
INFO[0039] Created pod in provider                       key=kube-system/kube-proxy-lszzc method=createOrUpdatePod name=kube-proxy-lszzc namespace=kube-system node=virtual-kubelet operatingSystem=Linux phase=Pending pod=kube-proxy-lszzc provider=openstack reason= uid=5025454f-a95c-4e71-82ec-d385fb3a8354 watchedNamespace= workerId=2
INFO[0040] Updated pod in provider                       key=kube-system/kube-proxy-lszzc method=createOrUpdatePod name=kube-proxy-lszzc namespace=kube-system node=virtual-kubelet operatingSystem=Linux phase=Pending pod=kube-proxy-lszzc provider=openstack reason= uid=5025454f-a95c-4e71-82ec-d385fb3a8354 watchedNamespace= workerId=3

I install k8s master on openstack controller node:

root@controller:~# kubectl get nodes
NAME              STATUS   ROLES    AGE     VERSION
controller        Ready    master   16m     v1.18.6
virtual-kubelet   Ready    agent    3m59s   v1.14.3

So how to deal with this problems ?

[hongbin]

@lmq1999 the failed pod is kube-proxy pod created by k8s. Since we are using virtual node so we don't need the kube-proxy process. Therefore, the failed pod is not important. However, let me know if you create a normal pod and failed.

[lmq1999]

Hmmmm I meet problems too with normal pod either

I try 2 pods

1st is in the README.md of this repo

2nd is the kubia.yaml from kubernetes in action (luska) https://github.com/luksa/kubernetes-in-action

Normaly they would schedule to k8s-worker node but in here they stuck in creating

NAMESPACE     NAME                                 READY   STATUS     RESTARTS   AGE
default       kubia-9j5wn                          0/1     Pending    0          93s
default       myapp-pod                            0/1     Creating   0          5m6s

myapp-pod already creating for 5m

debug mode:

+0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2020-07-27 03:58:01 +0000 UTC 2020-07-27 03:50:30 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {NetworkUnavailable False 2020-07-27 03:58:01 +0000 UTC 2020-07-27 03:50:30 +0000 UTC R000 UTC KubeletHasSufficientDisk kubelet has sufficient disk space available} {MemoryPressure False 2020-07-27 03:58:21 +0000 UTC 2020-07-27 0[0/2055]+0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2020-07-27 03:58:21 +0000 UTC 2020-07-27 03:50:30 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {NetworkUnavailable False 2020-07-27 03:58:21 +0000 UTC 2020-07-27 03:50:30 +0000 UTC R
outeCreated RouteController created a route}]" node.UID=0979c831-9910-4a7a-8658-4b7ea979aa84 node.cluster= node.name=virtual-kubelet node.resourceVersion=1926 node.taints="virtual-kubelet.io/provider=openstack:NoSchedule" operatingSystem=Linux provider=openstack watchedNamespace=                    
DEBU[0471] Successful node ping                          node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=              
DEBU[0471] Skipping pod status update                    method=syncProviderWrapper.syncPodStatuses nPods=2 node=virtual-kubelet operatingSystem=Linux
 provider=openstack watchedNamespace=                                                                                                   
DEBU[0471] Skipping pod status update                    method=syncProviderWrapper.syncPodStatuses nPods=2 node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=
DEBU[0471] Pod status update loop start                  node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=              
DEBU[0476] Skipping pod status update                    method=syncProviderWrapper.syncPodStatuses nPods=2 node=virtual-kubelet operatingSystem=Linux
 provider=openstack watchedNamespace=                                                                                                   
DEBU[0476] Skipping pod status update                    method=syncProviderWrapper.syncPodStatuses nPods=2 node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=
DEBU[0476] Pod status update loop start                  node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=              
DEBU[0480] Got queue object                              method=handleQueueItem node=virtual-kubelet operatingSystem=Linux provider=openstack watchedN
amespace= workerId=0                                                                                                                    
DEBU[0480] sync handled                                  key=default/myapp-pod method=syncHandler node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace= workerId=0
DEBU[0480] Got queue object                              method=handleQueueItem node=virtual-kubelet operatingSystem=Linux provider=openstack watchedN
amespace= workerId=4                                                                                                                                  
DEBU[0480] sync handled                                  key=kube-system/kube-proxy-4fzx9 method=syncHandler node=virtual-kubelet operatingSystem=Linu
x provider=openstack watchedNamespace= workerId=4                                                                                                     
DEBU[0480] Processed queue item                          method=handleQueueItem node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace= workerId=0                                                                                                                                  
DEBU[0480] Processed queue item                          method=handleQueueItem node=virtual-kubelet operatingSystem=Linux provider=openstack watchedN
amespace= workerId=4                                                                                                                    
DEBU[0481] got node from api server                      method=UpdateNodeStatus node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=                           
DEBU[0481] updated node status in api server             method=UpdateNodeStatus node=virtual-kubelet node.Status.Conditions="[{Ready True 2020-07-27 03:58:31 +0000 UTC 2020-07-27 03:50:30 +0000 UTC KubeletReady kubelet is ready.} {OutOfDisk False 2020-07-27 03:58:31 +0000 UTC 2020-07-27 03:50:30 +0
000 UTC KubeletHasSufficientDisk kubelet has sufficient disk space available} {MemoryPressure False 2020-07-27 03:58:31 +0000 UTC 2020-07-27 03:50:30
+0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2020-07-27 03:58:31 +0000 UTC 2020-07-27 03:50:30 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {NetworkUnavailable False 2020-07-27 03:58:31 +0000 UTC 2020-07-27 03:50:30 +0000 UTC R
outeCreated RouteController created a route}]" node.UID=0979c831-9910-4a7a-8658-4b7ea979aa84 node.cluster= node.name=virtual-kubelet node.resourceVersion=1949 node.taints="virtual-kubelet.io/provider=openstack:NoSchedule" operatingSystem=Linux provider=openstack watchedNamespace=
DEBU[0481] Successful node ping                          node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=
DEBU[0481] Skipping pod status update                    method=syncProviderWrapper.syncPodStatuses nPods=2 node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=
DEBU[0481] Skipping pod status update                    method=syncProviderWrapper.syncPodStatuses nPods=2 node=virtual-kubelet operatingSystem=Linux
 provider=openstack watchedNamespace=                                                                                                                 
DEBU[0481] Pod status update loop start                  node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace= 

I create k8s-master node by using kubeadm as well

kubeadm init --apiserver-advertise-address 192.168.122.146 --pod-network-cidr=10.10.10.0/24

The 10.10.10.0/24 is the same as external network from openstack:

root@controller:~# openstack subnet list
+--------------------------------------+-------------+--------------------------------------+------------------+
| ID                                   | Name        | Network                              | Subnet           |
+--------------------------------------+-------------+--------------------------------------+------------------+
| 5c04d6e5-7f35-4331-a9e8-7a5f23f5a846 | selfservice | a75c38ec-b7ff-4d14-8236-ef68e8a90ec4 | 10.10.10.0/24    |
| e01c75b7-ba01-4622-a1a3-305a368fa2e1 | provider    | 5a97816d-c2cc-4c0c-86e2-f6f91cc65431 | 192.168.122.0/24 |
+--------------------------------------+-------------+--------------------------------------+------------------+

[hongbin]

@lmq1999 , several things to check:

• if your Zun is correctly functioning: https://docs.openstack.org/zun/latest/install/verify.html
• if your pod is created in Zun (use command "openstack capsule list")?
• if the pod is not created in Zun, check if virtual-kubelet can communicate with Zun.
• if the pod is created but failed, check if there is any error message in the logs of the Zun processes (zun-api, zun-compute).

[lmq1999]

Zun run correctly

Pod is created but fail

+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| containers      | [{'uuid': '2744d973-e666-4354-ab6b-bbc195a04331', 'name': 'capsule-default-myapp-pod-eta-13', 'project_id': '73f9f1d50afe40829c65d17cafa42353', 'user_id': '157a1532cf2e4cdd8ce73a6982153ef2', 'image': 'busybox', 'cpu': None, 'cpu_policy': 'shared', 'memory': None, 'command': ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600'], 'status': 'Creating', 'status_reason': None, 'task_state': None, 'environment': {'KUBERNETES_PORT': 'tcp://10.96.0.1:443', 'KUBERNETES_PORT_443_TCP': 'tcp://10.96.0.1:443', 'KUBERNETES_PORT_443_TCP_ADDR': '10.96.0.1', 'KUBERNETES_PORT_443_TCP_PORT': '443', 'KUBERNETES_PORT_443_TCP_PROTO': 'tcp', 'KUBERNETES_SERVICE_HOST': '10.96.0.1', 'KUBERNETES_SERVICE_PORT': '443', 'KUBERNETES_SERVICE_PORT_HTTPS': '443'}, 'workdir': None, 'auto_remove': False, 'ports': [], 'hostname': None, 'labels': {}, 'addresses': {}, 'restart_policy': {'MaximumRetryCount': '0', 'Name': 'always'}, 'status_detail': None, 'interactive': False, 'tty': False, 'image_driver': None, 'security_groups': [], 'disk': 0, 'auto_heal': False, 'healthcheck': {}, 'registry_id': None, 'entrypoint': []}] |
| init_containers | []                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| uuid            | 0eeda473-16c4-4eb5-9ad8-376bb8673264                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| links           | [{'href': 'http://controller:9517/v1/capsules/0eeda473-16c4-4eb5-9ad8-376bb8673264', 'rel': 'self'}, {'href': 'http://controller:9517/capsules/0eeda473-16c4-4eb5-9ad8-376bb8673264', 'rel': 'bookmark'}]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| name            | default-myapp-pod                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| project_id      | 73f9f1d50afe40829c65d17cafa42353                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| user_id         | 157a1532cf2e4cdd8ce73a6982153ef2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| cpu             | 0.0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| memory          | 0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| status          | Error                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| status_reason   | No 'zun.vif_translators' driver found, looking for 'bridge'                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| labels          | {'ClusterName': '', 'CreationTimestamp': '2020-07-27 07:03:01 +0000 UTC', 'Namespace': 'default', 'NodeName': 'virtual-kubelet', 'PodName': 'myapp-pod', 'UID': '6e8c4abc-e862-4d36-9f7f-3e672fc1eec5'}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| addresses       | 10.10.10.137                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| restart_policy  | {'MaximumRetryCount': '0', 'Name': 'always'}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| annotations     | None                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| created_at      | 2020-07-27 07:04:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| updated_at      | 2020-07-27 07:04:19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| networks        | a75c38ec-b7ff-4d14-8236-ef68e8a90ec4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

zun-compute logs:

2020-07-27 07:13:20.723 6394 WARNING stevedore.named [req-aed2264b-16e7-4da3-8e0c-7dba5d2fa23f - - - - -] Could not load bridge
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager [req-aed2264b-16e7-4da3-8e0c-7dba5d2fa23f - - - - -] Unexpected exception: No 'zun.vif_translators' driver found, looking for 'bridge': stevedore.exception.NoMatches: No 'zun.vif_translators' driver found, looking for 'bridge'
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager Traceback (most recent call last):
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/zun/network/os_vif_util.py", line 236, in neutron_to_osvif_vif
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager     mgr = _VIF_MANAGERS[vif_translator]
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager KeyError: 'bridge'
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager 
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager During handling of the above exception, another exception occurred:
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager 
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager Traceback (most recent call last):
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/zun/compute/manager.py", line 368, in _do_container_create_base
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager     requested_volumes)
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/zun/container/cri/driver.py", line 52, in create_capsule
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager     self._create_pod_sandbox(context, capsule, requested_networks)
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/zun/container/cri/driver.py", line 79, in _create_pod_sandbox
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager     self._write_cni_metadata(context, capsule, requested_networks)
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/zun/container/cri/driver.py", line 116, in _write_cni_metadata
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager     subnets)
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/zun/network/os_vif_util.py", line 240, in neutron_to_osvif_vif
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager     name=vif_translator, invoke_on_load=False)
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager   File "/usr/lib/python3/dist-packages/stevedore/driver.py", line 61, in __init__
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager     warn_on_missing_entrypoint=warn_on_missing_entrypoint
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager   File "/usr/lib/python3/dist-packages/stevedore/named.py", line 89, in __init__
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager     self._init_plugins(extensions)
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager   File "/usr/lib/python3/dist-packages/stevedore/driver.py", line 113, in _init_plugins
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager     (self.namespace, name))
2020-07-27 07:13:20.724 6394 ERROR zun.compute.manager stevedore.exception.NoMatches: No 'zun.vif_translators' driver found, looking for 'bridge'

zun-api logs:

2020-07-27 07:13:12.551 13724 INFO zun.api.controllers.v1.capsules [req-d43ac4a7-5636-42e0-a72a-fafd9dd796b5 - - - - -] Policy doesn't support image_pull_policy
2020-07-27 07:13:14.061 13724 INFO eventlet.wsgi.server [req-d43ac4a7-5636-42e0-a72a-fafd9dd796b5 - - - - -] 10.0.0.125 "POST /v1/capsules HTTP/1.1" status: 202  len: 2420 time: 1.7096784
2020-07-27 07:13:16.124 13724 INFO eventlet.wsgi.server [req-0ac19f24-50fe-4eac-a551-b2fdcbda71a0 - - - - -] 10.0.0.125 "GET /v1/capsules/default-myapp-pod HTTP/1.1" status: 200  len: 2339 time: 0.0457959
2020-07-27 07:13:16.189 13724 INFO eventlet.wsgi.server [req-d95c1ad9-759c-48ed-9ef2-8dee7d3a23ca - - - - -] 10.0.0.125 "GET /v1/capsules/default-myapp-pod HTTP/1.1" status: 200  len: 2339 time: 0.0453143
2020-07-27 07:13:21.167 13725 INFO eventlet.wsgi.server [req-3b4fe0c1-1e26-41f5-9879-88073cd1cec8 - - - - -] 10.0.0.125 "GET /v1/capsules/default-myapp-pod HTTP/1.1" status: 200  len: 2603 time: 0.03993

[lmq1999]

I read the code in /usr/local/lib/python3.6/dist-packages/zun/network/os_vif_util.py and only see OVS, isn't it support linux bridge ?

[lmq1999]

Latest update:

After reconfig openstack from linuxbridge to ovs:

+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| containers      | [{'uuid': '7594ff37-f036-4ba0-bed4-77bf7fa90c8b', 'name': 'capsule-default-myapp-pod-pi-13', 'project_id': '73f9f1d50afe40829c65d17cafa42353', 'user_id': '157a1532cf2e4cdd8ce73a6982153ef2', 'image': 'busybox', 'cpu': None, 'cpu_policy': 'shared', 'memory': None, 'command': ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600'], 'status': 'Creating', 'status_reason': None, 'task_state': None, 'environment': {'KUBERNETES_PORT': 'tcp://10.96.0.1:443', 'KUBERNETES_PORT_443_TCP': 'tcp://10.96.0.1:443', 'KUBERNETES_PORT_443_TCP_ADDR': '10.96.0.1', 'KUBERNETES_PORT_443_TCP_PORT': '443', 'KUBERNETES_PORT_443_TCP_PROTO': 'tcp', 'KUBERNETES_SERVICE_HOST': '10.96.0.1', 'KUBERNETES_SERVICE_PORT': '443', 'KUBERNETES_SERVICE_PORT_HTTPS': '443'}, 'workdir': None, 'auto_remove': False, 'ports': [], 'hostname': None, 'labels': {}, 'addresses': {}, 'restart_policy': {'MaximumRetryCount': '0', 'Name': 'always'}, 'status_detail': None, 'interactive': False, 'tty': False, 'image_driver': None, 'security_groups': [], 'disk': 0, 'auto_heal': False, 'healthcheck': {}, 'registry_id': None, 'entrypoint': []}] |
| init_containers | []                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| uuid            | 8e0f709f-99f8-438b-be30-32f1b83e2e3a                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| links           | [{'href': 'http://controller:9517/v1/capsules/8e0f709f-99f8-438b-be30-32f1b83e2e3a', 'rel': 'self'}, {'href': 'http://controller:9517/capsules/8e0f709f-99f8-438b-be30-32f1b83e2e3a', 'rel': 'bookmark'}]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| name            | default-myapp-pod                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| project_id      | 73f9f1d50afe40829c65d17cafa42353                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| user_id         | 157a1532cf2e4cdd8ce73a6982153ef2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| cpu             | 0.0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| memory          | 0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| status          | Error                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| status_reason   | <_InactiveRpcError of RPC that terminated with:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|                 | 	status = StatusCode.UNAVAILABLE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
|                 | 	details = "failed to connect to all addresses"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|                 | 	debug_error_string = "{"created":"@1595845092.954850424","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3948,"referenced_errors":[{"created":"@1595845059.361132817","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":394,"grpc_status":14}]}"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
|                 | >                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| labels          | {'ClusterName': '', 'CreationTimestamp': '2020-07-27 10:18:02 +0000 UTC', 'Namespace': 'default', 'NodeName': 'virtual-kubelet', 'PodName': 'myapp-pod', 'UID': 'da0507bd-7411-4bf9-aa27-d51f9b46cfa1'}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| addresses       | 10.10.10.231                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| restart_policy  | {'MaximumRetryCount': '0', 'Name': 'always'}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| annotations     | None                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| created_at      | 2020-07-27 10:18:03                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| updated_at      | 2020-07-27 10:18:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| networks        | a4322dbc-4ffa-4a38-895d-16a32ee1deb9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

zun-api.log

2020-07-27 10:18:03.743 16291 INFO zun.api.controllers.v1.capsules [req-40283159-dcf3-435a-9020-e342defc1f16 - - - - -] Policy doesn't support image_pull_policy
2020-07-27 10:18:05.720 16291 INFO eventlet.wsgi.server [req-40283159-dcf3-435a-9020-e342defc1f16 - - - - -] 10.0.0.125 "POST /v1/capsules HTTP/1.1" status: 202  len: 2419 time: 2.3464501
2020-07-27 10:18:06.414 16291 INFO eventlet.wsgi.server [req-0e891c97-1d77-4c28-8f6b-939b001622c8 - - - - -] 10.0.0.125 "GET /v1/capsules/default-myapp-pod HTTP/1.1" status: 200  len: 2338 time: 0.0382912
2020-07-27 10:18:06.548 16291 INFO eventlet.wsgi.server [req-d22700a0-4bab-45d0-a2e1-61120404c979 - - - - -] 10.0.0.125 "GET /v1/capsules/default-myapp-pod HTTP/1.1" status: 200  len: 2338 time: 0.1146283
2020-07-27 10:18:11.495 16291 INFO eventlet.wsgi.server [req-90e60174-0924-4f09-9d11-5485ed708299 - - - - -] 10.0.0.125 "GET /v1/capsules/default-myapp-pod HTTP/1.1" status: 200  len: 2338 time: 0.0784547

zun-compute:

2020-07-27 10:12:05.841 4633 INFO zun.cmd.compute [-] Starting server in PID 4633
2020-07-27 10:12:05.919 4633 INFO zun.container.driver [-] Loading container driver 'docker'
2020-07-27 10:12:05.999 4633 INFO zun.volume.driver [-] Loading volume driver 'cinder'
2020-07-27 10:12:06.000 4633 INFO zun.volume.driver [-] Loading volume driver 'local'
2020-07-27 10:12:06.163 4633 INFO zun.image.driver [-] Loading container image driver 'glance'
2020-07-27 10:12:06.164 4633 INFO zun.image.driver [-] Loading container image driver 'docker'
2020-07-27 10:12:06.578 4633 INFO zun.volume.driver [-] Loading volume driver 'cinder'
2020-07-27 10:12:06.579 4633 INFO zun.volume.driver [-] Loading volume driver 'local'
2020-07-27 10:12:06.997 4633 WARNING zun.compute.compute_node_tracker [req-2b7a4024-7d42-42a1-99bc-7d20b818af1d - - - - -] No compute node record for: worker1: zun.common.exception.ComputeNodeNotFound: Compute node worker1 could not be found.
2020-07-27 10:12:07.049 4633 INFO zun.compute.compute_node_tracker [req-2b7a4024-7d42-42a1-99bc-7d20b818af1d - - - - -] Node created for :worker1
2020-07-27 10:12:50.039 4633 WARNING zun.compute.manager [req-9c52d922-6e55-4198-8d0e-5f1fd812ed7a 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Ignore the configured default disk size because the driver does not support disk quota.
2020-07-27 10:12:50.084 4633 INFO zun.compute.claims [req-9c52d922-6e55-4198-8d0e-5f1fd812ed7a 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Attempting claim: memory 512, cpu 1.00 CPU, disk 0
2020-07-27 10:12:50.085 4633 INFO zun.compute.claims [req-9c52d922-6e55-4198-8d0e-5f1fd812ed7a 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Total memory: 7976 MB, used: 0.00 MB
2020-07-27 10:12:50.085 4633 INFO zun.compute.claims [req-9c52d922-6e55-4198-8d0e-5f1fd812ed7a 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] memory limit: 7976.00 MB, free: 7976.00 MB
2020-07-27 10:12:50.086 4633 INFO zun.compute.claims [req-9c52d922-6e55-4198-8d0e-5f1fd812ed7a 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Total vcpu: 2 VCPU, used: 0.00 VCPU
2020-07-27 10:12:50.086 4633 INFO zun.compute.claims [req-9c52d922-6e55-4198-8d0e-5f1fd812ed7a 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] vcpu limit: 2.00 VCPU, free: 2.00 VCPU
2020-07-27 10:12:50.086 4633 INFO zun.compute.claims [req-9c52d922-6e55-4198-8d0e-5f1fd812ed7a 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Total disk: 17 GB, used: 0.00 GB
2020-07-27 10:12:50.086 4633 INFO zun.compute.claims [req-9c52d922-6e55-4198-8d0e-5f1fd812ed7a 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] disk limit not specified, defaulting to unlimited
2020-07-27 10:12:50.087 4633 INFO zun.compute.claims [req-9c52d922-6e55-4198-8d0e-5f1fd812ed7a 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Claim successful
2020-07-27 10:16:23.879 4633 WARNING zun.compute.manager [req-3f6e0118-f8c2-45f2-bf15-dfac6226dbc5 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Ignore the configured default disk size because the driver does not support disk quota.
2020-07-27 10:16:23.908 4633 INFO zun.compute.claims [req-3f6e0118-f8c2-45f2-bf15-dfac6226dbc5 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Attempting claim: memory 0, cpu 0.00 CPU, disk 0
2020-07-27 10:16:23.909 4633 INFO zun.compute.claims [req-3f6e0118-f8c2-45f2-bf15-dfac6226dbc5 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Total memory: 7976 MB, used: 512.00 MB
2020-07-27 10:16:23.909 4633 INFO zun.compute.claims [req-3f6e0118-f8c2-45f2-bf15-dfac6226dbc5 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] memory limit: 7976.00 MB, free: 7464.00 MB
2020-07-27 10:16:23.910 4633 INFO zun.compute.claims [req-3f6e0118-f8c2-45f2-bf15-dfac6226dbc5 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Total vcpu: 2 VCPU, used: 1.00 VCPU
2020-07-27 10:16:23.910 4633 INFO zun.compute.claims [req-3f6e0118-f8c2-45f2-bf15-dfac6226dbc5 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] vcpu limit not specified, defaulting to unlimited
2020-07-27 10:16:23.910 4633 INFO zun.compute.claims [req-3f6e0118-f8c2-45f2-bf15-dfac6226dbc5 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Total disk: 17 GB, used: 0.00 GB
2020-07-27 10:16:23.911 4633 INFO zun.compute.claims [req-3f6e0118-f8c2-45f2-bf15-dfac6226dbc5 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] disk limit not specified, defaulting to unlimited
2020-07-27 10:16:23.911 4633 INFO zun.compute.claims [req-3f6e0118-f8c2-45f2-bf15-dfac6226dbc5 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Claim successful
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager [req-d94ed33f-4e9a-4268-bddd-404de7322507 - - - - -] Unexpected exception: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1595844993.492038743","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3948,"referenced_errors":[{"created":"@1595844993.492036009","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":394,"grpc_status":14}]}"
>: grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1595844993.492038743","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3948,"referenced_errors":[{"created":"@1595844993.492036009","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":394,"grpc_status":14}]}"
>
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager Traceback (most recent call last):
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/zun/compute/manager.py", line 368, in _do_container_create_base
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager     requested_volumes)
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/zun/container/cri/driver.py", line 52, in create_capsule
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager     self._create_pod_sandbox(context, capsule, requested_networks)
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/zun/container/cri/driver.py", line 83, in _create_pod_sandbox
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager     runtime_handler=runtime,
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 826, in __call__
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager     return _end_unary_response_blocking(state, call, False, None)
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager     raise _InactiveRpcError(state)
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager 	status = StatusCode.UNAVAILABLE
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager 	details = "failed to connect to all addresses"
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager 	debug_error_string = "{"created":"@1595844993.492038743","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3948,"referenced_errors":[{"created":"@1595844993.492036009","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":394,"grpc_status":14}]}"
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager >
2020-07-27 10:16:33.492 4633 ERROR zun.compute.manager 
2020-07-27 10:18:05.723 4633 WARNING zun.compute.manager [req-261a1b41-629c-46e2-92fc-8d57702b0247 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Ignore the configured default disk size because the driver does not support disk quota.
2020-07-27 10:18:05.744 4633 INFO zun.compute.claims [req-261a1b41-629c-46e2-92fc-8d57702b0247 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Attempting claim: memory 0, cpu 0.00 CPU, disk 0
2020-07-27 10:18:05.745 4633 INFO zun.compute.claims [req-261a1b41-629c-46e2-92fc-8d57702b0247 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Total memory: 7976 MB, used: 512.00 MB
2020-07-27 10:18:05.745 4633 INFO zun.compute.claims [req-261a1b41-629c-46e2-92fc-8d57702b0247 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] memory limit: 7976.00 MB, free: 7464.00 MB
2020-07-27 10:18:05.746 4633 INFO zun.compute.claims [req-261a1b41-629c-46e2-92fc-8d57702b0247 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Total vcpu: 2 VCPU, used: 1.00 VCPU
2020-07-27 10:18:05.746 4633 INFO zun.compute.claims [req-261a1b41-629c-46e2-92fc-8d57702b0247 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] vcpu limit not specified, defaulting to unlimited
2020-07-27 10:18:05.746 4633 INFO zun.compute.claims [req-261a1b41-629c-46e2-92fc-8d57702b0247 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Total disk: 17 GB, used: 0.00 GB
2020-07-27 10:18:05.747 4633 INFO zun.compute.claims [req-261a1b41-629c-46e2-92fc-8d57702b0247 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] disk limit not specified, defaulting to unlimited
2020-07-27 10:18:05.747 4633 INFO zun.compute.claims [req-261a1b41-629c-46e2-92fc-8d57702b0247 157a1532cf2e4cdd8ce73a6982153ef2 73f9f1d50afe40829c65d17cafa42353 default - -] Claim successful
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager [req-16c30d4e-17af-4c17-b51f-efdfa3b6b3d6 - - - - -] Unexpected exception: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1595845092.954850424","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3948,"referenced_errors":[{"created":"@1595845059.361132817","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":394,"grpc_status":14}]}"
>: grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1595845092.954850424","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3948,"referenced_errors":[{"created":"@1595845059.361132817","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":394,"grpc_status":14}]}"
>
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager Traceback (most recent call last):
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/zun/compute/manager.py", line 368, in _do_container_create_base
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager     requested_volumes)
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/zun/container/cri/driver.py", line 52, in create_capsule
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager     self._create_pod_sandbox(context, capsule, requested_networks)
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/zun/container/cri/driver.py", line 83, in _create_pod_sandbox
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager     runtime_handler=runtime,
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 826, in __call__
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager     return _end_unary_response_blocking(state, call, False, None)
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager     raise _InactiveRpcError(state)
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager 	status = StatusCode.UNAVAILABLE
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager 	details = "failed to connect to all addresses"
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager 	debug_error_string = "{"created":"@1595845092.954850424","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3948,"referenced_errors":[{"created":"@1595845059.361132817","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":394,"grpc_status":14}]}"
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager >
2020-07-27 10:18:12.955 4633 ERROR zun.compute.manager 

[hongbin]

@lmq1999,

For the linux bridge support, I will add support to it.

For the "failed to connect to all addresses" error, it seems your zun-cni-daemon is not installed correctly. Could you double check (systemctl status zun-cni-daemon)?

[lmq1999]

everything seem file except that the VIF in log show is it linux-bridge :/

zun-cni status:

root@worker1:~# service zun-cni-daemon status
● zun-cni-daemon.service - OpenStack Container Service CNI daemon
   Loaded: loaded (/etc/systemd/system/zun-cni-daemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-07-28 04:00:58 UTC; 11min ago
 Main PID: 1088 (zun-cni-daemon:)
    Tasks: 13 (limit: 4915)
   CGroup: /system.slice/zun-cni-daemon.service
           ├─1088 zun-cni-daemon: master process [/usr/local/bin/zun-cni-daemon]
           ├─3488 zun-cni-daemon: master process [/usr/local/bin/zun-cni-daemon]
           ├─3497 zun-cni-daemon: watcher worker(0)
           └─3500 zun-cni-daemon: server worker(0)

Jul 28 04:00:58 worker1 systemd[1]: Started OpenStack Container Service CNI daemon.
Jul 28 04:04:32 worker1 zun-cni-daemon[1088]:  * Serving Flask app "zun-cni-daemon" (lazy loading)
Jul 28 04:04:32 worker1 zun-cni-daemon[1088]:  * Environment: production
Jul 28 04:04:32 worker1 zun-cni-daemon[1088]:    WARNING: This is a development server. Do not use it in a production deployment.
Jul 28 04:04:32 worker1 zun-cni-daemon[1088]:    Use a production WSGI server instead.
Jul 28 04:04:32 worker1 zun-cni-daemon[1088]:  * Debug mode: off

zun-cni log:

root@worker1:~# tail -f /var/log/zun/zun-cni-daemon.log
2020-07-27 10:04:17.184 20605 INFO cotyledon._service [req-d51bab0b-3ed0-46d2-aae1-6c794a0982e3 - - - - -] Caught SIGTERM signal, graceful exiting of service server(0) [20605]
2020-07-27 10:04:17.189 20573 INFO cotyledon._service_manager [-] Caught SIGTERM signal, graceful exiting of master process
2020-07-27 10:04:17.190 20602 INFO cotyledon._service [req-9b9cc37c-8c69-4d1f-939d-8caa44a8ee92 - - - - -] Caught SIGTERM signal, graceful exiting of service watcher(0) [20602]
2020-07-27 10:05:42.622 1054 INFO os_vif [-] Loaded VIF plugins: linux_bridge, noop, ovs
2020-07-27 10:05:42.797 2811 INFO werkzeug [-]  * Running on http://127.0.0.1:9036/ (Press CTRL+C to quit)
2020-07-27 10:34:35.290 2811 INFO cotyledon._service [req-d3390ab8-0991-4fdb-bf8e-4f6986a1032f - - - - -] Caught SIGTERM signal, graceful exiting of service server(0) [2811]
2020-07-27 10:34:35.310 1054 INFO cotyledon._service_manager [-] Caught SIGTERM signal, graceful exiting of master process
2020-07-27 10:34:35.293 2808 INFO cotyledon._service [req-8130e990-0954-4a6b-ad92-be2854f71dcf - - - - -] Caught SIGTERM signal, graceful exiting of service watcher(0) [2808]
2020-07-28 04:04:32.349 1088 INFO os_vif [-] Loaded VIF plugins: linux_bridge, noop, ovs
2020-07-28 04:04:32.774 3500 INFO werkzeug [-]  * Running on http://127.0.0.1:9036/ (Press CTRL+C to quit)

When I enable debug-mode:

zun-cni log:

root@worker1:~# tail -f /var/log/zun/zun-cni-daemon.log
2020-07-28 04:23:06.736 18114 DEBUG futurist.periodics [-] Submitting periodic callback 'zun.cni.daemon.service.CNIDaemonWatcherService.poll_vif_status' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:642
2020-07-28 04:23:07.737 18114 DEBUG futurist.periodics [-] Submitting periodic callback 'zun.cni.daemon.service.CNIDaemonWatcherService.poll_vif_status' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:642
2020-07-28 04:23:08.739 18114 DEBUG futurist.periodics [-] Submitting periodic callback 'zun.cni.daemon.service.CNIDaemonWatcherService.poll_vif_status' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:642
2020-07-28 04:23:51.630 18114 DEBUG futurist.periodics [-] Submitting periodic callback 'zun.cni.daemon.service.CNIDaemonWatcherService.sync_capsules' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:642
2020-07-28 04:23:51.631 18114 DEBUG zun.cni.daemon.service [-] Start syncing capsule states. sync_capsules /usr/local/lib/python3.6/dist-packages/zun/cni/daemon/service.py:153

zun-compute-log:

2020-07-28 04:24:54.590 19208 DEBUG eventlet.wsgi.server [-] (19208) accepted ('10.0.0.121', 53618) server /usr/lib/python3/dist-packages/eventlet/wsgi.py:985
2020-07-28 04:24:54.601 19208 INFO eventlet.wsgi.server [req-34aab59b-c246-439f-982a-9b8134719a7d - - - - -] 10.0.0.121 "GET / HTTP/1.1" status: 200  len: 641 time: 0.0097344
2020-07-28 04:24:54.885 19208 DEBUG zun.common.policy [req-17829629-ff67-41bd-8679-0b472373e81f - - - - -] Policy check for container:get_one:image_pull_policy failed with credentials {'is_admin': False, 'user_id': '157a1532cf2e4cdd8ce73a6982153ef2', 'user_domain_id': None, 'system_scope': None, 'domain_id': 'default', 'project_id': '73f9f1d50afe40829c65d17cafa42353', 'project_domain_id': None, 'roles': ['myrole'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} authorize /usr/local/lib/python3.6/dist-packages/zun/common/policy.py:147
2020-07-28 04:24:54.886 19208 DEBUG zun.common.policy [req-17829629-ff67-41bd-8679-0b472373e81f - - - - -] Policy check for container:get_one:host failed with credentials {'is_admin': False, 'user_id': '157a1532cf2e4cdd8ce73a6982153ef2', 'user_domain_id': None, 'system_scope': None, 'domain_id': 'default', 'project_id': '73f9f1d50afe40829c65d17cafa42353', 'project_domain_id': None, 'roles': ['myrole'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} authorize /usr/local/lib/python3.6/dist-packages/zun/common/policy.py:147
2020-07-28 04:24:54.886 19208 DEBUG zun.common.policy [req-17829629-ff67-41bd-8679-0b472373e81f - - - - -] Policy check for container:get_one:runtime failed with credentials {'is_admin': False, 'user_id': '157a1532cf2e4cdd8ce73a6982153ef2', 'user_domain_id': None, 'system_scope': None, 'domain_id': 'default', 'project_id': '73f9f1d50afe40829c65d17cafa42353', 'project_domain_id': None, 'roles': ['myrole'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} authorize /usr/local/lib/python3.6/dist-packages/zun/common/policy.py:147
2020-07-28 04:24:54.887 19208 DEBUG zun.common.policy [req-17829629-ff67-41bd-8679-0b472373e81f - - - - -] Policy check for container:get_one:privileged failed with credentials {'is_admin': False, 'user_id': '157a1532cf2e4cdd8ce73a6982153ef2', 'user_domain_id': None, 'system_scope': None, 'domain_id': 'default', 'project_id': '73f9f1d50afe40829c65d17cafa42353', 'project_domain_id': None, 'roles': ['myrole'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} authorize /usr/local/lib/python3.6/dist-packages/zun/common/policy.py:147
2020-07-28 04:24:54.888 19208 DEBUG zun.common.policy [req-17829629-ff67-41bd-8679-0b472373e81f - - - - -] Policy check for capsule:get:host failed with credentials {'is_admin': False, 'user_id': '157a1532cf2e4cdd8ce73a6982153ef2', 'user_domain_id': None, 'system_scope': None, 'domain_id': 'default', 'project_id': '73f9f1d50afe40829c65d17cafa42353', 'project_domain_id': None, 'roles': ['myrole'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} authorize /usr/local/lib/python3.6/dist-packages/zun/common/policy.py:147
2020-07-28 04:24:54.889 19208 INFO eventlet.wsgi.server [req-17829629-ff67-41bd-8679-0b472373e81f - - - - -] 10.0.0.121 "GET /v1/capsules/4fa2515a-e49d-445f-a0fa-0e8fea28244d HTTP/1.1" status: 200  len: 3129 time: 0.2853196

[hongbin]

zun-cni-daemon doesn't receive any requests from containerd. Could you check the status of containerd (systemctl status containerd)?

Perhaps, containerd doesn't have the right permission to accept grpc requests. Could you check the containerd's config file as well (/etc/containerd/config.toml)?

[lmq1999]

contained services status:

   Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-07-28 04:01:23 UTC; 31min ago
     Docs: https://containerd.io
  Process: 1602 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 1720 (containerd)
    Tasks: 189
   CGroup: /system.slice/containerd.service
           ├─ 1720 /usr/bin/containerd
           ├─ 4329 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/b8a856568f73b1dbf0bb0698cc0edab732f8e1c4e13a2d2a8b59832575f3076c -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           ├─ 4391 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/03157db729d273231f46cb7f543d53420dd6a115dc17d97d3eeb4124af0653b9 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           ├─ 4401 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/fe1184f49e49d98d93fe85d7733c5da5770baeabbcc3d1ff7e28fce375969dd2 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           ├─ 4478 /pause
           ├─ 4489 /pause
           ├─ 4495 /pause
           ├─ 4528 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/21edf2d4a24e28b269953b8c89f8919192d2e2bba4fde971b173c75afaa22a7b -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           ├─ 4556 /pause
           ├─ 4714 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/0c62d1ba49fb9ea754c8339b720053a636d9db21e968320c621575ddca0e8815 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           ├─ 4738 etcd --advertise-client-urls=https://10.0.0.125:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --initial-advertise-peer-urls=https://10.0.0.125:2380 --initial-cluster=k8s-master=
           ├─ 4762 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/9d62ca5b1cf4347d4a5f2fc36f9df601e6856700eed8e400d774c44b59af41c0 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           ├─ 4787 kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/etc/
           ├─ 4812 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/64478258df3706ab6bfa5aa3e32a442b7b49d9f89ed104a2746f418dbf1c2e62 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           ├─ 4835 kube-scheduler --authentication-kubeconfig=/etc/kubernetes/scheduler.conf --authorization-kubeconfig=/etc/kubernetes/scheduler.conf --bind-address=127.0.0.1 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true
           ├─ 4871 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/f11aef97036ab74e081786493b2c969af6e574e6ba6fa3a92fca3c0da245fd18 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           ├─ 4902 kube-apiserver --advertise-address=10.0.0.125 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-caf
           ├─ 7975 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/288ece3b3c4a7f0283d1bbf3d6d04eaf81ab1622d743b196852756c0c3fb5aee -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           ├─ 7999 /pause
           ├─ 8265 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/69ae396832550fe6c61d689339c1e81fede6568d61d6093fc131c45b9cbf3768 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           ├─ 8325 /pause
           ├─ 8735 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/0c63a0f0b721595f3d52bf38c4eee4bc15f98f533b25dd4153a3f94e1c450050 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           ├─ 8836 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=k8s-master
           ├─ 9217 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/193a5e32393733fb51dc7ae7f5a0b31924b61f3acbb7b32b1dfe8578f84d8ca9 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           ├─ 9242 /opt/bin/flanneld --ip-masq --kube-subnet-mgr
           ├─ 9904 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/33676735f514c7ea05521317020e0d4a6153ae89a7fb91012a2c5a49c62a368a -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           ├─ 9948 /pause
           ├─10097 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/888a4dc06dc0c9c973730e54005ec95a2c3a16f47d689c9fcb901d44145d3a07 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           ├─10124 /pause
           ├─10327 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/913229ee11594da79db4b80a369d79805dbbaaff08613a551d56a3bc960fa246 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           ├─10413 /coredns -conf /etc/coredns/Corefile
           ├─10514 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/f134aeb3d70efcc1755b8efaae339b2f0354f49ecba2fb49ff7ed25f0c679838 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/co
           └─10541 /coredns -conf /etc/coredns/Corefile

Jul 28 04:06:21 k8s-master containerd[1720]: time="2020-07-28T04:06:21.170296081Z" level=info msg="shim reaped" id=55db7cab4c0b650e20d9dfdd6111a78fdf331879420a7de63438bef2d6307752
Jul 28 04:06:26 k8s-master containerd[1720]: time="2020-07-28T04:06:26.629739409Z" level=info msg="shim containerd-shim started" address=/containerd-shim/bfec9c282a83fb0be840e0edc96c01198ad723d90bb73d6e096dc26a94b71616.sock debug=false pid=9187
Jul 28 04:06:26 k8s-master containerd[1720]: time="2020-07-28T04:06:26.746569650Z" level=info msg="shim containerd-shim started" address=/containerd-shim/ad13efc632aa50e62c92e2fd9baa2be4c7270e083d5e14ee5e6ccdee1717a323.sock debug=false pid=9217
Jul 28 04:06:28 k8s-master containerd[1720]: time="2020-07-28T04:06:28.238123304Z" level=info msg="shim containerd-shim started" address=/containerd-shim/40ff38c3977e740f26be17caf1263899aed09e8ed68e6873e908935e5f80c934.sock debug=false pid=9344
Jul 28 04:06:28 k8s-master containerd[1720]: time="2020-07-28T04:06:28.694249680Z" level=info msg="shim reaped" id=dd0d9cf7a5bc826d02ea5d96000ece44231cfc6f345759532b6148fbb880de52
Jul 28 04:06:29 k8s-master containerd[1720]: time="2020-07-28T04:06:29.592948522Z" level=info msg="shim reaped" id=ecf75fe29006355f3b45c3cbdf692cb405026cf9587a907926b35d691b771b9e
Jul 28 04:06:37 k8s-master containerd[1720]: time="2020-07-28T04:06:37.742604617Z" level=info msg="shim containerd-shim started" address=/containerd-shim/6d1688b78e46a28ae3b303f0fc8b40b3f04f30efaa8974c919eb5896e46ad136.sock debug=false pid=9904

the config file:

#   Copyright 2018-2020 Docker Inc.

#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at

#       http://www.apache.org/licenses/LICENSE-2.0

#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

disabled_plugins = ["cri"]

#root = "/var/lib/containerd"
#state = "/run/containerd"
#subreaper = true
#oom_score = 0

#[grpc]
#  address = "/run/containerd/containerd.sock"
#  uid = 0
#  gid = 0

#[debug]
#  address = "/run/containerd/debug.sock"
#  uid = 0
#  gid = 0
#  level = "info"

well I only see this file in openstack compute node, not in k8s node

[hongbin]

Right. This file needs to be configured in compute nodes. Will it work if you configure the "gid" then restart the containerd process?

[grpc]
...
gid = ZUN_GROUP_ID

Replace ZUN_GROUP_ID with the real group ID of zun user. You can retrieve the ID by (for example):

getent group zun | cut -d: -f3

[lmq1999]

hmmm, i change like you said to gid=997

disabled_plugins = ["cri"]

#root = "/var/lib/containerd"
#state = "/run/containerd"
#subreaper = true
#oom_score = 0

[grpc]
#  address = "/run/containerd/containerd.sock"
#uid = 0
gid = 997

#[debug]
#  address = "/run/containerd/debug.sock"
#  uid = 0
#  gid = 0
#  level = "info"

root@worker1:~# getent group zun | cut -d: -f3
997

after create pod in virtual-kubelet, this new error appear:

+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| containers      | [{'uuid': 'f749866d-77bb-4fba-9ff3-0eb4bcdbd101', 'name': 'capsule-default-myapp-pod-rho-18', 'project_id': '73f9f1d50afe40829c65d17cafa42353', 'user_id': '157a1532cf2e4cdd8ce73a6982153ef2', 'image': 'busybox', 'cpu': None, 'cpu_policy': 'shared', 'memory': None, 'command': ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600'], 'status': 'Creating', 'status_reason': None, 'task_state': None, 'environment': {'KUBERNETES_PORT': 'tcp://10.96.0.1:443', 'KUBERNETES_PORT_443_TCP': 'tcp://10.96.0.1:443', 'KUBERNETES_PORT_443_TCP_ADDR': '10.96.0.1', 'KUBERNETES_PORT_443_TCP_PORT': '443', 'KUBERNETES_PORT_443_TCP_PROTO': 'tcp', 'KUBERNETES_SERVICE_HOST': '10.96.0.1', 'KUBERNETES_SERVICE_PORT': '443', 'KUBERNETES_SERVICE_PORT_HTTPS': '443'}, 'workdir': None, 'auto_remove': False, 'ports': [], 'hostname': None, 'labels': {}, 'addresses': {}, 'restart_policy': {'MaximumRetryCount': '0', 'Name': 'always'}, 'status_detail': None, 'interactive': False, 'tty': False, 'image_driver': None, 'security_groups': [], 'disk': 0, 'auto_heal': False, 'healthcheck': {}, 'registry_id': None, 'entrypoint': []}] |
| init_containers | []                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| uuid            | 071ea789-df74-4348-81bf-4140448e7187                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| links           | [{'href': 'http://controller:9517/v1/capsules/071ea789-df74-4348-81bf-4140448e7187', 'rel': 'self'}, {'href': 'http://controller:9517/capsules/071ea789-df74-4348-81bf-4140448e7187', 'rel': 'bookmark'}]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| name            | default-myapp-pod                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| project_id      | 73f9f1d50afe40829c65d17cafa42353                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| user_id         | 157a1532cf2e4cdd8ce73a6982153ef2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| cpu             | 0.0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| memory          | 0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| status          | Error                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| status_reason   | <_InactiveRpcError of RPC that terminated with:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
|                 | 	status = StatusCode.UNIMPLEMENTED                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|                 | 	details = "unknown service runtime.v1alpha2.RuntimeService"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|                 | 	debug_error_string = "{"created":"@1595911426.107134797","description":"Error received from peer unix:/run/containerd/containerd.sock","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"unknown service runtime.v1alpha2.RuntimeService","grpc_status":12}"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|                 | >                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| labels          | {'ClusterName': '', 'CreationTimestamp': '2020-07-28 04:43:38 +0000 UTC', 'Namespace': 'default', 'NodeName': 'virtual-kubelet', 'PodName': 'myapp-pod', 'UID': '52c65dbc-980c-4205-bd04-c84b3c492557'}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| addresses       | 10.10.10.97                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| restart_policy  | {'MaximumRetryCount': '0', 'Name': 'always'}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| annotations     | None                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| created_at      | 2020-07-28 04:43:38                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| updated_at      | 2020-07-28 04:43:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| networks        | a4322dbc-4ffa-4a38-895d-16a32ee1deb9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

containerd logs:

Jul 28 04:42:47 worker1 systemd[1]: Stopping containerd container runtime...
Jul 28 04:42:47 worker1 systemd[1]: Stopped containerd container runtime.
Jul 28 04:42:47 worker1 systemd[1]: Starting containerd container runtime...
Jul 28 04:42:47 worker1 systemd[1]: Started containerd container runtime.
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.124072976Z" level=info msg="starting containerd" revision=7ad184331fa3e55e52b890ea95e65ba581ae3429 version=1.2.13
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.124481607Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.124508591Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.124672575Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/cont
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.124769284Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127220693Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotte
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127250094Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapsho
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127300400Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127456631Z" level=info msg="skip loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshot
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127470019Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127481935Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/containerd/i
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127488848Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/containerd
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127577552Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127592720Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127626742Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.se
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127638201Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.servi
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127647284Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127657153Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.servic
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127667192Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.servic
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127676906Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.se
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127685417Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.ser
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127694671Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127735439Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.127765051Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.129126076Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.129155279Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.129197648Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.129209509Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.129218399Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.129226319Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.129234557Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.129244152Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.129252118Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.129261247Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.129269694Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.129298882Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.129309788Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.129318602Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.129327065Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.130184755Z" level=info msg=serving... address="/run/containerd/containerd.sock"
Jul 28 04:42:47 worker1 containerd[6239]: time="2020-07-28T04:42:47.130204661Z" level=info msg="containerd successfully booted in 0.006635s"

[lmq1999]

Lastest update:

After I use containerd config to create a : containerd config default > config.toml

And also change GID to 997 like you said:

and I got this error, seem like be back to CNI:

+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| containers      | [{'uuid': '0df03acf-785a-4c31-a925-cc59622cca88', 'name': 'capsule-default-myapp-pod-psi-9', 'project_id': '73f9f1d50afe40829c65d17cafa42353', 'user_id': '157a1532cf2e4cdd8ce73a6982153ef2', 'image': 'busybox', 'cpu': None, 'cpu_policy': 'shared', 'memory': None, 'command': ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600'], 'status': 'Creating', 'status_reason': None, 'task_state': None, 'environment': {'KUBERNETES_PORT': 'tcp://10.96.0.1:443', 'KUBERNETES_PORT_443_TCP': 'tcp://10.96.0.1:443', 'KUBERNETES_PORT_443_TCP_ADDR': '10.96.0.1', 'KUBERNETES_PORT_443_TCP_PORT': '443', 'KUBERNETES_PORT_443_TCP_PROTO': 'tcp', 'KUBERNETES_SERVICE_HOST': '10.96.0.1', 'KUBERNETES_SERVICE_PORT': '443', 'KUBERNETES_SERVICE_PORT_HTTPS': '443'}, 'workdir': None, 'auto_remove': False, 'ports': [], 'hostname': None, 'labels': {}, 'addresses': {}, 'restart_policy': {'MaximumRetryCount': '0', 'Name': 'always'}, 'status_detail': None, 'interactive': False, 'tty': False, 'image_driver': None, 'security_groups': [], 'disk': 0, 'auto_heal': False, 'healthcheck': {}, 'registry_id': None, 'entrypoint': []}] |
| init_containers | []                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| uuid            | e990ea6c-abdd-48d4-b45b-870dc1d3b6a2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| links           | [{'href': 'http://controller:9517/v1/capsules/e990ea6c-abdd-48d4-b45b-870dc1d3b6a2', 'rel': 'self'}, {'href': 'http://controller:9517/capsules/e990ea6c-abdd-48d4-b45b-870dc1d3b6a2', 'rel': 'bookmark'}]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| name            | default-myapp-pod                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| project_id      | 73f9f1d50afe40829c65d17cafa42353                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| user_id         | 157a1532cf2e4cdd8ce73a6982153ef2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| cpu             | 0.0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| memory          | 0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| status          | Error                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| status_reason   | <_InactiveRpcError of RPC that terminated with:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|                 | 	status = StatusCode.UNKNOWN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
|                 | 	details = "failed to setup network for sandbox "2b9b60ff3162aa28722e6cdd92490defed31994c820c379259d224066d9eb194": Got invalid status code from CNI daemon; Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|                 |   File "/usr/local/lib/python3.6/dist-packages/zun/cni/api.py", line 75, in run                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|                 |     vif = self._add(params)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|                 |   File "/usr/local/lib/python3.6/dist-packages/zun/cni/api.py", line 129, in _add                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|                 |     resp = self._make_request('addNetwork', params, httplib.ACCEPTED)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                 |   File "/usr/local/lib/python3.6/dist-packages/zun/cni/api.py", line 167, in _make_request                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
|                 |     raise exception.CNIError('Got invalid status code from CNI daemon')                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
|                 | zun.common.exception.CNIError: Got invalid status code from CNI daemon                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|                 | "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|                 | 	debug_error_string = "{"created":"@1595912825.832394318","description":"Error received from peer unix:/run/containerd/containerd.sock","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"failed to setup network for sandbox "2b9b60ff3162aa28722e6cdd92490defed31994c820c379259d224066d9eb194": Got invalid status code from CNI daemon; Traceback (most recent call last):\n  File "/usr/local/lib/python3.6/dist-packages/zun/cni/api.py", line 75, in run\n    vif = self._add(params)\n  File "/usr/local/lib/python3.6/dist-packages/zun/cni/api.py", line 129, in _add\n    resp = self._make_request('addNetwork', params, httplib.ACCEPTED)\n  File "/usr/local/lib/python3.6/dist-packages/zun/cni/api.py", line 167, in _make_request\n    raise exception.CNIError('Got invalid status code from CNI daemon')\nzun.common.exception.CNIError: Got invalid status code from CNI daemon\n","grpc_status":2}"                                                                                                                                                                                               |
|                 | >                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| labels          | {'ClusterName': '', 'CreationTimestamp': '2020-07-28 05:06:47 +0000 UTC', 'Namespace': 'default', 'NodeName': 'virtual-kubelet', 'PodName': 'myapp-pod', 'UID': '9df92ccf-cc1d-473e-af12-707d1e559757'}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| addresses       | 10.10.10.20                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| restart_policy  | {'MaximumRetryCount': '0', 'Name': 'always'}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| annotations     | None                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| created_at      | 2020-07-28 05:06:47                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| updated_at      | 2020-07-28 05:07:05                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| networks        | a4322dbc-4ffa-4a38-895d-16a32ee1deb9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[lmq1999]

CNI daemon logs:

2020-07-28 07:13:08.837 23254 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'zun-rootwrap', '/etc/zun/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/zun/zun
.conf', '--privsep_context', 'vif_plug_ovs.privsep.vif_plug', '--privsep_sock_path', '/tmp/tmp_pnf0hwm/privsep.sock']
2020-07-28 07:13:09.623 23254 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap                                              
2020-07-28 07:13:09.508 23272 INFO oslo.privsep.daemon [-] privsep daemon starting 
2020-07-28 07:13:09.520 23272 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0                                                     
2020-07-28 07:13:09.523 23272 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN/CAP_NET_ADMIN/none
2020-07-28 07:13:09.524 23272 INFO oslo.privsep.daemon [-] privsep daemon running as pid 23272                                                           
2020-07-28 07:13:10.154 23254 INFO os_vif [-] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:02:1a:bb,bridge_name='qbr8cd87d70-5c',has_traffic_filtering=True,id=8cd
87d70-5cb7-4397-a5a1-7478a676de91,network=Network(a4322dbc-4ffa-4a38-895d-16a32ee1deb9),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap8cd8
7d70-5c')                                                                                                   
2020-07-28 07:13:10.157 23254 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'zun-rootwrap', '/etc/zun/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/zun/zun
.conf', '--privsep_context', 'zun.common.privileged.cni', '--privsep_sock_path', '/tmp/tmpxuvn729u/privsep.sock']
2020-07-28 07:13:10.938 23254 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap                                 
2020-07-28 07:13:10.820 23382 INFO oslo.privsep.daemon [-] privsep daemon starting
2020-07-28 07:13:10.823 23382 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0                  
2020-07-28 07:13:10.826 23382 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/CAP_NET_ADMIN|CAP_SYS_$DMIN|CAP_SYS_PTRACE/none                                                                                                                                 
2020-07-28 07:13:10.826 23382 INFO oslo.privsep.daemon [-] privsep daemon running as pid 23382
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service [-] Error when processing addNetwork request. CNI Params: {'CNI_COMMAND': 'ADD', 'CNI_CONTAINERID': '995dfcab5c850280ddb01f
c407fefda4cf2116b37f130e9071df61de5eb6b5a4', 'CNI_NETNS': '/var/run/netns/cni-0e5ba6ca-74a9-0709-3973-db740eb0634e', 'CNI_ARGS': 'IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NA
ME=5973904c-be15-4c30-9df0-b0d48e8f98f2;K8S_POD_INFRA_CONTAINER_ID=995dfcab5c850280ddb01fc407fefda4cf2116b37f130e9071df61de5eb6b5a4', 'CNI_IFNAME': 'eth0', 'CNI_PATH': '/opt/cni/bin'
}: pyroute2.netlink.exceptions.NetlinkError: (1, 'Operation not permitted')
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service Traceback (most recent call last):                                                      
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service   File "/usr/local/lib/python3.6/dist-packages/zun/cni/daemon/service.py", line 66, in add
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service     vif = self.plugin.add(params)                                                     
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service   File "/usr/local/lib/python3.6/dist-packages/zun/cni/plugins/zun_cni_registry.py", line 46, in add
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service     vifs = self._do_work(params, b_base.connect)                                      
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service   File "/usr/local/lib/python3.6/dist-packages/zun/cni/plugins/zun_cni_registry.py", line 146, in _do_work
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service     container_id=params.CNI_CONTAINERID)                                             
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service   File "/usr/local/lib/python3.6/dist-packages/zun/cni/binding/base.py", line 132, in connect
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service     driver.connect(vif, ifname, netns, container_id)                                          
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service   File "/usr/local/lib/python3.6/dist-packages/zun/cni/binding/bridge.py", line 88, in connect
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service     h_br.add_port(host_ifname)                                                                
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service   File "/usr/lib/python3/dist-packages/pyroute2/ipdb/transactional.py", line 209, in __exit__
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service     self.commit()                                                                     
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service   File "/usr/lib/python3/dist-packages/pyroute2/ipdb/interfaces.py", line 1078, in commit
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service     raise error                                                                       
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service   File "/usr/lib/python3/dist-packages/pyroute2/ipdb/interfaces.py", line 769, in commit
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service     run(nl.link, 'update', index=i, master=self['index'])               
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service   File "/usr/lib/python3/dist-packages/pyroute2/ipdb/interfaces.py", line 504, in _run
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service     raise error                                            
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service   File "/usr/lib/python3/dist-packages/pyroute2/ipdb/interfaces.py", line 499, in _run                                     
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service     return cmd(*argv, **kwarg)                            
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service   File "/usr/lib/python3/dist-packages/pyroute2/iproute/linux.py", line 1332, in link
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service     msg_flags=msg_flags)
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service   File "/usr/lib/python3/dist-packages/pyroute2/netlink/nlsocket.py", line 373, in nlm_request
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service     return tuple(self._genlm_request(*argv, **kwarg))                                    
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service   File "/usr/lib/python3/dist-packages/pyroute2/netlink/nlsocket.py", line 864, in nlm_request
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service     callback=callback):                                                                                                    
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service   File "/usr/lib/python3/dist-packages/pyroute2/netlink/nlsocket.py", line 376, in get                                     
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service     return tuple(self._genlm_get(*argv, **kwarg))
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service   File "/usr/lib/python3/dist-packages/pyroute2/netlink/nlsocket.py", line 701, in get                                     
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service     raise msg['header']['error']                      
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service pyroute2.netlink.exceptions.NetlinkError: (1, 'Operation not permitted')
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service                        
2020-07-28 07:13:12.333 23254 INFO werkzeug [-] 127.0.0.1 - - [28/Jul/2020 07:13:12] "POST /addNetwork HTTP/1.1" 500 -

[lmq1999]

I also have question, my kubernetes setup using:

kubeadm init --apiserver-advertise-address 10.0.0.251 --pod-network-cidr=10.244.0.0/16

that pod-network-cidr is not belong to any external, internal network from openstack
I also use flannel for k8s, is there anything wrong with that setup when using VK or not ?