*****In example, on a Standard External Storage configuration, we got the following “Events” from `kubectl describe node compute-0`***** Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Starting 60m kubelet, compute-0 Starting kubelet. Normal NodeHasSufficientMemory 60m (x2 over 60m) kubelet, compute-0 Node compute-0 status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 60m (x2 over 60m) kubelet, compute-0 Node compute-0 status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 60m (x2 over 60m) kubelet, compute-0 Node compute-0 status is now: NodeHasSufficientPID Normal NodeAllocatableEnforced 60m kubelet, compute-0 Updated Node Allocatable limit across pods —>>> Normal Starting 59m kube-proxy, compute-0 Starting kube-proxy. <<<—- Normal NodeReady 59m kubelet, compute-0 Node compute-0 status is now: NodeReady ***** The event reports that it has been on that state from 60mins. After 5-10 more minutes (around 75minues from the start of the event), describing the controller-0 shows that it’s ready and pod running***** wrsroot@controller-0 ~(keystone_admin)]$ kubectl describe node compute-0 Name: compute-0 Roles: Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/hostname=compute-0 openstack-compute-node=enabled openvswitch=enabled remote-storage=enabled sriov=enabled Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: 0 projectcalico.org/IPv4Address: 192.168.206.8/24 projectcalico.org/IPv4IPIPTunnelAddr: 172.16.154.0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Thu, 04 Apr 2019 15:02:41 +0000 Taints: Unschedulable: false Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- NetworkUnavailable False Thu, 04 Apr 2019 15:02:58 +0000 Thu, 04 Apr 2019 15:02:58 +0000 CalicoIsUp Calico is running on this node MemoryPressure False Thu, 04 Apr 2019 16:03:57 +0000 Thu, 04 Apr 2019 15:02:41 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Thu, 04 Apr 2019 16:03:57 +0000 Thu, 04 Apr 2019 15:02:41 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Thu, 04 Apr 2019 16:03:57 +0000 Thu, 04 Apr 2019 15:02:41 +0000 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Thu, 04 Apr 2019 16:03:57 +0000 Thu, 04 Apr 2019 15:02:57 +0000 KubeletReady kubelet is posting ready status Addresses: InternalIP: 10.10.52.112 Hostname: compute-0 Capacity: cpu: 112 ephemeral-storage: 20027216Ki memory: 196618880Ki pods: 110 Allocatable: cpu: 112 ephemeral-storage: 18457082236 memory: 196516480Ki pods: 110 System Info: Machine ID: 6a9bcc3dd7dd490fb8a06a23c43dd4f7 System UUID: 80297BA4-FA80-E811-906E-00163566263E Boot ID: 9f1fb60a-d551-4077-9e3b-54b918f3425e Kernel Version: 3.10.0-957.1.3.el7.1.tis.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: docker://18.6.3 Kubelet Version: v1.13.5 Kube-Proxy Version: v1.13.5 PodCIDR: 172.16.2.0/24 Non-terminated Pods: (13 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- kube-system calico-node-k7krc 250m (0%) 0 (0%) 0 (0%) 0 (0%) 61m kube-system kube-proxy-47jrc 0 (0%) 0 (0%) 0 (0%) 0 (0%) 61m openstack ceilometer-compute-t4wmj 100m (0%) 2 (1%) 128Mi (0%) 1Gi (0%) 28m openstack libvirt-rx5w9 0 (0%) 0 (0%) 0 (0%) 0 (0%) 41m openstack neutron-dhcp-agent-compute-0-75ea0372-bnzgp 0 (0%) 0 (0%) 0 (0%) 0 (0%) 39m openstack neutron-l3-agent-compute-0-31b0f4b3-zd7rn 0 (0%) 0 (0%) 0 (0%) 0 (0%) 39m openstack neutron-metadata-agent-compute-0-75ea0372-gkhhs 0 (0%) 0 (0%) 0 (0%) 0 (0%) 39m openstack neutron-ovs-agent-compute-0-75ea0372-8crjl 0 (0%) 0 (0%) 0 (0%) 0 (0%) 39m openstack neutron-sriov-agent-compute-0-75ea0372-q4qm6 0 (0%) 0 (0%) 0 (0%) 0 (0%) 39m openstack nova-compute-compute-0-75ea0372-hfwg8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 41m openstack openvswitch-db-rc8nv 0 (0%) 0 (0%) 0 (0%) 0 (0%) 41m openstack openvswitch-vswitchd-6bl7r 0 (0%) 0 (0%) 0 (0%) 0 (0%) 41m openstack osh-openstack-garbd-garbd-749c85d565-sthhc 0 (0%) 0 (0%) 0 (0%) 0 (0%) 47m Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 350m (0%) 2 (1%) memory 128Mi (0%) 1Gi (0%) ephemeral-storage 0 (0%) 0 (0%) Events: *****However, even after this particular event is cleared, we still got errors when trying to create severs***** controller-0:~$ openstack server create --flavor m1.small --image centos --nic net-id=f615488d-98f7-4423-890f-60bfdeaaa26c vm2 +-------------------------------------+-------------------------------------------------+ | Field | Value | +-------------------------------------+-------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | None | | OS-EXT-SRV-ATTR:hypervisor_hostname | None | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-STS:power_state | NOSTATE | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | None | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | | | adminPass | esbXNn525JMs | | config_drive | | | created | 2019-04-04T16:27:32Z | | flavor | m1.small (eb46ac6c-b10b-401d-a067-a5ae55ebc320) | | hostId | | | id | 41d660b4-cb4f-405f-b9dc-e390abef3e42 | | image | centos (499659bf-6e2b-43ee-ab3c-e8301197a468) | | key_name | None | | name | vm2 | | progress | 0 | | project_id | c4ef2cefa6e64f92a4d9a939994ccc7b | | properties | | | security_groups | name='default' | | status | BUILD | | updated | 2019-04-04T16:27:32Z | | user_id | f3193f0770aa431c8d99b8c35309bff5 | | volumes_attached | | +-------------------------------------+-------------------------------------------------+ controller-0:~$ openstack server list +--------------------------------------+------+--------+----------+--------+----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+------+--------+----------+--------+----------+ | 41d660b4-cb4f-405f-b9dc-e390abef3e42 | vm2 | ERROR | | centos | m1.small | | a7d3bf34-fd54-434a-901e-b076b74aeb7b | vm1 | ERROR | | cirros | m1.tiny | +--------------------------------------+------+--------+----------+--------+----------+ controller-0:~$ openstack server show vm1 | fault | {u'message': u'No valid host was found. ', u'code': 500, u'details': u' File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 1348, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 800, in _schedule_instances\n return_alternates=return_alternates)\n File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call\n retry=self.retry)\n File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/transport.py", line 128, in _send\n retry=retry)\n File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send\n call_monitor_timeout, retry=retry)\n File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send\n raise result\n', u'created': u'2019-04-04T16:21:57Z'} | Checking the logs for these pod/container, we found the following errors: controller-0:~$ kubectl logs nova-compute-compute-0-75ea0372-zsnnb -n openstack nova-compute 2019-04-04 16:22:30,860.860 168396 ERROR nova.compute.manager [req-b3bffaba-a62a-4e56-be63-3362c51a36df - - - - -] No compute node record for host compute-0: ComputeHostNotFound_Remote: Compute host compute-0 could not be found. 2019-04-04 16:22:30,902.902 168396 ERROR nova.compute.manager [req-b3bffaba-a62a-4e56-be63-3362c51a36df - - - - -] Error updating resources for node compute-0.: PciDeviceNotFoundById: PCI device 0000:b3:02.3 not found