OpenShift service deployment faling with "error on port vxlan0: []"

[Jiří Stránský]

openshift-ansible run passes fine without errors, but OpenShift Origin Node is nevertheless failing with:

Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: E0104 10:05:15.681710 764887 kubelet.go:1170] Image garbage collection failed: unable to find data for container /
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: I0104 10:05:15.687717 764887 server.go:298] Adding debug handlers to kubelet server.
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: I0104 10:05:15.690373 764887 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: I0104 10:05:15.690400 764887 status_manager.go:140] Starting to sync pod status with apiserver
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: I0104 10:05:15.690410 764887 kubelet.go:1749] Starting kubelet main sync loop.
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: I0104 10:05:15.690424 764887 kubelet.go:1760] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: I0104 10:05:15.690618 764887 container_manager_linux.go:396] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: I0104 10:05:15.690692 764887 volume_manager.go:247] The desired_state_of_world populator starts
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: I0104 10:05:15.690696 764887 volume_manager.go:249] Starting Kubelet Volume Manager
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: W0104 10:05:15.708616 764887 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: E0104 10:05:15.708748 764887 kubelet.go:2075] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: I0104 10:05:15.716481 764887 factory.go:342] Registering Docker factory
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: W0104 10:05:15.716506 764887 manager.go:247] Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: I0104 10:05:15.716511 764887 factory.go:54] Registering systemd factory
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: I0104 10:05:15.716744 764887 factory.go:86] Registering Raw factory
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: I0104 10:05:15.716967 764887 manager.go:1106] Started watching for new ooms in manager
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: I0104 10:05:15.720746 764887 oomparser.go:185] oomparser using systemd
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: I0104 10:05:15.723814 764887 manager.go:288] Starting recovery of all containers
Jan 04 10:05:15 overcloud-controller-0 origin-node[764864]: F0104 10:05:15.760139 764887 node.go:309] error: SDN node startup failed: error on port vxlan0: []

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/532827

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/532828

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/532829

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/532827
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e2c18c34ec13aed5936a4d6eabc6aced38b84781
Submitter: Zuul
Branch: master

commit e2c18c34ec13aed5936a4d6eabc6aced38b84781
Author: Jiri Stransky <email address hidden>
Date: Thu Jan 11 14:40:43 2018 +0100

    OpenShift: Don't pre-install Docker

    We installed docker our traditional way when deploying with Kubespray
    and disabled Kubespray's management of Docker, because Kubespray
    installs non-CentOS Docker binaries.

    However, openshift-ansible installs Docker from CentOS, we don't need
    to install it using the Docker composable service too. That way
    openshift-ansible will be the authority on Docker configuration when
    deploying OpenShift.

    Change-Id: I1352d4050e2f38300068d858b19e0b4a31cf50a7
    Related-Bug: #1741224

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/532828
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=254d1dee4b201c89523c8216317a3c54ca27a2e4
Submitter: Zuul
Branch: master

commit 254d1dee4b201c89523c8216317a3c54ca27a2e4
Author: Jiri Stransky <email address hidden>
Date: Thu Jan 11 14:48:12 2018 +0100

    OpenShift: Accept generic global parameters

    This will allow arbitrary config of global variables for
    openshift-ansible, e.g. customizing SDN params according to:

    https://docs.openshift.org/3.6/install_config/configuring_sdn.html

    Also remove the setting which was meant to disable OVS service
    handlers in openshift-ansible -- that wouldn't solve the problem
    fully.

    Change-Id: Ib87e5d38797da166826af90659e3d05da3352dcf
    Related-Bug: #1741224

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/532829
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5ebcd23f0d606196d68687224ab3379e924e8a65
Submitter: Zuul
Branch: master

commit 5ebcd23f0d606196d68687224ab3379e924e8a65
Author: Jiri Stransky <email address hidden>
Date: Thu Jan 11 14:50:09 2018 +0100

    OpenShift: Properly disable bare metal OVS

    We're deploying containerized OpenShift, which means openshift-ansible
    deploys also containerized OVS. When not disabled explicitly, the bare
    metal OVS service seemed to persist at least partially, and it likely
    caused issues with the containerized OVS, where nodes in `kubectl get
    nodes` would go from Ready status to NotReady shortly after the
    deployment finished.

    Change-Id: I8952198be7f78a699cf363af2e10f26714e94850
    Closes-Bug: #1741224

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0rc1 release candidate.