tripleo

[Train] standalone-upgrade failing due to ovn_north/south_db_init not starting_

Bug #1860112 reported by Jose Luis Franco on 2020-01-17

This bug report is a duplicate of: Bug #1853012: Standalone upgrade job is failing - On ovn upgrade. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	New	Critical	Unassigned	tripleo ussuri-2 "tripleo ussuri-2"

Bug Description

The standalone-upgrade job fails when trying to start both containers ovn_north_db_init and ovn_south_db_init:

2020-01-17 02:27:00 | "Start container ovn_north_db_init as ovn_north_db_init.",
2020-01-17 02:27:00 | "$ podman run --name ovn_north_db_init --label config_id=tripleo_step3 --label container_name=ovn_north_db_init --label managed_by=tripleo-Standalone --label config_data={\"command\": \"ovsdb-tool create /var/lib/openvswitch/ovnnb.db /usr/share/openvswitch/ovn-nb.ovsschema\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/kolla/config_files/ovn_north_db_server.json:/var/lib/kolla/config_files/config.json:ro\", \"/lib/modules:/lib/modules:ro\", \"/var/lib/openvswitch/ovn:/var/lib/openvswitch:shared,z\", \"/var/lib/openvswitch/ovn:/run/openvswitch:shared,z\", \"/var/log/containers/openvswitch:/var/log/openvswitch:z\"], \"image\": \"docker.io/tripleotrain/centos-binary-ovn-nb-db-server:f6d2fa668cbee0290e645f76e1f4e054bd9bbd70_0930f1a9\", \"detach\": false, \"net\": \"host\", \"privileged\": false} --conmon-pidfile=/var/run/ovn_north_db_init.pid --log-driver k8s-file --log-opt path=/var/log/containers/stdouts/ovn_north_db_init.log --net=host --privileged=false --user=root --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/lib/kolla/config_files/ovn_north_db_server.json:/var/lib/kolla/config_files/config.json:ro --volume=/lib/modules:/lib/modules:ro --volume=/var/lib/openvswitch/ovn:/var/lib/openvswitch:shared,z --volume=/var/lib/openvswitch/ovn:/run/openvswitch:shared,z --volume=/var/log/containers/openvswitch:/var/log/openvswitch:z --cpuset-cpus=0,1,2,3,4,5,6,7 docker.io/tripleotrain/centos-binary-ovn-nb-db-server:f6d2fa668cbee0290e645f76e1f4e054bd9bbd70_0930f1a9 ovsdb-tool create /var/lib/openvswitch/ovnnb.db /usr/share/openvswitch/ovn-nb.ovsschema",
2020-01-17 02:27:00 | "",
2020-01-17 02:27:00 | "net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory",
2020-01-17 02:27:00 | "net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5)",
2020-01-17 02:27:00 | "PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory",
2020-01-17 02:27:00 | "PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4)",
2020-01-17 02:27:00 | "2020-01-17T02:25:46Z|00001|lockfile|WARN|/var/lib/openvswitch/.ovnnb.db.~lock~: failed to lock file: Resource temporarily unavailable",
2020-01-17 02:27:00 | "ovsdb-tool: I/O error: /var/lib/openvswitch/ovnnb.db: failed to lock lockfile (Resource temporarily unavailable)",

2020-01-17 02:27:00 | "$ podman run --name ovn_south_db_init --label config_id=tripleo_step3 --label container_name=ovn_south_db_init --label managed_by=tripleo-Standalone --label config_data={\"command\": \"ovsdb-tool create /var/lib/openvswitch/ovnsb.db /usr/share/openvswitch/ovn-sb.ovsschema\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/kolla/config_files/ovn_north_db_server.json:/var/lib/kolla/config_files/config.json:ro\", \"/lib/modules:/lib/modules:ro\", \"/var/lib/openvswitch/ovn:/var/lib/openvswitch:shared,z\", \"/var/lib/openvswitch/ovn:/run/openvswitch:shared,z\", \"/var/log/containers/openvswitch:/var/log/openvswitch:z\"], \"image\": \"docker.io/tripleotrain/centos-binary-ovn-sb-db-server:f6d2fa668cbee0290e645f76e1f4e054bd9bbd70_0930f1a9\", \"detach\": false, \"net\": \"host\", \"privileged\": false} --conmon-pidfile=/var/run/ovn_south_db_init.pid --log-driver k8s-file --log-opt path=/var/log/containers/stdouts/ovn_south_db_init.log --net=host --privileged=false --user=root --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/lib/kolla/config_files/ovn_north_db_server.json:/var/lib/kolla/config_files/config.json:ro --volume=/lib/modules:/lib/modules:ro --volume=/var/lib/openvswitch/ovn:/var/lib/openvswitch:shared,z --volume=/var/lib/openvswitch/ovn:/run/openvswitch:shared,z --volume=/var/log/containers/openvswitch:/var/log/openvswitch:z --cpuset-cpus=0,1,2,3,4,5,6,7 docker.io/tripleotrain/centos-binary-ovn-sb-db-server:f6d2fa668cbee0290e645f76e1f4e054bd9bbd70_0930f1a9 ovsdb-tool create /var/lib/openvswitch/ovnsb.db /usr/share/openvswitch/ovn-sb.ovsschema",
2020-01-17 02:27:00 | "",
2020-01-17 02:27:00 | "net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory",
2020-01-17 02:27:00 | "net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5)",
2020-01-17 02:27:00 | "PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory",
2020-01-17 02:27:00 | "PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4)",
2020-01-17 02:27:00 | "2020-01-17T02:25:30Z|00001|lockfile|WARN|/var/lib/openvswitch/.ovnsb.db.~lock~: failed to lock file: Resource temporarily unavailable",
2020-01-17 02:27:00 | "ovsdb-tool: I/O error: /var/lib/openvswitch/ovnsb.db: failed to lock lockfile (Resource temporarily unavailable)",