docker pull from ceph.io fails on overcloud nodes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
High
|
Unassigned |
Bug Description
Description
===========
Overcloud deploy fails with Step2_Execution ERROR with resource_type: OS::Mistral:
/var/log/
Steps to reproduce
==================
Deploy undercloud using quickstart.sh downloaded as of yesterday on a classic virthost setup.
I ran:
THT=/
openstack overcloud deploy --templates $THT\
--libvirt-type qemu --control-flavor oooq_control --compute-flavor oooq_compute --ceph-
--control-scale 3 --compute-scale 1 --ceph-
-e /home/stack/
-e $THT/environmen
--disable-
-e $THT/environmen
-e $THT/environmen
-e $THT/environmen
-e $THT/environmen
-e /home/stack/
-e $THT/environmen
-e $THT/environmen
-e $THT/environmen
--ntp-server pool.ntp.org \
-e $THT/environmen
-e /home/stack/
--no-cleanup
/home/stack/tht is freshly cloned and has no customizations.
Expected result
===============
Overcloud will deploy successfully.
Actual result
=============
Failure as noted in the description.
Environment
===========
1. master with everything fresh as of yesterday.
2. [root@overcloud
Docker version 1.13.1, build 3f45913-unsupported
3. [root@overcloud
; generated by /usr/sbin/
search localdomain
nameserver 192.168.23.1
nameserver 10.10.160.2
192.168.23.1 is the IP on br-ex on the virthost
10.10.160.2 is a RedHat nameserver, to reach it you have to transit 192.168.23.1
4. 192.168.23.1 is not reachable. Indeed there is no route to it.
[root@overcloud
10.0.0.0/24 dev vlan10 proto kernel scope link src 10.0.0.6
169.254.169.254 via 192.168.24.1 dev br-ex
172.16.0.0/24 dev vlan50 proto kernel scope link src 172.16.0.19
172.17.0.0/24 dev vlan20 proto kernel scope link src 172.17.0.19
172.18.0.0/24 dev vlan30 proto kernel scope link src 172.18.0.12
172.19.0.0/24 dev vlan40 proto kernel scope link src 172.19.0.18
172.31.0.0/24 dev docker0 proto kernel scope link src 172.31.0.1
192.168.24.0/24 dev br-ex proto kernel scope link src 192.168.24.14
5. Adding a default route as found with other deployments is not by itself sufficient:
[root@overcloud
6. However, the default route plus using a big hammer on iptables on the undercloud
[root@undercloud]# iptables --policy FORWARD ACCEPT
is sufficient, afterwards:
[root@overcloud
default via 10.0.0.1 dev vlan10
10.0.0.0/24 dev vlan10 proto kernel scope link src 10.0.0.6
169.254.169.254 via 192.168.24.1 dev br-ex
172.16.0.0/24 dev vlan50 proto kernel scope link src 172.16.0.19
172.17.0.0/24 dev vlan20 proto kernel scope link src 172.17.0.19
172.18.0.0/24 dev vlan30 proto kernel scope link src 172.18.0.12
172.19.0.0/24 dev vlan40 proto kernel scope link src 172.19.0.18
172.31.0.0/24 dev docker0 proto kernel scope link src 172.31.0.1
[root@overcloud
Trying to pull repository docker.
tag-stable-
af4b0a2388c6: Already exists
e7c4d76f7e7f: Pull complete
399a492126ac: Pull complete
2f1a9602903e: Pull complete
ebd6704c1ee1: Pull complete
90badf46365f: Pull complete
959af06d9a0d: Pull complete
7bca9530c0bf: Pull complete
Digest: sha256:
Status: Downloaded newer image for docker.
Changed in tripleo: | |
status: | New → Triaged |
importance: | Undecided → High |
milestone: | none → queens-rc1 |
Changed in tripleo: | |
milestone: | queens-rc1 → rocky-1 |
Changed in tripleo: | |
milestone: | rocky-1 → rocky-2 |
Changed in tripleo: | |
status: | Fix Committed → Fix Released |
The docker pull commands to overcloud failed because the overcloud nodes in a virthost setup lacked a default route. This lack turned out to be a temporary problem in tripleo- heat-templates, fixed I think by commit ad8446ecc0d1058 060b9734a65c19d 797255678a.