kilo->liberty upgrades fail due to missing repo containers

[Darren Birkett]

In this change:

https://review.openstack.org/#/c/328485/

a step was added to the upgrade script that destroys the repo containers. Later in the script, they get rebuilt, but there is a task in between that tries to contact the repo servers (clean up mariadb repos), and this fails fatally:

12:19:13 TASK: [Remove MariaDB repositories left over from Juno] ***********************
12:19:13 changed: [jrpcaioiad-81d]
12:19:14 changed: [jrpcaioiad-81d_elasticsearch_container-ef33a62e]
12:19:14 changed: [jrpcaioiad-81d_nova_scheduler_container-e1dd6ae1]
12:19:14 changed: [jrpcaioiad-81d_horizon_container-2b72374a]
12:19:14 changed: [jrpcaioiad-81d_horizon_container-45109a74]
12:19:14 changed: [jrpcaioiad-81d_utility_container-8c0e348d]
12:19:16 changed: [jrpcaioiad-81d_keystone_container-a65f3207]
12:19:16 changed: [jrpcaioiad-81d_keystone_container-7526b45d]
12:19:16 changed: [jrpcaioiad-81d_nova_cert_container-b755f2f3]
12:19:16 changed: [jrpcaioiad-81d_cinder_scheduler_container-6a434007]
12:19:16 changed: [jrpcaioiad-81d_logstash_container-71b9e26e]
12:19:17 failed: [jrpcaioiad-81d_swift_proxy_container-a4359d97] => {"changed": true, "cmd": "sed -i '/http:.*maria.*/d' /etc/apt/sources.list.d/*", "delta": "0:00:00.010600", "end": "2016-07-14 12:19:16.236612", "rc": 2, "start": "2016-07-14 12:19:16.226012", "warnings": ["Consider using template or lineinfile module rather than running sed"]}
12:19:17 stderr: sed: can't read /etc/apt/sources.list.d/*: No such file or directory
12:19:17 ...ignoring
12:19:17 changed: [jrpcaioiad-81d_neutron_server_container-6d5d6bab]
12:19:17 changed: [jrpcaioiad-81d_glance_container-9b5ce985]
12:19:18 changed: [jrpcaioiad-81d_neutron_agents_container-144ca684]
12:19:18 changed: [jrpcaioiad-81d_nova_api_os_compute_container-1c94f342]
12:19:18 changed: [jrpcaioiad-81d_nova_api_metadata_container-50ecf431]
12:19:19 failed: [jrpcaioiad-81d_kibana_container-1feee4d4] => {"changed": true, "cmd": "sed -i '/http:.*maria.*/d' /etc/apt/sources.list.d/*", "delta": "0:00:00.010808", "end": "2016-07-14 12:19:18.577255", "rc": 2, "start": "2016-07-14 12:19:18.566447", "warnings": ["Consider using template or lineinfile module rather than running sed"]}
12:19:19 stderr: sed: can't read /etc/apt/sources.list.d/*: No such file or directory
12:19:19 ...ignoring
12:19:19 changed: [jrpcaioiad-81d_nova_console_container-09ad63cd]
12:19:19 changed: [jrpcaioiad-81d_nova_conductor_container-757806dc]
12:19:20 changed: [jrpcaioiad-81d_galera_container-041dcc35]
12:19:20 changed: [jrpcaioiad-81d_cinder_api_container-8e97d53e]
12:19:20 changed: [jrpcaioiad-81d_galera_container-fa4f4c37]
12:19:21 changed: [jrpcaioiad-81d_galera_container-36fc1014]
12:19:21 failed: [jrpcaioiad-81d_rsyslog_container-7b87b4bd] => {"changed": true, "cmd": "sed -i '/http:.*maria.*/d' /etc/apt/sources.list.d/*", "delta": "0:00:00.006978", "end": "2016-07-14 12:19:20.963399", "rc": 2, "start": "2016-07-14 12:19:20.956421", "warnings": ["Consider using template or lineinfile module rather than running sed"]}
12:19:21 stderr: sed: can't read /etc/apt/sources.list.d/*: No such file or directory
12:19:21 ...ignoring
12:19:21 failed: [jrpcaioiad-81d_rabbit_mq_container-7bf33d21] => {"changed": true, "cmd": "sed -i '/http:.*maria.*/d' /etc/apt/sources.list.d/*", "delta": "0:00:00.006868", "end": "2016-07-14 12:19:20.987048", "rc": 2, "start": "2016-07-14 12:19:20.980180", "warnings": ["Consider using template or lineinfile module rather than running sed"]}
12:19:21 stderr: sed: can't read /etc/apt/sources.list.d/*: No such file or directory
12:19:21 ...ignoring
12:19:23 failed: [jrpcaioiad-81d_rabbit_mq_container-d94fdef5] => {"changed": true, "cmd": "sed -i '/http:.*maria.*/d' /etc/apt/sources.list.d/*", "delta": "0:00:00.007515", "end": "2016-07-14 12:19:22.122361", "rc": 2, "start": "2016-07-14 12:19:22.114846", "warnings": ["Consider using template or lineinfile module rather than running sed"]}
12:19:23 stderr: sed: can't read /etc/apt/sources.list.d/*: No such file or directory
12:19:23 ...ignoring
12:19:23 failed: [jrpcaioiad-81d_rabbit_mq_container-aaaf9a68] => {"changed": true, "cmd": "sed -i '/http:.*maria.*/d' /etc/apt/sources.list.d/*", "delta": "0:00:00.006434", "end": "2016-07-14 12:19:22.138389", "rc": 2, "start": "2016-07-14 12:19:22.131955", "warnings": ["Consider using template or lineinfile module rather than running sed"]}
12:19:23 stderr: sed: can't read /etc/apt/sources.list.d/*: No such file or directory
12:19:23 ...ignoring
12:19:23 failed: [jrpcaioiad-81d_memcached_container-0a3e1eda] => {"changed": true, "cmd": "sed -i '/http:.*maria.*/d' /etc/apt/sources.list.d/*", "delta": "0:00:00.004646", "end": "2016-07-14 12:19:22.152193", "rc": 2, "start": "2016-07-14 12:19:22.147547", "warnings": ["Consider using template or lineinfile module rather than running sed"]}
12:19:23 stderr: sed: can't read /etc/apt/sources.list.d/*: No such file or directory
12:19:23 ...ignoring
12:19:24 changed: [jrpcaioiad-81d_heat_apis_container-637cf283]
12:19:24 changed: [jrpcaioiad-81d_heat_engine_container-f3a10d39]
12:21:36 fatal: [jrpcaioiad-81d_repo_container-2d7f005c] => SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh
12:27:27 fatal: [jrpcaioiad-81d_repo_container-6fa4e0ef] => SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh
12:27:27
12:27:27 PLAY RECAP ********************************************************************
12:27:27 Remove MariaDB repositories left over from Juno ----------------------- 493.92s
12:27:27 to retry, use: --limit @/jenkins/mariadb-apt-cleanup.retry

We either need to recreate those containers again sooner, or limit the cleanup task (and any other tasks in between the destroy/create steps) to skip the repo containers.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/342183
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=6992c9d8f89f8582d04e2493a183336ba87d70ad
Submitter: Jenkins
Branch: liberty

commit 6992c9d8f89f8582d04e2493a183336ba87d70ad
Author: git-harry <email address hidden>
Date: Thu Jul 14 14:39:53 2016 +0100

    Fix upgrade failure due to missing repo containers

    Commit 368b7e2b0ef5abf792373f232087d0a4e595a5b9 has introduced a bug
    causing the upgrade script to fail when running the playbook
    mariadb-apt-cleanup.yml. The failure happens because the repo containers
    no longer exist and so the playbook fails to connect to those
    containers.

    This commit modifies the hosts targeted by mariadb-apt-cleanup.yml to
    remove the repo containers.

    Closes-Bug: 1603071
    Change-Id: I9c314528e7868376b5fe249efa904b7ee6d53144

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 12.1.0 release.