apt-transport-https not getting installed correctly during upgrades on new containers

[Jacob Wagner]

When running run-upgrade.sh, the new kilo containers (repo, nova_console, etc) get placed in the inventory file, then the custom play to ensure apt-transport-https is run before bringing these new containers up. This causes the repo-containers plays to fail.

TASK: [memcached_server | Ensure the memcache user exists] ********************
changed: [605021-infra02_repo_container-1af46214]
changed: [605023-infra03_repo_container-dd1ce825]
changed: [605020-infra01_repo_container-fda60b69]

TASK: [memcached_server | Update apt sources] *********************************
failed: [605023-infra03_repo_container-dd1ce825] => {"attempts": 5, "failed": true}
msg: Task failed as maximum retries was encountered
failed: [605020-infra01_repo_container-fda60b69] => {"attempts": 5, "failed": true}
msg: Task failed as maximum retries was encountered
failed: [605021-infra02_repo_container-1af46214] => {"attempts": 5, "failed": true}
msg: Task failed as maximum retries was encountered

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/root/repo-server.retry

605020-infra01_repo_container-fda60b69 : ok=9 changed=6 unreachable=0 failed=1
605021-infra02_repo_container-1af46214 : ok=9 changed=6 unreachable=0 failed=1
605023-infra03_repo_container-dd1ce825 : ok=9 changed=6 unreachable=0 failed=1

root@605020-infra01:/opt/rpc-openstack/os-ansible-deployment/playbooks# lxc-attach -
ansible.cfg inventory/ lxc-hosts-setup.yml os-cinder-install.yml os-keystone-install.yml os-tempest-install.yml repo-clone-mirror.yml roles/ setup-infrastructure.yml
defaults/ library/ memcached-install.yml os-glance-install.yml os-neutron-install.yml plugins/ repo-install.yml rsyslog-install.yml setup-openstack.yml
galera-install.yml lxc-containers-create.yml openstack-hosts-setup.yml os-heat-install.yml os-nova-install.yml rabbitmq-install.yml repo-server.yml setup-everything.yml utility-install.yml
haproxy-install.yml lxc-containers-destroy.yml os-ceilometer-install.yml os-horizon-install.yml os-swift-install.yml repo-build.yml repo-store-source.yml setup-hosts.yml vars/
root@605020-infra01:/opt/rpc-openstack/os-ansible-deployment/playbooks# lxc-attach -n 605020-infra01_repo_container-fda60b69
root@605020-infra01_repo_container-fda60b69:/# cd
root@605020-infra01_repo_container-fda60b69:~# apt-get update
E: The method driver /usr/lib/apt/methods/https could not be found.
N: Is the package apt-transport-https installed?
E: The method driver /usr/lib/apt/methods/https could not be found.
N: Is the package apt-transport-https installed?

[Nolan Brubaker (nolan-brubaker)]

We believe we had a patch in progress. Both Ian Cordasco and I are testing this in parallel.

This bug leaves deployments in a broken, unusable state after run-upgrade.sh is run, hence it's been marked as critical, and I targeted it to 11.1.0.

If 11.1.0 ships without it (which may happen), deployers will have to wait for another revision to be able to safely upgrade OSAD itself.

[Nolan Brubaker (nolan-brubaker)]

The patch we are testing can be found here; it's not on Gerrit since initially we weren't entirely sure where the fix had to go.

https://gist.github.com/sigmavirus24/de3d0079ee92c15f2035

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/207687
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=560fbbdb077c0f8d6f8bfa9b48e967ccef86664a
Submitter: Jenkins
Branch: kilo

commit 560fbbdb077c0f8d6f8bfa9b48e967ccef86664a
Author: Ian Cordasco <email address hidden>
Date: Thu Jul 30 21:03:23 2015 -0500

    Properly parse and quote arguments in upgrade script

    This patch fixes the following:

    1. Properly quote arguments to run_lock function
    2. Properly parse out the playbook filename in run_lock

    Specifically the upgrade steps where we were using

       "-e 'rabbitmq_upgrade=true' setup-infrastructure.yml"
       "/tmp/fix_container_interfaces.yml || true"

    Were causing issues and this patch resolves them.

    Closes-bug: 1479916
    Change-Id: I809085d6da493f7f7d545547a0d984c0e7b1bf45

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/208546
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=ef3e74311380253652256fc630816d249f0ede91
Submitter: Jenkins
Branch: master

commit ef3e74311380253652256fc630816d249f0ede91
Author: Ian Cordasco <email address hidden>
Date: Thu Jul 30 21:03:23 2015 -0500

    Properly parse and quote arguments in upgrade script

    This patch fixes the following:

    1. Properly quote arguments to run_lock function
    2. Properly parse out the playbook filename in run_lock

    Specifically the upgrade steps where we were using

       "-e 'rabbitmq_upgrade=true' setup-infrastructure.yml"
       "/tmp/fix_container_interfaces.yml || true"

    Were causing issues and this patch resolves them.

    Closes-bug: 1479916
    Change-Id: I809085d6da493f7f7d545547a0d984c0e7b1bf45
    (cherry picked from commit 560fbbdb077c0f8d6f8bfa9b48e967ccef86664a)

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.14 release.