after upgrade from Ocata to Pike with containers /var/lib/docker not present on undercloud node

[Artem Hrechanychenko]

after upgrade from Ocata to Pike with containers /var/lib/docker not present on undercloud node:

ls: cannot access /var/lib/docker/containers: No such file or directory

On clean Pike with containers installation this directory are present:
[stack@undercloud ~]$ sudo ls -la /var/lib/docker/
total 28
drwx--x--x. 10 root root 120 Jun 9 19:02 .
drwxr-xr-x. 75 root root 4096 Jun 9 19:02 ..
drwx------. 2 root root 6 Jun 9 19:02 containers
drwx------. 3 root root 22 Jun 9 19:02 image
drwxr-x---. 3 root root 19 Jun 9 19:02 network
drwx------. 241 root root 20480 Jun 9 19:32 overlay2
drwx------. 2 root root 6 Jun 9 19:02 swarm
drwx------. 2 root root 6 Jun 9 19:32 tmp
drwx------. 2 root root 6 Jun 9 19:02 trust
drwx------. 2 root root 25 Jun 9 19:02 volumes

Steps to reproduce
==================
Deploy overcloud Ocata

openstack overcloud deploy \
    --libvirt-type qemu \
    --ntp-server clock.redhat.com \
    --control-scale 1 \
    --control-flavor oooq_control \
    --compute-flavor oooq_compute \
    --templates $THT \
    -e $THT/environments/low-memory-usage.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
-e /home/stack/network-environment.yaml

Upgrade Undercloud node

### UPGRADE UNDERCLOUD ###

    # master repos
    cd /home/stack/ && source stackrc

    sudo curl -L -o /etc/yum.repos.d/delorean.repo https://trunk.rdoproject.org/centos7-master/current-passed-ci/delorean.repo
    sudo curl -L -o /etc/yum.repos.d/delorean-current.repo https://trunk.rdoproject.org/centos7/current/delorean.repo
    sudo sed -i 's/\[delorean\]/\[delorean-current\]/' /etc/yum.repos.d/delorean-current.repo
    sudo /bin/bash -c 'printf "\nincludepkgs=diskimage-builder,instack,instack-undercloud,os-apply-config,os-collect-config,os-net-config,os-refresh-config,python-tripleoclient,openstack-tripleo-common*,openstack-tripleo-heat-templates,openstack-tripleo-image-elements,openstack-tripleo,openstack-tripleo-puppet-elements,openstack-puppet-modules,openstack-tripleo-ui,puppet-*" >> /etc/yum.repos.d/delorean-current.repo'
    sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo https://trunk.rdoproject.org/centos7/delorean-deps.repo

    sudo systemctl stop openstack-*
    sudo systemctl stop neutron-*
    sudo systemctl stop httpd

    sudo yum -y update instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient
openstack undercloud upgrade

 # create an envrionment file to make overcloud fetch the images from the undercloud
    # (192.168.24.1 is undercloud IP that must be pingable from the overcloud)
    echo > ~/containers-default-parameters.yaml 'parameter_defaults:
      DockerNamespace: 192.168.24.1:8787/tripleoupstream
      DockerNamespaceIsRegistry: true
    '
    sleep 3
    printf "\n"

    cat > ~/containers-upgrade-repos.yaml <<'EOEF'
parameter_defaults:
  UpgradeInitCommand: |
    set -ex
    pushd /etc/yum.repos.d/
    rm -rf delorean*
    REPO_PREFIX=/etc/yum.repos.d
    DELOREAN_REPO_URL=https://trunk.rdoproject.org/centos7/current-tripleo
    DELOREAN_REPO_FILE=delorean.repo
    sudo curl -Lvo $REPO_PREFIX/delorean-deps.repo https://trunk.rdoproject.org/centos7/delorean-deps.repo
    sudo sed -i -e 's%priority=.*%priority=30%' $REPO_PREFIX/delorean-deps.repo
    cat $REPO_PREFIX/delorean-deps.repo
    # Enable last known good RDO Trunk Delorean repository
    sudo curl -Lvo $REPO_PREFIX/delorean.repo $DELOREAN_REPO_URL/$DELOREAN_REPO_FILE
    sudo sed -i -e 's%priority=.*%priority=20%' $REPO_PREFIX/delorean.repo
    cat $REPO_PREFIX/delorean.repo
    # Enable latest RDO Trunk Delorean repository
    sudo curl -Lvo $REPO_PREFIX/delorean-current.repo https://trunk.rdoproject.org/centos7/current/delorean.repo
    sudo sed -i -e 's%priority=.*%priority=10%' $REPO_PREFIX/delorean-current.repo
    sudo sed -i 's/\[delorean\]/\[delorean-current\]/' $REPO_PREFIX/delorean-current.repo
    sudo /bin/bash -c "cat <<-EOF>>$REPO_PREFIX/delorean-current.repo
    includepkgs=diskimage-builder,instack,instack-undercloud,os-apply-config,os-collect-config,os-net-config,os-refresh-config,python-tripleoclient,openstack-tripleo-common*,openstack-tripleo-heat-templates,openstack-tripleo-image-elements,openstack-tripleo,openstack-tripleo-puppet-elements,openstack-puppet-modules,openstack-tripleo-ui,puppet-*
    EOF"
    cat $REPO_PREFIX/delorean-current.repo
    popd
    yum clean all
EOEF

download_docker_images_to_local_registry
    cd /home/stack/ && source stackrc
    sudo openstack overcloud container image upload --verbose --config-file /usr/share/tripleo-common/container-images/overcloud_containers.yaml

Upgrade overcloud
 cd /home/stack/ && source stackrc
    export THT=/usr/share/openstack-tripleo-heat-templates
    openstack overcloud deploy \
    --libvirt-type qemu \
    --ntp-server clock.redhat.com \
    --control-scale 1 \
    --control-flavor oooq_control \
    --compute-flavor oooq_compute \
    --templates $THT \
    -e $THT/environments/low-memory-usage.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
    -e /home/stack/network-environment.yaml \
    -e $THT/environments/docker.yaml \
    -e $THT/environments/major-upgrade-composable-steps-docker.yaml \
    -e ~/containers-default-parameters.yaml \
-e ~/containers-upgrade-repos.yaml

Check docker containers state on overcloud nodes

Expected result
===============
directory /var/lib/docker are present

Actual result
ls: cannot access /var/lib/docker/containers: No such file or directory

[David Peacock (davidjpeacock)]

It's suspected that this is no longer relevant with Ocata being end of life.