Containerized iSCSI connections not propagated to the host

Bug #1810338 reported by Alan Bishop
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Medium
Alan Bishop

Bug Description

Several openstack services (nova-compute, glance-api, cinder-backup, etc.) create iSCSI connections to a storage backend provided by the cinder-volume service. These connections update information maintained in /var/lib/iscsi, and that data isn't propagated to the host when the openstack service is running in a container.

This can lead to problems during system shut down, when the host needs to log off all iSCSI connections (this is done by the host's iscsi-shutdown systemd service). However, connections created by containerized openstack services aren't "visible" to the host, and the connection's aren't terminated. This causes the shutdown sequence to hang.

The problem can be reproduced using the following steps:

1. Create a nova instance and attach it to a cinder volume via iSCSI
2. Stop the instance ("openstack server stop") but do not delete the instance
3. Reboot the Compute node, and observe it hang on shutdown

After step 1, you can see the nova_compute container's iSCSI node info:

[root@overcloud-compute-0 ~]# docker exec -ti nova_compute ls /var/lib/iscsi/nodes
iqn.2010-10.org.openstack:volume-440caa2b-1bfe-4a13-a0c4-0d86a096d524
[root@overcloud-compute-0 ~]#

But the node is not visible on the host:

[root@overcloud-compute-0 ~]# ls /var/lib/iscsi/nodes
[root@overcloud-compute-0 ~]#

Changed in tripleo:
milestone: none → stein-2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/628218

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/628218
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=243cb34615a1482ba676ad91afa10b3c76d800d6
Submitter: Zuul
Branch: master

commit 243cb34615a1482ba676ad91afa10b3c76d800d6
Author: Alan Bishop <email address hidden>
Date: Thu Jan 3 16:29:21 2019 +0000

    Bind mount /var/lib/iscsi in containers using iSCSI

    Services that create iSCSI connections need to share the connection info
    that gets created in /var/lib/iscsi. It's especially important that the
    host has knowledge of *all* connections so that it can disconnect them
    whenever the host shuts down or reboots.

    Closes-Bug: #1810338
    Change-Id: I803ed2ba9ff52f9a02c550a28d21cc9102568c8e

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/629132
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a15509f999a27482c731dc501ccfa6f2e647a227
Submitter: Zuul
Branch: master

commit a15509f999a27482c731dc501ccfa6f2e647a227
Author: Cédric Jeanneret <email address hidden>
Date: Tue Jan 8 11:14:41 2019 +0100

    Ensure /var/lib/iscsi actually exists before mounting it

    Change I803ed2ba9ff52f9a02c550a28d21cc9102568c8e adds this directory
    as bind-mount for many services, but there is no guarantee this directory
    actually exists on the node.

    This might break the deploy, as it will prevent the iscsi container to
    start as expected when using podman.

    Although this directory is managed by a package (iscsi-initiator-utils),
    this one isn't always present (i.e. on undercloud, or deployed servers,
    or standalone).

    Related-Bug: #1810338
    Change-Id: I8fc52b2a872fd77b342a0f20e4602b21d9e33fed

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.3.0

This issue was fixed in the openstack/tripleo-heat-templates 10.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/630321

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.openstack.org/630321
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f37b5e0622f95e5b49f101c01e7a8d4d5b1bf052
Submitter: Zuul
Branch: stable/rocky

commit f37b5e0622f95e5b49f101c01e7a8d4d5b1bf052
Author: Alan Bishop <email address hidden>
Date: Thu Jan 3 16:29:21 2019 +0000

    Bind mount /var/lib/iscsi in containers using iSCSI

    Services that create iSCSI connections need to share the connection info
    that gets created in /var/lib/iscsi. It's especially important that the
    host has knowledge of *all* connections so that it can disconnect them
    whenever the host shuts down or reboots.

    Note: Minor changes were included in the backport to stable/rocky.
    - The ":z" propagation suffix added to the volume mounts in stein were
      omitted.
    - In stein the glance-api service templates were moved to the unified
      deployment/glance-api directory, so the rocky changes are made in the
      original docker/services directory.

    Closes-Bug: #1810338
    Change-Id: I803ed2ba9ff52f9a02c550a28d21cc9102568c8e
    (cherry picked from commit 243cb34615a1482ba676ad91afa10b3c76d800d6)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/631072

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.openstack.org/631072
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=51fc8a67448464478a14a9b2eeb63860a98d6da8
Submitter: Zuul
Branch: stable/queens

commit 51fc8a67448464478a14a9b2eeb63860a98d6da8
Author: Alan Bishop <email address hidden>
Date: Thu Jan 3 16:29:21 2019 +0000

    Bind mount /var/lib/iscsi in containers using iSCSI

    Services that create iSCSI connections need to share the connection info
    that gets created in /var/lib/iscsi. It's especially important that the
    host has knowledge of *all* connections so that it can disconnect them
    whenever the host shuts down or reboots.

    Note: Minor changes were included in the backport to stable/rocky.
    - The ":z" propagation suffix added to the volume mounts in stein were
      omitted.
    - In stein the glance-api service templates were moved to the unified
      deployment/glance-api directory, so the rocky changes are made in the
      original docker/services directory.

    Closes-Bug: #1810338
    Change-Id: I803ed2ba9ff52f9a02c550a28d21cc9102568c8e
    (cherry picked from commit 243cb34615a1482ba676ad91afa10b3c76d800d6)
    (cherry picked from commit f37b5e0622f95e5b49f101c01e7a8d4d5b1bf052)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 8.3.0

This issue was fixed in the openstack/tripleo-heat-templates 8.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 9.3.0

This issue was fixed in the openstack/tripleo-heat-templates 9.3.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.