Containerized Ironic BM (ILO) provisioning issue: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/

[Bogdan Dobrelya]

Some of the ironic_* containers in t-h-t are missing the kolla_config to recursively chown /var/lib/ironic* shared paths, and that brings data owning races across other ironic containers starting on earlier or the same steps/order.

For example, when ironic_pxe_http fails to access the /var/lib/ironic/httpboot path shared with ironic_conductor, the latter reports errors like:

DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/...

This breaks HW provisioning.

Those data races are nice to have fixed, but that seems not the root cause for the reported errors. I think OVB jobs with containerized Ironic running undercloud are used to pass for quite a while as ovb provisions VMs and not BM nodes with ILO, so the error is not fatal for VMs. But it is for BM nodes...

Related bz https://bugzilla.redhat.com/show_bug.cgi?id=1608529

[Bogdan Dobrelya (bogdando)]

Partially addressed by: https://review.openstack.org/#/c/584265/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/586210

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/586211

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/586210
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9d9daebdda3093e795b83075349b12d23f9241e8
Submitter: Zuul
Branch: master

commit 9d9daebdda3093e795b83075349b12d23f9241e8
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Jul 26 15:44:29 2018 +0300

    Fix ironic containers data owning races

    Complements I5ad4956174346f0$24945852da09f3841c3378c2b.
    Fix ironic_pxe/inspector containers missing the kolla_config sections
    to recursively chown /var/lib/ironic* shared paths. That removes data
    owning races across other ironic containers starting on earlier or the
    same steps/order.

    For example, when ironic_pxe_http fails to access the
    /var/lib/ironic/httpboot path shared with ironic_conductor, the latter
    reports errors like:

    DriverLoadError: Driver, hardware type or interface ilo-pxe could not
    be loaded. Reason: [Errno 13] Permission denied:
    '/var/lib/ironic/httpboot/...

    Fixing the dir owner allows HW provisioning to complete w/o such
    errors.

    Change-Id: I34b54077c1c610a4bed73129e9784b2fa353e944
    Closes-bug: #1783762

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/586196
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=62f5c0b41228c265f54ba6a39c915cf24870c8d9
Submitter: Zuul
Branch: stable/queens

commit 62f5c0b41228c265f54ba6a39c915cf24870c8d9
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Jul 20 10:45:34 2018 +0300

    Fix containerized ironic bind-mounts

    We obvserve errors "DriverLoadError: Driver, hardware type or interface
    ilo-pxe could not be loaded" as it uses the host level access to
    /var/lib/ironic, which is owned by root. When we have the
    host/container(s) shared access to a bind mounted dir, it needs to be
    shared instead of the default private mode.

    Additionally, fix the missing /var/lib/ironic ownership change in
    the ironix-pxe's kolla config.

    Partial-bug: #1783762
    Change-Id: I5ad4956174346f024945852da09f3841c3378c2b
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 90a7a22f157b0f50d024d730e8b97ba268bc58ad)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/586211
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=cb475320a8d7d1293edff7621984443f1da8c4ed
Submitter: Zuul
Branch: stable/queens

commit cb475320a8d7d1293edff7621984443f1da8c4ed
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Jul 26 15:44:29 2018 +0300

    Fix ironic containers data owning races

    Complements I5ad4956174346f0$24945852da09f3841c3378c2b.
    Fix ironic_pxe/inspector containers missing the kolla_config sections
    to recursively chown /var/lib/ironic* shared paths. That removes data
    owning races across other ironic containers starting on earlier or the
    same steps/order.

    For example, when ironic_pxe_http fails to access the
    /var/lib/ironic/httpboot path shared with ironic_conductor, the latter
    reports errors like:

    DriverLoadError: Driver, hardware type or interface ilo-pxe could not
    be loaded. Reason: [Errno 13] Permission denied:
    '/var/lib/ironic/httpboot/...

    Fixing the dir owner allows HW provisioning to complete w/o such
    errors.

    Change-Id: I34b54077c1c610a4bed73129e9784b2fa353e944
    Closes-bug: #1783762
    (cherry picked from commit 9d9daebdda3093e795b83075349b12d23f9241e8)

[Bogdan Dobrelya (bogdando)]

Downstream testing shows the issue has not yet been resolved

[Bogdan Dobrelya (bogdando)]

Downstream testing with the fixed backport of https://review.openstack.org/586211 shows the issue is gone

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/619515

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/619516

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/619515
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=28de50b4767a1b96113d4cf7181c6675b322b50a
Submitter: Zuul
Branch: stable/pike

commit 28de50b4767a1b96113d4cf7181c6675b322b50a
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Jul 20 10:45:34 2018 +0300

    Fix containerized ironic bind-mounts

    We obvserve errors "DriverLoadError: Driver, hardware type or interface
    ilo-pxe could not be loaded" as it uses the host level access to
    /var/lib/ironic, which is owned by root. When we have the
    host/container(s) shared access to a bind mounted dir, it needs to be
    shared instead of the default private mode.

    Additionally, fix the missing /var/lib/ironic ownership change in
    the ironix-pxe's kolla config.

    Partial-bug: #1783762
    Change-Id: I5ad4956174346f024945852da09f3841c3378c2b
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 90a7a22f157b0f50d024d730e8b97ba268bc58ad)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/619516
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=dcc6aeeef3f3d92b985d5824394a0def2adc49a8
Submitter: Zuul
Branch: stable/pike

commit dcc6aeeef3f3d92b985d5824394a0def2adc49a8
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Jul 26 15:44:29 2018 +0300

    Fix ironic containers data owning races

    Complements I5ad4956174346f0$24945852da09f3841c3378c2b.
    Fix ironic_pxe/inspector containers missing the kolla_config sections
    to recursively chown /var/lib/ironic* shared paths. That removes data
    owning races across other ironic containers starting on earlier or the
    same steps/order.

    For example, when ironic_pxe_http fails to access the
    /var/lib/ironic/httpboot path shared with ironic_conductor, the latter
    reports errors like:

    DriverLoadError: Driver, hardware type or interface ilo-pxe could not
    be loaded. Reason: [Errno 13] Permission denied:
    '/var/lib/ironic/httpboot/...

    Fixing the dir owner allows HW provisioning to complete w/o such
    errors.

    Change-Id: I34b54077c1c610a4bed73129e9784b2fa353e944
    Closes-bug: #1783762
    (cherry picked from commit 9d9daebdda3093e795b83075349b12d23f9241e8)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.18 release.