containers might not be restarted after config change

Bug #1848775 reported by Radosław Piliszek
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
Medium
Unassigned
Ussuri
Fix Released
Medium
Unassigned
Victoria
Fix Released
Medium
Unassigned
Wallaby
Fix Committed
Medium
Unassigned
Xena
Fix Released
Medium
Unassigned

Bug Description

When config is templated out and kolla ansible fails afterwards, the next run will not restart containers.

Revision history for this message
Mark Goddard (mgoddard) wrote :

I thought about this recently. We could also miss a restart if we to a kolla-ansible genconfig followed by a kolla-ansible deploy-containers if the config changes but the image does not.

One possible solution I thought of is to switch the handler to touch a file, then check for that file in deploy-containers and kick the restart handler if it's present. That doesn't fix your issue though.

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Why not? We could do the exact same in deploy.

Revision history for this message
Mark Goddard (mgoddard) wrote :

If the handler that writes out the file never runs then config gets changed without leaving a trace.

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :
Changed in kolla-ansible:
status: Triaged → In Progress
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/773246
Committed: https://opendev.org/openstack/kolla-ansible/commit/c3afbd3c5efb84a1df272bd44a46d4db60c69f0b
Submitter: "Zuul (22348)"
Branch: master

commit c3afbd3c5efb84a1df272bd44a46d4db60c69f0b
Author: Radosław Piliszek <email address hidden>
Date: Fri Jul 31 17:54:52 2020 +0200

    Check config when checking the containers

    The proposed approach allows for checking whether config
    files are current, e.g. cases when the deployment was aborted after
    config files were generated but before they were injected into the
    containers which lead to old config staying in containers.

    After this patch we can do:
      kolla-ansible genconfig
      kolla-ansible deploy-containers
    and it would do what we expected rather than being a noop
    in the second part.

    We also lose the need to have notifies
    and whens in config and handler sections respectively.
    This is optimised in a separate patch.

    Future work:
    - optimise for large files
      - could we get away with comparing timestamps and sizes?
        container's should have a newer timestamp due to copy,
        could also preserve it

    Change-Id: I1d26e48e1958f13b854d8afded4bfba5021a2dec
    Closes-Bug: #1848775
    Depends-On: https://review.opendev.org/c/openstack/kolla/+/773257
    Co-Authored-By: Mark Goddard <email address hidden>

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/803671

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/803672

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/803673

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/803673
Committed: https://opendev.org/openstack/kolla-ansible/commit/a43ca26734546be18e0bd5c04afca873fc8864ad
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit a43ca26734546be18e0bd5c04afca873fc8864ad
Author: Radosław Piliszek <email address hidden>
Date: Fri Jul 31 17:54:52 2020 +0200

    Check config when checking the containers

    The proposed approach allows for checking whether config
    files are current, e.g. cases when the deployment was aborted after
    config files were generated but before they were injected into the
    containers which lead to old config staying in containers.

    After this patch we can do:
      kolla-ansible genconfig
      kolla-ansible deploy-containers
    and it would do what we expected rather than being a noop
    in the second part.

    We also lose the need to have notifies
    and whens in config and handler sections respectively.
    This is optimised in a separate patch.

    Future work:
    - optimise for large files
      - could we get away with comparing timestamps and sizes?
        container's should have a newer timestamp due to copy,
        could also preserve it

    Change-Id: I1d26e48e1958f13b854d8afded4bfba5021a2dec
    Closes-Bug: #1848775
    Depends-On: https://review.opendev.org/c/openstack/kolla/+/773257
    Co-Authored-By: Mark Goddard <email address hidden>
    (cherry picked from commit c3afbd3c5efb84a1df272bd44a46d4db60c69f0b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/803672
Committed: https://opendev.org/openstack/kolla-ansible/commit/7cc41b391e860c91361e70b5b121e8da0baa9fdc
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 7cc41b391e860c91361e70b5b121e8da0baa9fdc
Author: Radosław Piliszek <email address hidden>
Date: Fri Jul 31 17:54:52 2020 +0200

    Check config when checking the containers

    The proposed approach allows for checking whether config
    files are current, e.g. cases when the deployment was aborted after
    config files were generated but before they were injected into the
    containers which lead to old config staying in containers.

    After this patch we can do:
      kolla-ansible genconfig
      kolla-ansible deploy-containers
    and it would do what we expected rather than being a noop
    in the second part.

    We also lose the need to have notifies
    and whens in config and handler sections respectively.
    This is optimised in a separate patch.

    Future work:
    - optimise for large files
      - could we get away with comparing timestamps and sizes?
        container's should have a newer timestamp due to copy,
        could also preserve it

    Change-Id: I1d26e48e1958f13b854d8afded4bfba5021a2dec
    Closes-Bug: #1848775
    Depends-On: https://review.opendev.org/c/openstack/kolla/+/773257
    Co-Authored-By: Mark Goddard <email address hidden>
    (cherry picked from commit c3afbd3c5efb84a1df272bd44a46d4db60c69f0b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/803671
Committed: https://opendev.org/openstack/kolla-ansible/commit/767f1e4b2339aa84d241758e046c490bcf1a1d40
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 767f1e4b2339aa84d241758e046c490bcf1a1d40
Author: Radosław Piliszek <email address hidden>
Date: Fri Jul 31 17:54:52 2020 +0200

    Check config when checking the containers

    The proposed approach allows for checking whether config
    files are current, e.g. cases when the deployment was aborted after
    config files were generated but before they were injected into the
    containers which lead to old config staying in containers.

    After this patch we can do:
      kolla-ansible genconfig
      kolla-ansible deploy-containers
    and it would do what we expected rather than being a noop
    in the second part.

    We also lose the need to have notifies
    and whens in config and handler sections respectively.
    This is optimised in a separate patch.

    Future work:
    - optimise for large files
      - could we get away with comparing timestamps and sizes?
        container's should have a newer timestamp due to copy,
        could also preserve it

    Change-Id: I1d26e48e1958f13b854d8afded4bfba5021a2dec
    Closes-Bug: #1848775
    Depends-On: https://review.opendev.org/c/openstack/kolla/+/773257
    Co-Authored-By: Mark Goddard <email address hidden>
    (cherry picked from commit c3afbd3c5efb84a1df272bd44a46d4db60c69f0b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 12.2.0

This issue was fixed in the openstack/kolla-ansible 12.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 13.0.0.0rc1

This issue was fixed in the openstack/kolla-ansible 13.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 10.4.0

This issue was fixed in the openstack/kolla-ansible 10.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 11.2.0

This issue was fixed in the openstack/kolla-ansible 11.2.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.