kolla

Nova containers fail on password rotation

Bug #2045558 reported by Alex Welsh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Status tracked in Caracal
Antelope
Fix Released
Undecided
Unassigned
Bobcat
Fix Released
Undecided
Unassigned
Caracal
Fix Released
Undecided
Alex Welsh
Yoga
Fix Released
Undecided
Unassigned
Zed
Fix Released
Undecided
Unassigned

Bug Description

The Nova API extended start script runs nova-manage db sync without the --local-cell argument.

If the database password is changed, cell0 will become inaccessible and cause an error.

To reproduce: deploy Nova, then update the nova database password and redeploy.

Seen on Ubuntu Jammy Yoga but expected to impact everyone.

Error:

  msg: Container exited with non-zero return code 1
  rc: 1
  stderr: |-
    + sudo -E kolla_set_configs
    INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
    INFO:__main__:Validating config file
    INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
    INFO:__main__:Copying service configuration files
    INFO:__main__:Copying /var/lib/kolla/config_files/nova.conf to /etc/nova/nova.conf
    INFO:__main__:Setting permission for /etc/nova/nova.conf
    INFO:__main__:Writing out command to execute
    INFO:__main__:Setting permission for /var/log/kolla/nova
    INFO:__main__:Setting permission for /var/log/kolla/nova/apache-access.log
    INFO:__main__:Setting permission for /var/log/kolla/nova/nova-api-error.log.1
    INFO:__main__:Setting permission for /var/log/kolla/nova/apache-error.log
    INFO:__main__:Setting permission for /var/log/kolla/nova/nova-api-access.log
    INFO:__main__:Setting permission for /var/log/kolla/nova/nova-api-error.log
    INFO:__main__:Setting permission for /var/log/kolla/nova/nova-manage.log
    INFO:__main__:Setting permission for /var/log/kolla/nova/nova-scheduler.log
    INFO:__main__:Setting permission for /var/log/kolla/nova/nova-api.log.1
    INFO:__main__:Setting permission for /var/log/kolla/nova/nova-api.log
    INFO:__main__:Setting permission for /var/log/kolla/nova/nova-novncproxy.log
    INFO:__main__:Setting permission for /var/log/kolla/nova/nova-metadata-error.log
    INFO:__main__:Setting permission for /var/log/kolla/nova/nova-conductor.log
    INFO:__main__:Setting permission for /var/log/kolla/nova/nova-metadata-access.log
    ++ cat /run_command
    + CMD=false
    + ARGS=
    + sudo kolla_copy_cacerts
    + [[ ! -n '' ]]
    + . kolla_extend_start
    ++ [[ ! -d /var/log/kolla/nova ]]
    +++ stat -c %a /var/log/kolla/nova
    ++ [[ 2755 != \7\5\5 ]]
    ++ chmod 755 /var/log/kolla/nova
    ++ . /usr/local/bin/kolla_nova_extend_start
    +++ [[ -n '' ]]
    +++ [[ -n 0 ]]
    +++ nova-manage api_db sync
    Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
    +++ nova-manage db sync
    Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
  stderr_lines: <omitted>
  stdout: |-
    ERROR: Could not access cell0.
    Has the nova_api database been created?
    Has the nova_cell0 database been created?
    Has "nova-manage api_db sync" been run?
    Has "nova-manage cell_v2 map_cell0" been run?
    Is [api_database]/connection set in nova.conf?
    Is the cell0 database connection URL correct?
    Error: (pymysql.err.OperationalError) (1045, "Access denied for user 'nova'@'alex-mn-controller-02' (using password: YES)")
    (Background on this error at: https://sqlalche.me/e/14/e3q8)
  stdout_lines: <omitted>

See original description
Alex Welsh (alex-welsh)
Changed in kolla:
assignee: nobody → Alex Welsh (alex-welsh)
OpenStack Infra (hudson-openstack)
Changed in kolla:
status: New → In Progress
Alex Welsh (alex-welsh)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/902057
Committed: https://opendev.org/openstack/kolla/commit/de1487f05187387aea3f113dcbb6926734b1ce6b
Submitter: "Zuul (22348)"
Branch: master

commit de1487f05187387aea3f113dcbb6926734b1ce6b
Author: Alex-Welsh <email address hidden>
Date: Tue Nov 28 13:08:58 2023 +0000

    Sync only local cell in nova bootstrap & upgrade

    Added the --local_cell argument to nova db sync commands during
    bootstrap and upgrade.

    This was previously thought to have no effect [1], but has since been
    discovered to fail when rotating the nova database password.

    [1] https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/nova/tasks/bootstrap_service.yml#L2-L3

    Closes-Bug: #2045558
    Change-Id: Ic64eb51325b3503a14ebab9b9ff2f4d9caec734a

Changed in kolla:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/kolla/+/904261

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla/+/904262

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla/+/904263

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla/+/904264

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/904261
Committed: https://opendev.org/openstack/kolla/commit/8412887491cada67fa2e6c4ae146cc5bd75973fb
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 8412887491cada67fa2e6c4ae146cc5bd75973fb
Author: Alex-Welsh <email address hidden>
Date: Tue Nov 28 13:08:58 2023 +0000

    Sync only local cell in nova bootstrap & upgrade

    Added the --local_cell argument to nova db sync commands during
    bootstrap and upgrade.

    This was previously thought to have no effect [1], but has since been
    discovered to fail when rotating the nova database password.

    [1] https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/nova/tasks/bootstrap_service.yml#L2-L3

    Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/904086

    Closes-Bug: #2045558
    Change-Id: Ic64eb51325b3503a14ebab9b9ff2f4d9caec734a
    (cherry picked from commit de1487f05187387aea3f113dcbb6926734b1ce6b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/904262
Committed: https://opendev.org/openstack/kolla/commit/48031708d54df2ad65757a21d4f9c6d9cd788b59
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 48031708d54df2ad65757a21d4f9c6d9cd788b59
Author: Alex-Welsh <email address hidden>
Date: Tue Nov 28 13:08:58 2023 +0000

    Sync only local cell in nova bootstrap & upgrade

    Added the --local_cell argument to nova db sync commands during
    bootstrap and upgrade.

    This was previously thought to have no effect [1], but has since been
    discovered to fail when rotating the nova database password.

    [1] https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/nova/tasks/bootstrap_service.yml#L2-L3

    Closes-Bug: #2045558
    Change-Id: Ic64eb51325b3503a14ebab9b9ff2f4d9caec734a
    (cherry picked from commit de1487f05187387aea3f113dcbb6926734b1ce6b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/904263
Committed: https://opendev.org/openstack/kolla/commit/9dc637d4755e4ae00dedd3a367fa64b42159d49d
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 9dc637d4755e4ae00dedd3a367fa64b42159d49d
Author: Alex-Welsh <email address hidden>
Date: Tue Nov 28 13:08:58 2023 +0000

    Sync only local cell in nova bootstrap & upgrade

    Added the --local_cell argument to nova db sync commands during
    bootstrap and upgrade.

    This was previously thought to have no effect [1], but has since been
    discovered to fail when rotating the nova database password.

    [1] https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/nova/tasks/bootstrap_service.yml#L2-L3

    Closes-Bug: #2045558
    Change-Id: Ic64eb51325b3503a14ebab9b9ff2f4d9caec734a
    (cherry picked from commit de1487f05187387aea3f113dcbb6926734b1ce6b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/904264
Committed: https://opendev.org/openstack/kolla/commit/82fe25505592ea121e91a33f467bdb059da01a6a
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 82fe25505592ea121e91a33f467bdb059da01a6a
Author: Alex-Welsh <email address hidden>
Date: Tue Nov 28 13:08:58 2023 +0000

    Sync only local cell in nova bootstrap & upgrade

    Added the --local_cell argument to nova db sync commands during
    bootstrap and upgrade.

    This was previously thought to have no effect [1], but has since been
    discovered to fail when rotating the nova database password.

    [1] https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/nova/tasks/bootstrap_service.yml#L2-L3

    Closes-Bug: #2045558
    Change-Id: Ic64eb51325b3503a14ebab9b9ff2f4d9caec734a
    (cherry picked from commit de1487f05187387aea3f113dcbb6926734b1ce6b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 16.3.0

This issue was fixed in the openstack/kolla 16.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 17.1.0

This issue was fixed in the openstack/kolla 17.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 14.12.0

This issue was fixed in the openstack/kolla 14.12.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 15.4.0

This issue was fixed in the openstack/kolla 15.4.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.