Ironic-UI reboot needed after installation

[Francesco Di Nucci]

I've been finally able to test the installation of the Ironic-UI plugin ( bug 2033929 ), enabling it on a pre-existing installation. The puppet agent run gave no errors, but the dashboard didn't work, showing "Something went wrong! An unexpected error has occurred. Try refreshing the page. If that doesn't help, contact your local administrator.".

Haven't been able to find the root cause, solved with a reboot. Maybe some services need to be reloaded after installation?

[Takashi Kajinami (kajinamit)]

Please share details about the operating system and the packages used in the env.

Also, can we have horizon log and apache log so that we can review the problem more in detail ? We may need puppet log as well.

My rough guess is that the django cache files were not updated properly after ironic-ui was installed but I need these logs to confirm it. The strange thing is that we don't see that problem in our CI.

[Francesco Di Nucci (d1nuc0m)]

Operating System: AlmaLinux 9.2

Packages in the env: see attached file

Logs: they were deleted as in the development environment we have a short log retention, we'll try to recreate the issue ASAP and post them as well

[Francesco Di Nucci (d1nuc0m)]

UPdate, we:
* re-created the system from scratch without Ironic UI on AlmaLinux 9.3
* enabled Ironic-UI adding just "include horizon::dashboards::ironic" to the manifest
* this time the dashboard kept on working, but "Admin/System/Ironic Bare Metal Provisioning" wasn't showing until a reboot

Attached there are the puppet run (redacted of sensitive info) and the Python Packages. There seemed to be nothing relevant in horizon/apache log, but we will try again and upload also these logs (before/after Ironic UI install)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/puppet-horizon/+/905123

[Takashi Kajinami (kajinamit)]

The log in #3 revealed that the current implementation lacks the required notifications so that httpd is restarted after ironic-ui is additionally installed.

I've submitted the fix for this (see #4) which also fixes a few more notification path (config change -> compress, config change -> service restart) and I guess these would be relevant to the original problem.

[Takashi Kajinami (kajinamit)]

Hmm. My observation in #5 might be wrong because I see

```
Notice: /Stage[main]/Horizon::Deps/Anchor[horizon::compress::end]: Triggered 'refresh' from 2 events
Notice: /Stage[main]/Horizon::Deps/Anchor[horizon::service::begin]: Triggered 'refresh' from 1 event
```

and deps let the service::begin anchor notify the httpd service.
I'll dig into the current notification paths more carefully to see if I can find anything else.

[Takashi Kajinami (kajinamit)]

OK I noticed the lack of notifications is regression caused by https://review.opendev.org/c/openstack/puppet-horizon/+/894562, but according to the log you are using an older version (<= stable/2023.1) and the problem does not exist in that branch.

(NOTE: It looks like the same problem appears in older branches in case compress_offline is false but according to the triggered resources you have compress_offline: true, I believe.)

Could you please check whether the httpd service is restarted by puppet after ironic-ui is additionally installed ?

If you can reproduce the problem, may I ask you to check if any of
 1. restarting httpd
 2. restarting memcached
resolves the problem ?

[Francesco Di Nucci (d1nuc0m)]

Thanks, don't know if it can be useful but here it is the puppet log (debug level, redacted of sensitive info) when ironic-UI has been enabled.

compress_offline is not set explicitly, so it should assume the default value, and if I'm not wrong as of '/spec/acceptance/horizon_with_apache_spec.rb' the default is compress_offline: false

About the services, restarting httpd was enough to show "Admin/System/Ironic Bare Metal Provisioning", without restarting memcached

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-horizon/+/905123
Committed: https://opendev.org/openstack/puppet-horizon/commit/d8098793ea571231d6668eac2973831e1d2fbb4b
Submitter: "Zuul (22348)"
Branch: master

commit d8098793ea571231d6668eac2973831e1d2fbb4b
Author: Takashi Kajinami <email address hidden>
Date: Tue Jan 9 23:45:52 2024 +0900

    Restart httpd after config change

    This fixes the missing notifications so that the httpd service is
    properly restarted after config files are updated.

    This also realigns package resources for separate dasobhards to
    the global install phase because these packages should be installed
    before config phase.

    Related-Bug: #2048037
    Change-Id: I8331c6c528391401c57b450be6bf75829179a9f8

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/puppet-horizon/+/905524

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-horizon/+/905524
Committed: https://opendev.org/openstack/puppet-horizon/commit/c40b6c453e50ea4f7534a67fb3f25c92c2496eb2
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit c40b6c453e50ea4f7534a67fb3f25c92c2496eb2
Author: Takashi Kajinami <email address hidden>
Date: Tue Jan 9 23:45:52 2024 +0900

    Restart httpd after config change

    This fixes the missing notifications so that the httpd service is
    properly restarted after config files are updated.

    This also realigns package resources for separate dasobhards to
    the global install phase because these packages should be installed
    before config phase.

    Related-Bug: #2048037
    Change-Id: I8331c6c528391401c57b450be6bf75829179a9f8
    (cherry picked from commit d8098793ea571231d6668eac2973831e1d2fbb4b)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/puppet-horizon/+/905742

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-horizon/+/905742
Committed: https://opendev.org/openstack/puppet-horizon/commit/1d4360ca7042c6bec1b356ece24a58facbe4c0e1
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 1d4360ca7042c6bec1b356ece24a58facbe4c0e1
Author: Takashi Kajinami <email address hidden>
Date: Tue Jan 9 23:45:52 2024 +0900

    Restart httpd after config change

    This fixes the missing notifications so that the httpd service is
    properly restarted after config files are updated.

    This also realigns package resources for separate dasobhards to
    the global install phase because these packages should be installed
    before config phase.

    Conflicts:
            manifests/init.pp

    Backport note:
    This resolved conflict caused by [1].

    [1] d967012a7a3bdeab4f131c03661fec8ca60b6805

    Related-Bug: #2048037
    Change-Id: I8331c6c528391401c57b450be6bf75829179a9f8
    (cherry picked from commit d8098793ea571231d6668eac2973831e1d2fbb4b)
    (cherry picked from commit c40b6c453e50ea4f7534a67fb3f25c92c2496eb2)

[Takashi Kajinami (kajinamit)]

So we merged the change which may potentially fix the problem.
By any chance do you have env/time to try these packages ?

We could not reproduce the reported problem in CI so far, likely because we do not cover the specific scenario there.

[Francesco Di Nucci (d1nuc0m)]

> By any chance do you have env/time to try these packages ?

Done, now it works! Created a dashboard instance without Ironic-UI, logged in (ok), enabled it, puppet run, logged in again and there is "Admin/System/Ironic Bare Metal Provisioning".

Thank you very much!

[Takashi Kajinami (kajinamit)]

Thanks, so I'll close this bug now upon that confirmation.

[Takashi Kajinami (kajinamit)]

This was actually fixed by the related fix.