serial console for baremetal nodes doesn't work in HA

[Narinder Gupta]

there is a way exist to get the access of console for virtual machine. For Ironic deployed bare metal nodes we can enable the serial console provided serial console is enabled and forwarded to right terminal in the kernel parameters.
This is the request to provide serial console with the Nova for Ironic deployed nodes.

Changes are required in nova compute and cloud controller as per documentation.

Nova Compute
[serial_console]
enabled = true
proxyclient_address = ip address
base_url = ws://hostname:6083/

nova cloud controller
[serial_console]
enabled = true
base_url = ws://hostname:6083/
#serialproxy_host=hostname
#serialproxy_port=6083

https://docs.openstack.org/nova/latest/admin/remote-console-access.html#serial

[Felipe Reyes (freyes)]

nova-cloud-controller supports enabling the serial console - https://charmhub.io/nova-cloud-controller/configure#enable-serial-console

Please can you confirm if this functionality is not working for Ironic?, be aware that enabling this option, as the description says, will disable the functionality provided by "openstack console log show"

[Felipe Reyes (freyes)]

Ironic needs extra bits to configure for the enablement of serial consoles - https://docs.openstack.org/ironic/latest/admin/console.html

[Felipe Reyes (freyes)]

I'm reading the charm's code in charm-ironic-conductor and IIUC when the "enabled-hw-types" is set to "ipmi" (the default) the configuration should include "ipmi-shellinabox" and "ipmi-socat" for "enabled_console_interfaces" while allows to configure a serial console[0]

So please check what's the value configured in enabled-hw-types[1], make sure "ipmi" is included.

Then configure a node with the following the instructions available at [2] for a web console or a serial consoled based on socat

[0] https://opendev.org/openstack/charm-ironic-conductor/src/branch/stable/ussuri/src/lib/charm/openstack/ironic/ironic.py#L64
[1] https://charmhub.io/ironic-conductor/configure?channel=ussuri/edge#enabled-hw-types
[2] https://docs.openstack.org/ironic/latest/admin/console.html

[Narinder Gupta (narindergupta)]

It seems nova-cloud controller does not populate serial proxy information.

https://docs.openstack.org/nova/ussuri/admin/remote-console-access.html#serial

Check the flow https://docs.openstack.org/nova/ussuri/_images/serial-console-flow.svg where serial proxy host and port are not available.Same is true with nova-compute-ironic as well.

[Billy Olsen (billy-olsen)]

Speaking with Narinder, this needs to be specified as the VIP or the hostname specifically.

[Felipe Reyes (freyes)]

This issue comes from nova-cloud-controller which is not updating the value passed in the relation to nova-compute

In this reproduce the units nova-cloud-controller got its virtual ip set after environment ran all the relation hooks, then adding 2 units shows that the new units set the VIP in the relation, but /0 doesn't.

```
$ juju show-unit nova-ironic/0
...
  - relation-id: 32
    endpoint: cloud-compute
    related-endpoint: cloud-compute
    application-data: {}
    related-units:
      nova-cloud-controller/0:
        in-scope: true
        data:
          ...
          serial_console_base_url: ws://10.0.0.127:6083/
      nova-cloud-controller/3:
        in-scope: true
        data:
          ...
          serial_console_base_url: ws://10.0.0.11:6083/
```

[Felipe Reyes (freyes)]

** workaround **

```
juju remove-relation nova-ironic nova-cloud-controller
# wait until the relation is completely removed
juju add-relation nova-ironic nova-cloud-controller
```

To verify the workaround succeeded run:

$ juju config nova-cloud-controller vip
10.0.0.11
$ juju show-unit nova-ironic/0 | grep serial_console_base_url
          serial_console_base_url: ws://10.0.0.11:6083/
          serial_console_base_url: ws://10.0.0.11:6083/
          serial_console_base_url: ws://10.0.0.11:6083/

the vip needs to be set in the serial_console_base_url key for the relation.

[Felipe Reyes (freyes)]

It's worth to mention that this is happening when scaling out from single unit of nova-cloud-controller to HA (3 units + hacluster), while if you change the VIP, the change gets reflected in the relation:

$ juju config nova-cloud-controller vip
10.0.0.11
$ juju show-unit nova-ironic/0 | grep serial_console_base_url
          serial_console_base_url: ws://10.0.0.12:6083/
          serial_console_base_url: ws://10.0.0.12:6083/
          serial_console_base_url: ws://10.0.0.12:6083/

[Felipe Reyes (freyes)]

This is the set of events that were executed, so when ha-relation-joined runs we need to update the relation data:

$ juju show-status-log nova-cloud-controller/0 --days 3
Time Type Status Message
20 Apr 2023 20:37:13Z juju-unit executing running config-changed hook
20 Apr 2023 20:37:39Z juju-unit idle
20 Apr 2023 20:37:52Z workload active Unit is ready
20 Apr 2023 20:38:36Z juju-unit executing running ha-relation-created hook
20 Apr 2023 20:38:37Z juju-unit idle
20 Apr 2023 20:39:46Z juju-unit executing running ha-relation-joined hook for nova-cloud-controller-hacluster/0
20 Apr 2023 20:39:53Z juju-unit executing running ha-relation-changed hook for nova-cloud-controller-hacluster/0
20 Apr 2023 20:39:59Z juju-unit idle
20 Apr 2023 20:45:18Z juju-unit executing running cluster-relation-joined hook for nova-cloud-controller/4
20 Apr 2023 20:45:24Z juju-unit executing running cluster-relation-changed hook for nova-cloud-controller/4
20 Apr 2023 20:45:40Z juju-unit executing running cluster-relation-joined hook for nova-cloud-controller/3
20 Apr 2023 20:45:47Z juju-unit executing running cluster-relation-changed hook for nova-cloud-controller/3
20 Apr 2023 20:46:02Z juju-unit idle
20 Apr 2023 20:48:04Z workload blocked Missing relations: memcache
20 Apr 2023 20:50:29Z juju-unit executing running memcache-relation-created hook
20 Apr 2023 20:51:09Z juju-unit executing running cluster-relation-changed hook for nova-cloud-controller/4
20 Apr 2023 20:52:02Z juju-unit executing running ha-relation-changed hook for nova-cloud-controller-hacluster/0
20 Apr 2023 20:52:09Z juju-unit executing running cluster-relation-changed hook for nova-cloud-controller/3
20 Apr 2023 20:52:24Z workload waiting Incomplete relations: memcache
20 Apr 2023 20:52:25Z juju-unit idle
20 Apr 2023 20:52:28Z juju-unit executing running memcache-relation-joined hook for memcached/0
20 Apr 2023 20:52:38Z juju-unit executing running memcache-relation-changed hook for memcached/0
20 Apr 2023 20:52:45Z juju-unit idle

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881288

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881442

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881443

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881444

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881445

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881446

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881447

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881448

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881288
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/366df4c07e2cd6b36345051c90585d27916e81a6
Submitter: "Zuul (22348)"
Branch: master

commit 366df4c07e2cd6b36345051c90585d27916e81a6
Author: Felipe Reyes <email address hidden>
Date: Fri Apr 21 19:36:01 2023 -0400

    Update nova relations data on ha-relation-changed

    When taking the nova-cloud-controller from single unit to full HA by
    increasing the number of units from 1 to 3 and relating it to hacluster,
    the data set on the cloud-compute relation is not updated, because the
    update_nova_relation() function is only called on
    cloud-compute-relation-joined and config-changed, none of these hooks
    are executed when scaling out the application.

    This patch introduces a call to update_nova_relation() on
    ha-relation-changed.

    Test case on an environment deployed with a single unit of
    nova-cloud-controller:

        export NOVA_CC_VIP=10.0.0.11
        juju config nova-cloud-controller vip=$NOVA_CC_VIP
        juju deploy --series jammy --channel 2.4/stable hacluster \
            nova-cloud-controller-hacluster
        juju add-unit -n 2 nova-cloud-controller
        juju deploy --series jammy memcached
        juju add-relation memcached nova-cloud-controller
        juju add-relation nova-cloud-controller nova-cloud-controller-hacluster

    Change-Id: Ib08bf9b6e1ce2b69be4d99ffe0726b59d81f4bc9
    Closes-Bug: #2002154

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881650

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881442
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/d8a8aa2b42fdd20929b7a12d0322524c93474f61
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit d8a8aa2b42fdd20929b7a12d0322524c93474f61
Author: Felipe Reyes <email address hidden>
Date: Fri Apr 21 19:36:01 2023 -0400

    Update nova relations data on ha-relation-changed

    When taking the nova-cloud-controller from single unit to full HA by
    increasing the number of units from 1 to 3 and relating it to hacluster,
    the data set on the cloud-compute relation is not updated, because the
    update_nova_relation() function is only called on
    cloud-compute-relation-joined and config-changed, none of these hooks
    are executed when scaling out the application.

    This patch introduces a call to update_nova_relation() on
    ha-relation-changed.

    Test case on an environment deployed with a single unit of
    nova-cloud-controller:

        export NOVA_CC_VIP=10.0.0.11
        juju config nova-cloud-controller vip=$NOVA_CC_VIP
        juju deploy --series jammy --channel 2.4/stable hacluster \
            nova-cloud-controller-hacluster
        juju add-unit -n 2 nova-cloud-controller
        juju deploy --series jammy memcached
        juju add-relation memcached nova-cloud-controller
        juju add-relation nova-cloud-controller nova-cloud-controller-hacluster

    Change-Id: Ib08bf9b6e1ce2b69be4d99ffe0726b59d81f4bc9
    Closes-Bug: #2002154

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881443
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/323923ed20d342b90997d3e5e9531932782f0a12
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 323923ed20d342b90997d3e5e9531932782f0a12
Author: Felipe Reyes <email address hidden>
Date: Fri Apr 21 19:36:01 2023 -0400

    Update nova relations data on ha-relation-changed

    When taking the nova-cloud-controller from single unit to full HA by
    increasing the number of units from 1 to 3 and relating it to hacluster,
    the data set on the cloud-compute relation is not updated, because the
    update_nova_relation() function is only called on
    cloud-compute-relation-joined and config-changed, none of these hooks
    are executed when scaling out the application.

    This patch introduces a call to update_nova_relation() on
    ha-relation-changed.

    Test case on an environment deployed with a single unit of
    nova-cloud-controller:

        export NOVA_CC_VIP=10.0.0.11
        juju config nova-cloud-controller vip=$NOVA_CC_VIP
        juju deploy --series jammy --channel 2.4/stable hacluster \
            nova-cloud-controller-hacluster
        juju add-unit -n 2 nova-cloud-controller
        juju deploy --series jammy memcached
        juju add-relation memcached nova-cloud-controller
        juju add-relation nova-cloud-controller nova-cloud-controller-hacluster

    Change-Id: Ib08bf9b6e1ce2b69be4d99ffe0726b59d81f4bc9
    Closes-Bug: #2002154

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881445
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/35e03099a0af3df46efbf0856824d940f6f002ec
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 35e03099a0af3df46efbf0856824d940f6f002ec
Author: Felipe Reyes <email address hidden>
Date: Fri Apr 21 19:36:01 2023 -0400

    Update nova relations data on ha-relation-changed

    When taking the nova-cloud-controller from single unit to full HA by
    increasing the number of units from 1 to 3 and relating it to hacluster,
    the data set on the cloud-compute relation is not updated, because the
    update_nova_relation() function is only called on
    cloud-compute-relation-joined and config-changed, none of these hooks
    are executed when scaling out the application.

    This patch introduces a call to update_nova_relation() on
    ha-relation-changed.

    Test case on an environment deployed with a single unit of
    nova-cloud-controller:

        export NOVA_CC_VIP=10.0.0.11
        juju config nova-cloud-controller vip=$NOVA_CC_VIP
        juju deploy --series jammy --channel 2.4/stable hacluster \
            nova-cloud-controller-hacluster
        juju add-unit -n 2 nova-cloud-controller
        juju deploy --series jammy memcached
        juju add-relation memcached nova-cloud-controller
        juju add-relation nova-cloud-controller nova-cloud-controller-hacluster

    Change-Id: Ib08bf9b6e1ce2b69be4d99ffe0726b59d81f4bc9
    Closes-Bug: #2002154

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881447
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/d09ed28af0fb2a3761452d6afa2f30bd6020249e
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit d09ed28af0fb2a3761452d6afa2f30bd6020249e
Author: Felipe Reyes <email address hidden>
Date: Fri Apr 21 19:36:01 2023 -0400

    Update nova relations data on ha-relation-changed

    When taking the nova-cloud-controller from single unit to full HA by
    increasing the number of units from 1 to 3 and relating it to hacluster,
    the data set on the cloud-compute relation is not updated, because the
    update_nova_relation() function is only called on
    cloud-compute-relation-joined and config-changed, none of these hooks
    are executed when scaling out the application.

    This patch introduces a call to update_nova_relation() on
    ha-relation-changed.

    Test case on an environment deployed with a single unit of
    nova-cloud-controller:

        export NOVA_CC_VIP=10.0.0.11
        juju config nova-cloud-controller vip=$NOVA_CC_VIP
        juju deploy --series jammy --channel 2.4/stable hacluster \
            nova-cloud-controller-hacluster
        juju add-unit -n 2 nova-cloud-controller
        juju deploy --series jammy memcached
        juju add-relation memcached nova-cloud-controller
        juju add-relation nova-cloud-controller nova-cloud-controller-hacluster

    Change-Id: Ib08bf9b6e1ce2b69be4d99ffe0726b59d81f4bc9
    Closes-Bug: #2002154

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881444
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/c33790155b92b79eda05693cd1abac4c00cc7dd0
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit c33790155b92b79eda05693cd1abac4c00cc7dd0
Author: Felipe Reyes <email address hidden>
Date: Fri Apr 21 19:36:01 2023 -0400

    Update nova relations data on ha-relation-changed

    When taking the nova-cloud-controller from single unit to full HA by
    increasing the number of units from 1 to 3 and relating it to hacluster,
    the data set on the cloud-compute relation is not updated, because the
    update_nova_relation() function is only called on
    cloud-compute-relation-joined and config-changed, none of these hooks
    are executed when scaling out the application.

    This patch introduces a call to update_nova_relation() on
    ha-relation-changed.

    Test case on an environment deployed with a single unit of
    nova-cloud-controller:

        export NOVA_CC_VIP=10.0.0.11
        juju config nova-cloud-controller vip=$NOVA_CC_VIP
        juju deploy --series jammy --channel 2.4/stable hacluster \
            nova-cloud-controller-hacluster
        juju add-unit -n 2 nova-cloud-controller
        juju deploy --series jammy memcached
        juju add-relation memcached nova-cloud-controller
        juju add-relation nova-cloud-controller nova-cloud-controller-hacluster

    Change-Id: Ib08bf9b6e1ce2b69be4d99ffe0726b59d81f4bc9
    Closes-Bug: #2002154

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881448
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/90e3f9e723368b6be42a72a7241fe3e562f20640
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 90e3f9e723368b6be42a72a7241fe3e562f20640
Author: Felipe Reyes <email address hidden>
Date: Fri Apr 21 19:36:01 2023 -0400

    Update nova relations data on ha-relation-changed

    When taking the nova-cloud-controller from single unit to full HA by
    increasing the number of units from 1 to 3 and relating it to hacluster,
    the data set on the cloud-compute relation is not updated, because the
    update_nova_relation() function is only called on
    cloud-compute-relation-joined and config-changed, none of these hooks
    are executed when scaling out the application.

    This patch introduces a call to update_nova_relation() on
    ha-relation-changed.

    Test case on an environment deployed with a single unit of
    nova-cloud-controller:

        export NOVA_CC_VIP=10.0.0.11
        juju config nova-cloud-controller vip=$NOVA_CC_VIP
        juju deploy --series jammy --channel 2.4/stable hacluster \
            nova-cloud-controller-hacluster
        juju add-unit -n 2 nova-cloud-controller
        juju deploy --series jammy memcached
        juju add-relation memcached nova-cloud-controller
        juju add-relation nova-cloud-controller nova-cloud-controller-hacluster

    Change-Id: Ib08bf9b6e1ce2b69be4d99ffe0726b59d81f4bc9
    Closes-Bug: #2002154

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881446
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/74331fde3ed6177781076472d27c8232f7097083
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 74331fde3ed6177781076472d27c8232f7097083
Author: Felipe Reyes <email address hidden>
Date: Fri Apr 21 19:36:01 2023 -0400

    Update nova relations data on ha-relation-changed

    When taking the nova-cloud-controller from single unit to full HA by
    increasing the number of units from 1 to 3 and relating it to hacluster,
    the data set on the cloud-compute relation is not updated, because the
    update_nova_relation() function is only called on
    cloud-compute-relation-joined and config-changed, none of these hooks
    are executed when scaling out the application.

    This patch introduces a call to update_nova_relation() on
    ha-relation-changed.

    Test case on an environment deployed with a single unit of
    nova-cloud-controller:

        export NOVA_CC_VIP=10.0.0.11
        juju config nova-cloud-controller vip=$NOVA_CC_VIP
        juju deploy --series jammy --channel 2.4/stable hacluster \
            nova-cloud-controller-hacluster
        juju add-unit -n 2 nova-cloud-controller
        juju deploy --series jammy memcached
        juju add-relation memcached nova-cloud-controller
        juju add-relation nova-cloud-controller nova-cloud-controller-hacluster

    Change-Id: Ib08bf9b6e1ce2b69be4d99ffe0726b59d81f4bc9
    Closes-Bug: #2002154

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/881650
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/6cbc38e0085e51348b31fd9e21575821c578578d
Submitter: "Zuul (22348)"
Branch: stable/train

commit 6cbc38e0085e51348b31fd9e21575821c578578d
Author: Felipe Reyes <email address hidden>
Date: Fri Apr 21 19:36:01 2023 -0400

    Update nova relations data on ha-relation-changed

    When taking the nova-cloud-controller from single unit to full HA by
    increasing the number of units from 1 to 3 and relating it to hacluster,
    the data set on the cloud-compute relation is not updated, because the
    update_nova_relation() function is only called on
    cloud-compute-relation-joined and config-changed, none of these hooks
    are executed when scaling out the application.

    This patch introduces a call to update_nova_relation() on
    ha-relation-changed.

    Test case on an environment deployed with a single unit of
    nova-cloud-controller:

        export NOVA_CC_VIP=10.0.0.11
        juju config nova-cloud-controller vip=$NOVA_CC_VIP
        juju deploy --series jammy --channel 2.4/stable hacluster \
            nova-cloud-controller-hacluster
        juju add-unit -n 2 nova-cloud-controller
        juju deploy --series jammy memcached
        juju add-relation memcached nova-cloud-controller
        juju add-relation nova-cloud-controller nova-cloud-controller-hacluster

    Change-Id: Ib08bf9b6e1ce2b69be4d99ffe0726b59d81f4bc9
    Closes-Bug: #2002154
    (cherry picked from commit 366df4c07e2cd6b36345051c90585d27916e81a6)

[Narinder Gupta (narindergupta)]

i am seeing sol deactivate issued causes the connection not availbale to end user. Attaching logs to this bug for further investigation.

[Felipe Reyes (freyes)]

2023-07-24 11:23:56.981 1846629 ERROR nova.console.websocketproxy nova.exception.ValidationError: Origin header does not match this host.

this is an interesting exception.

[Felipe Reyes (freyes)]

https://docs.openstack.org/nova/latest/configuration/config.html#console.allowed_origins

there is in place a cross site security for the serial console proxy, so when we are using a standalone script to connect to the websocket to get to the ipmi console the service is dropping the connection.

I will check if it's possible to circumvent this.

[Felipe Reyes (freyes)]

I filed a bug about the allowed_origins - https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/2030094