Cannot delete TLS Terminated listener if container has been deleted in barbican

Bug #1852599 reported by Drew Freiberger
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
High
Unassigned
Rocky
High
Unassigned
Stein
High
Unassigned
Train
High
Unassigned
octavia
Fix Released
Undecided
Unassigned
octavia (Ubuntu)
High
Unassigned
Disco
High
Unassigned
Eoan
High
Unassigned
Focal
High
Unassigned

Bug Description

[Impact]
octavia lb's listener can't be delete when there is no secret container which is already deleted.

[Test Case]

1. deploy and setup octavia env
2. run below script
- https://pastebin.ubuntu.com/p/xPWKnPqFhw/
3. check if there is listener-1 is there or not ( in this case, there is )

[Regression]

This fix needs restarting octavia services. so downtime will be there shortly.

[Others]

After patching with https://review.opendev.org/#/c/691693/,

I Was able to see the same error , but listener is actually deleted.

[Original Description]

There seems to be a fault condition in amphora builds that can lead to not being able to delete a failed loadbalancer/listener.

If the barbican container for a tls terminated endpoint listener is deleted before the listener/amphora is deleted, the query to update to the listener's tls certificates fails during the delete_pools flow.

It exhibits the following in the client:

DELETE call to None for https://octavia.mysite:9876/v2.0/lbaas/listeners/4f46a6a3-a756-41b2-9148-c060bf28e621 used request id req-8146cc8c-334a-4155-9fa6-489d4cf1ecbf
Request returned failure status: 500
Not Found: Not Found. Sorry but your container is in another castle. (HTTP 500) (Request-ID: req-8146cc8c-334a-4155-9fa6-489d4cf1ecbf)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 29, in wrapper
    response = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 204, in listener_delete
    response = self.delete(url)
  File "/usr/lib/python3/dist-packages/osc_lib/api/api.py", line 185, in delete
    return self._request('DELETE', url, **params)
  File "/usr/lib/python3/dist-packages/osc_lib/api/api.py", line 141, in _request
    return session.request(url, method, **kwargs)
  File "/usr/lib/python3/dist-packages/osc_lib/session.py", line 40, in request
    resp = super(TimingSession, self).request(url, method, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 737, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500) (Request-ID: req-8146cc8c-334a-4155-9fa6-489d4cf1ecbf)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3/dist-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3/dist-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3/dist-packages/octaviaclient/osc/v2/listener.py", line 152, in take_action
    listener_id=listener_id)
  File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 38, in wrapper
    request_id=e.request_id)
octaviaclient.api.v2.octavia.OctaviaClientException: Not Found: Not Found. Sorry but your container is in another castle. (HTTP 500) (Request-ID: req-8146cc8c-334a-4155-9fa6-489d4cf1ecbf)
clean_up DeleteListener: Not Found: Not Found. Sorry but your container is in another castle. (HTTP 500) (Request-ID: req-8146cc8c-334a-4155-9fa6-489d4cf1ecbf)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 29, in wrapper
    response = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 204, in listener_delete
    response = self.delete(url)
  File "/usr/lib/python3/dist-packages/osc_lib/api/api.py", line 185, in delete
    return self._request('DELETE', url, **params)
  File "/usr/lib/python3/dist-packages/osc_lib/api/api.py", line 141, in _request
    return session.request(url, method, **kwargs)
  File "/usr/lib/python3/dist-packages/osc_lib/session.py", line 40, in request
    resp = super(TimingSession, self).request(url, method, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 737, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500) (Request-ID: req-8146cc8c-334a-4155-9fa6-489d4cf1ecbf)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/osc_lib/shell.py", line 134, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python3/dist-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python3/dist-packages/osc_lib/shell.py", line 169, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3/dist-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3/dist-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3/dist-packages/octaviaclient/osc/v2/listener.py", line 152, in take_action
    listener_id=listener_id)
  File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 38, in wrapper
    request_id=e.request_id)
octaviaclient.api.v2.octavia.OctaviaClientException: Not Found: Not Found. Sorry but your container is in another castle. (HTTP 500) (Request-ID: req-8146cc8c-334a-4155-9fa6-489d4cf1ecbf)

The server side log shows:

2019-11-05 14:41:48.288 2020577 WARNING octavia.controller.worker.controller_worker [req-046fe879-63c2-4804-9732-9985faf380e6 - ef8b2568c694461499c074c641a57a14 - - -] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.ListenersUpdate' (1e77205e-486a-432b-b236-8e806c3c2b7e) transitioned into state 'FAILURE' from state 'RUNNING'
5 predecessors (most recent first):
  Atom 'octavia.controller.worker.tasks.model_tasks.DeleteModelObject' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'object': <octavia.common.data_models.Pool object at 0x7fd217b21978>}, 'provides': None}
  |__Atom 'octavia.controller.worker.tasks.database_tasks.CountPoolChildrenForQuota' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'pool': <octavia.common.data_models.Pool object at 0x7fd217b21978>}, 'provides': {'HM': 1, 'member': 3}}
     |__Atom 'octavia.controller.worker.tasks.database_tasks.MarkPoolPendingDeleteInDB' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'pool': <octavia.common.data_models.Pool object at 0x7fd217b21978>}, 'provides': None}
        |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.PoolToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'pool': <octavia.common.data_models.Pool object at 0x7fd217b21978>, 'listeners': [<octavia.common.data_models.Listener object at 0x7fd217b7f400>], 'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7fd216dc6550>}, 'provides': None}
           |__Flow 'octavia-delete-pool-flow': barbicanclient.exceptions.HTTPClientError: Not Found: Not Found. Sorry but your container is in another castle.
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker Traceback (most recent call last):
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/certificates/manager/barbican.py", line 114, in get_cert
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker return pkcs12.PKCS12Cert(cert_secret.payload)
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/v1/secrets.py", line 193, in payload
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker self._fetch_payload()
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/v1/secrets.py", line 261, in _fetch_payload
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker if not self.payload_content_type and not self.content_types:
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/v1/secrets.py", line 184, in payload_content_type
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker if not self._payload_content_type and self.content_types:
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/v1/secrets.py", line 34, in wrapper
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker self._fill_lazy_properties()
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/v1/secrets.py", line 414, in _fill_lazy_properties
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker result = self._api.get(self._secret_ref)
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/client.py", line 70, in get
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker return super(_HTTPClient, self).get(*args, **kwargs).json()
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 375, in get
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker return self.request(url, 'GET', **kwargs)
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/client.py", line 63, in request
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker self._check_status_code(resp)
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/client.py", line 107, in _check_status_code
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker status
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker barbicanclient.exceptions.HTTPClientError: Not Found: Not Found. Sorry but your container is in another castle.
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker During handling of the above exception, another exception occurred:
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker Traceback (most recent call last):
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker result = task.execute(**arguments)
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 78, in execute
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker self.amphora_driver.update(listener, loadbalancer.vip)
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 162, in update
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker certs = self._process_tls_certificates(listener)
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 284, in _process_tls_certificates
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker self.cert_manager, listener)
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/common/tls_utils/cert_parser.py", line 353, in load_certificates_data
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker check_only=True))
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/certificates/manager/barbican.py", line 122, in get_cert
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker check_only=check_only, service_name=service_name
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/certificates/manager/barbican_legacy.py", line 160, in get_cert
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker LOG.error('Error getting cert %s: %s', cert_ref, str(e))
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker self.force_reraise()
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker six.reraise(self.type_, self.value, self.tb)
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker raise value
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/certificates/manager/barbican_legacy.py", line 138, in get_cert
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker container_ref=cert_ref
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/v1/containers.py", line 540, in get
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker response = self._api.get(container_ref)
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/client.py", line 70, in get
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker return super(_HTTPClient, self).get(*args, **kwargs).json()
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 375, in get
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker return self.request(url, 'GET', **kwargs)
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/client.py", line 63, in request
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker self._check_status_code(resp)
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/client.py", line 107, in _check_status_code
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker status
2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker barbicanclient.exceptions.HTTPClientError: Not Found: Not Found. Sorry but your container is in another castle.

Revision history for this message
Drew Freiberger (afreiberger) wrote :

I tried at first undeleting the container entity in the mysql/barbican DB for the missing barbican container, however, that provided project ID/permissions issues in barbican.

Since the container was deleted in barbican, I tried using:
openstack loadbalancer listener set --default-tls-container-ref '' <listener-uuid>

But that was giving the "container is in another castle" error as well.

I modified the listener table in the octavia database to zero out the default-tls-container-ref string after validating that the containers were deleted in barbican.

mysql> select * from listener where load_balancer_id = 'c066406f-0f5e-427b-bba8-ab3e8cf18603'\G
*************************** 1. row ***************************
                  project_id: ef8b2568c694461499c074c641a57a14
                          id: 51823607-bb4d-488c-87c1-1e22e0a11d81
                        name: NULL
                 description: NULL
                    protocol: TERMINATED_HTTPS
               protocol_port: 443
            connection_limit: -1
            load_balancer_id: c066406f-0f5e-427b-bba8-ab3e8cf18603
          tls_certificate_id: https://barbican-internal.mysite:9312/v1/containers/fb1f6514-000f-4991-8ccd-4da923e19ab2
             default_pool_id: e05d9cad-e1c1-4b3c-9204-dcd79e1d5ee3
         provisioning_status: ACTIVE
            operating_status: ONLINE
                     enabled: 1
                   peer_port: 1025
              insert_headers: �� }�.
                  created_at: 2019-11-01 20:03:58
                  updated_at: 2019-11-05 14:02:04
         timeout_client_data: 50000
      timeout_member_connect: 5000
         timeout_member_data: 50000
         timeout_tcp_inspect: 0
client_ca_tls_certificate_id: NULL
       client_authentication: NONE
     client_crl_container_id: NULL
1 row in set (0.00 sec)

mysql> update listener set tls_certificate_id='' where load_balancer_id = 'c066406f-0f5e-427b-bba8-ab3e8cf18603';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0

This then allowed for a cascaded loadbalancer delete.

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Hi Drew, thanks for the bug. Please could you:

1. List the commands (in order) that reproduce this issue?
2. Provide the relevant parts of the model that are involved -- which applications.

 Do you have an ordering of commands that doesn't cause the issue?

Thanks

Changed in charm-octavia:
status: New → Incomplete
Seyeong Kim (seyeongkim)
tags: added: sts
Ryan Beisner (1chb1n)
Changed in charm-octavia:
status: Incomplete → Invalid
Revision history for this message
Ryan Beisner (1chb1n) wrote :

As an upstream Octavia issue, this is not likely to be solved at the Charm level.

It is raised appropriately in upstream Octavia's Storyboard (bug tracker):

https://storyboard.openstack.org/#!/story/2006901

Revision history for this message
Ryan Beisner (1chb1n) wrote :

I've chatted with upstream, was advised that Story 2006901 is actually a dup of an existing bug/story:

Deleted secrets prevent CUD operations and trigger amphora failover
https://storyboard.openstack.org/#!/story/2006676

Revision history for this message
Ryan Beisner (1chb1n) wrote :

The fixes have merged upstream, and we will need to do stable updates in distro and UCA.

Queens: https://review.opendev.org/#/c/691757/
Rocky: https://review.opendev.org/#/c/691748/
Stein: https://review.opendev.org/#/c/691693/
Train: https://review.opendev.org/#/c/690984/
Master (Ussuri): https://review.opendev.org/#/c/688221/

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in octavia (Ubuntu):
status: New → Confirmed
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Drew, this is fixed upstream in all releases supported by the charm (rocky+) but hasn't been included in any stable point releases yet. We could cherry-pick it but I'd prefer to pick the fix up in stable point releases as it simplifies testing efforts. johnsonm in #openstack-lbaas thinks stable point releases are coming soon. Are you ok with waiting on the stable point releases?

Changed in octavia (Ubuntu):
status: Confirmed → Triaged
importance: Undecided → High
Changed in octavia (Ubuntu Disco):
status: New → Triaged
Changed in octavia (Ubuntu Eoan):
status: New → Triaged
importance: Undecided → High
Changed in octavia (Ubuntu Disco):
importance: Undecided → High
no longer affects: cloud-archive/ussuri
Changed in cloud-archive:
status: New → Triaged
importance: Undecided → High
no longer affects: cloud-archive/queens
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Fyi on stable point releases: https://review.opendev.org/#/c/697688/

Revision history for this message
Seyeong Kim (seyeongkim) wrote :

@corey

Do we need to backport this or octavia 4.1.1 ( or for the other releases) will be released soon for UCA ?

Revision history for this message
Seyeong Kim (seyeongkim) wrote :

It seems that rocky already has fix since 1 hour ago.

Revision history for this message
Seyeong Kim (seyeongkim) wrote :
Revision history for this message
Seyeong Kim (seyeongkim) wrote :
Revision history for this message
Seyeong Kim (seyeongkim) wrote :
Revision history for this message
Seyeong Kim (seyeongkim) wrote :
Revision history for this message
Seyeong Kim (seyeongkim) wrote :
Revision history for this message
Seyeong Kim (seyeongkim) wrote :

focal has it

tags: added: sru-needed
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "lp1852599_disco.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Seyeong Kim (seyeongkim)
Changed in octavia (Ubuntu Focal):
status: Triaged → Fix Released
Seyeong Kim (seyeongkim)
description: updated
Revision history for this message
Corey Bryant (corey.bryant) wrote :

For Eoan/Train, this fix is included in the 5.0.1 stable point releases via https://bugs.launchpad.net/cloud-archive/+bug/1858933.

Changed in cloud-archive:
status: Triaged → Fix Released
Revision history for this message
Corey Bryant (corey.bryant) wrote :

For Stein (Disco is EOL this month), this fix is included in the 4.1.1 stable point release via https://bugs.launchpad.net/cloud-archive/+bug/1858934.

Changed in octavia (Ubuntu Disco):
status: Triaged → Won't Fix
Mathew Hodson (mhodson)
affects: charm-octavia → octavia
Changed in octavia:
status: Invalid → Fix Released
Revision history for this message
Corey Bryant (corey.bryant) wrote :
Changed in octavia (Ubuntu Eoan):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers