container-list fails with unexpected bay-id

[Adrian Otto]

stack@ao-magnum-metal:~$ cat demo.json
{"name":"demo","image_id":"centos:centos6","bay_uuid":"74a09f86-d3b1-4f75-aca5-f4e03a941ba6","command":"sleep 365d"}
stack@ao-magnum-metal:~$ magnum container-create --json demo.json
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| uuid | 4a12301e-e0ba-45c2-964f-f07757deb146 |
| links | [{u'href': u'http://104.130.20.198:9511/v1/containers/4a12301e-e0ba-45c2-964f-f07757deb146', u'rel': u'self'}, {u'href': u'http://104.130.20.198:9511/containers/4a12301e-e0ba-45c2-964f-f07757deb146', u'rel': u'bookmark'}] |
| bay_uuid | 74a09f86-d3b1-4f75-aca5-f4e03a941ba6 |
| updated_at | None |
| image_id | centos:centos6 |
| command | sleep 365d |
| created_at | 2015-05-07T09:35:27+00:00 |
| name | demo |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
stack@ao-magnum-metal:~$ magnum container-list
ERROR: Bay 59b8e0ac-412e-44ab-b38b-8e31241ce821 could not be found. (HTTP 400)

stack@ao-magnum-metal:~$ magnum bay-list
+--------------------------------------+---------+------------+-----------------+
| uuid | name | node_count | status |
+--------------------------------------+---------+------------+-----------------+
| 74a09f86-d3b1-4f75-aca5-f4e03a941ba6 | beehive | 2 | CREATE_COMPLETE |
| d8c00ae1-b1b7-4ff6-9933-a09736c7d43a | testbay | 2 | CREATE_COMPLETE |
+--------------------------------------+---------+------------+-----------------+

I don't know why "Bay 59b8e0ac-412e-44ab-b38b-8e31241ce821" is being referenced when a "bay-list" command is run. I don't have a bay with that UUID, and the demo.json file does not reference this id either. What's happening?

[Adrian Otto (aotto)]

This is the contents of the containers table:

mysql> select * from container\G
*************************** 1. row ***************************
created_at: 2015-05-07 06:31:43
updated_at: NULL
        id: 1
      uuid: a80b4219-5ad9-44b2-bbea-136a21619df4
      name: demo
  image_id: centos:centos6
project_id: 5b093590f5d64ac0a84beeb8f1047ffc
   user_id: 5cc61d296fab44ff9103193a525119bd
   command: NULL
  bay_uuid: 59b8e0ac-412e-44ab-b38b-8e31241ce821
*************************** 2. row ***************************
created_at: 2015-05-07 09:33:56
updated_at: NULL
        id: 2
      uuid: a2b4212a-a6b3-4e95-b545-05ecc2cd5358
      name: demo
  image_id: centos:centos6
project_id: 5b093590f5d64ac0a84beeb8f1047ffc
   user_id: 5cc61d296fab44ff9103193a525119bd
   command: NULL
  bay_uuid: 74a09f86-d3b1-4f75-aca5-f4e03a941ba6
*************************** 3. row ***************************
created_at: 2015-05-07 09:35:27
updated_at: NULL
        id: 3
      uuid: 4a12301e-e0ba-45c2-964f-f07757deb146
      name: demo
  image_id: centos:centos6
project_id: 5b093590f5d64ac0a84beeb8f1047ffc
   user_id: 5cc61d296fab44ff9103193a525119bd
   command: sleep 365d
  bay_uuid: 74a09f86-d3b1-4f75-aca5-f4e03a941ba6
3 rows in set (0.00 sec)

My guess is that I had a container in a bay that I deleted, and the container is still there, but the bay is not. So perhaps the bug is that when bays are deleted, the containers in the pods they create are not deleted.

[Adrian Otto (aotto)]

Furthermore, if I try to delete that container, I get a BayNotFound error:

stack@ao-magnum-metal:~$ magnum container-delete a80b4219-5ad9-44b2-bbea-136a21619df4
Delete for container a80b4219-5ad9-44b2-bbea-136a21619df4 failed: Bay 59b8e0ac-412e-44ab-b38b-8e31241ce821 could not be found.
Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
    executor_callback))

  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
    executor_callback)

  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
    result = func(ctxt, **new_args)

  File "/opt/stack/magnum/magnum/conductor/handlers/docker_conductor.py", line 107, in container_delete
    docker = self.get_docker_client(context, container_uuid)

  File "/opt/stack/magnum/magnum/conductor/handlers/docker_conductor.py", line 84, in get_docker_client
    return cls._docker_for_container(context, container)

  File "/opt/stack/magnum/magnum/conductor/handlers/docker_conductor.py", line 77, in _docker_for_container
    bay = objects.Bay.get_by_uuid(context, container.bay_uuid)

  File "/opt/stack/magnum/magnum/objects/base.py", line 110, in wrapper
    result = fn(cls, context, *args, **kwargs)

  File "/opt/stack/magnum/magnum/objects/bay.py", line 106, in get_by_uuid
    db_bay = cls.dbapi.get_bay_by_uuid(context, uuid)

  File "/opt/stack/magnum/magnum/db/sqlalchemy/api.py", line 189, in get_bay_by_uuid
    raise exception.BayNotFound(bay=bay_uuid)

BayNotFound: Bay 59b8e0ac-412e-44ab-b38b-8e31241ce821 could not be found.
 (HTTP 404)

[Adrian Otto (aotto)]

I think this bug actually got fixed. Maybe the patch was not linked here, but this stopped happening last week. We should check for it in stable/kilo and trunk to be sure.

[Adrian Otto (aotto)]

More about the problem. This initially happened because when I deleted a swarm bay I did not delete the container in it first. That caused an orphaned entry in the containers table that was causing this symptom. The fix is to delete all containers in a bay when the bay is deleted to prevent this.

[Drago (dragorosson)]

Should this be updated to Invalid or Wont fix due to the /containers endpoint being removed?

[Madhuri Kumari (madhuri-rai07)]

This bug is invalid now as we have removed container endpoint from Magnum.