magnum throws db exception on failed bay deletion

[Steven Dake]

[sdake@bigiron kolla]$ magnum bay-list
magn+--------------------------------------+-------+------------+---------------+
| uuid | name | node_count | status |
+--------------------------------------+-------+------------+---------------+
| 78dc040a-c5e4-4bcc-8fc2-565fbb6069c0 | steak | 3 | CREATE_FAILED |
+--------------------------------------+-------+------------+---------------+
u[sdake@bigiron kolla]$ magnum bay-delete steak
Delete for bay steak failed: Remote error: OperationalError (OperationalError) no such table: bay u'SELECT bay.created_at AS bay_created_at, bay.updated_at AS bay_updated_at, bay.id AS bay_id, bay.project_id AS bay_project_id, bay.user_id AS bay_user_id, bay.uuid AS bay_uuid, bay.name AS bay_name, bay.baymodel_id AS bay_baymodel_id, bay.stack_id AS bay_stack_id, bay.api_address AS bay_api_address, bay.node_addresses AS bay_node_addresses, bay.node_count AS bay_node_count, bay.status AS bay_status, bay.discovery_url AS bay_discovery_url \nFROM bay \nWHERE bay.project_id = ? AND bay.uuid = ?' (u'2cd3341ff4ed49ceb0ba7fa2ed5d21ad', u'78dc040a-c5e4-4bcc-8fc2-565fbb6069c0')
[u' (HTTP 500)

then deleting the heat stack and bay delete:
[sdake@bigiron kolla]$ heat delete steak-lonk7cvtpo3a
WARNING (shell) DEPRECATED! Use stack-delete instead.
+--------------------------------------+--------------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+--------------------+--------------------+----------------------+
| e2ff24b6-3b56-4680-9ab2-3493c4ae7016 | steak-lonk7cvtpo3a | DELETE_IN_PROGRESS | 2015-05-03T01:34:15Z |
+--------------------------------------+--------------------+--------------------+----------------------+
[sdake@bigiron kolla]$ heat list
WARNING (shell) DEPRECATED! Use stack-list instead.
+----+------------+--------------+---------------+
| id | stack_name | stack_status | creation_time |
+----+------------+--------------+---------------+
+----+------------+--------------+---------------+
[sdake@bigiron kolla]$ magnum bay-list
magn+--------------------------------------+-------+------------+---------------+
| uuid | name | node_count | status |
+--------------------------------------+-------+------------+---------------+
| 78dc040a-c5e4-4bcc-8fc2-565fbb6069c0 | steak | 3 | CREATE_FAILED |
+--------------------------------------+-------+------------+---------------+
[sdake@bigiron kolla]$ magnum bay-delete steak
[sdake@bigiron kolla]$

deleting the heat stack manually gets rid of the exception.

[hongbin (hongbin034)]

I seem to encounter a similar error before. In my case, I did a 'git pull' without migrating the db schema. The problem was solved by running "magnum-db-manage upgrade" with a restart.

[Steven Dake (sdake)]

Hongbin,

I was running this in a container, and the database was properly migrated. The cause IIRC is one of the dependent services Cinder was unavailable and this triggered Heat to fail to create the stack. Trying to delete the bay afterwards resulted in this exception.

I would recommend trying to reproduce by killing cinder services in devstack, creating a baymodel, followed by a bay.

[Digambar (digambarpatil15)]

Hi Adrian, sdake

      I have tried reproducing this bug but did get the error -

      See below logs -

    digambar@Openstack_iot_diga:~$ magnum bay-list
              +--------------------------------------+----------+------------+---------------+
              | uuid | name | node_count | status |
              +--------------------------------------+----------+------------+---------------+
              | 3e830da5-120e-488e-8c47-3ff20cb8d897 | swarmbay | 2 | CREATE_FAILED |
             +--------------------------------------+----------+------------+---------------+

  digambar@Openstack_iot_diga:~$ magnum bay-delete swarmbay

  digambar@Openstack_iot_diga:~$ magnum bay-list
             +--------------------------------------+----------+------------+--------------------+
              | uuid | name | node_count | status |
             +--------------------------------------+----------+------------+--------------------+
             | 3e830da5-120e-488e-8c47-3ff20cb8d897 | swarmbay | 2 | DELETE_IN_PROGRESS |
            +--------------------------------------+----------+------------+--------------------+

after deletion process, when I did bay-list then all the bays are deleted which are in failed state.

  digambar@Openstack_iot_diga:~$ magnum bay-list
             +------+------+------------+--------+
             | uuid | name | node_count | status |
             +------+------+------------+--------+
             +------+------+------------+--------+

sdake, I have tried this without stoping the cinder service in devstack.

[Digambar (digambarpatil15)]

I am able to delete the failed bay without deleting the heat stack.

[Murali Allada (murali-allada)]

We need to test and see if this still occurs.

[Stephen Watson (stephen-watson)]

I could not reproduce by completing the following steps:

1) Stop c-api service via screen
2) Create a new K8s cluster-template:
magnum cluster-template-create --name k8s-bug-template \
    --image-id fedora-atomic-latest \
    --keypair-id magnum_key \
    --external-network-id public \
    --dns-nameserver 8.8.8.8 \
    --flavor-id m1.small \
    --docker-volume-size 5 \
    --coe kubernetes
3) Create a new K8s cluster using previous template:
magnum cluster-create --name k8s-bug-cluster \
    --cluster-template k8s-bug-template \
    --node-count 1
4) Verify that cluster failed to create:
swatson1@swatson1-linux:/opt/stack/magnum$ magnum cluster-list
+--------------------------------------+-----------------+------------+--------------+--------------------+
| uuid | name | node_count | master_count | status |
+--------------------------------------+-----------------+------------+--------------+--------------------+
| 76a1aa9d-33bf-45d1-aefa-c6e9ccd2d7cf | k8s-cluster | 1 | 1 | CREATE_COMPLETE |
| 34987730-2e82-4cb1-a90e-5df054e2e2ff | k8s-bug-cluster | 1 | 1 | CREATE_IN_PROGRESS |
+--------------------------------------+-----------------+------------+--------------+--------------------+
swatson1@swatson1-linux:/opt/stack/magnum$ magnum cluster-list
+--------------------------------------+-----------------+------------+--------------+-----------------+
| uuid | name | node_count | master_count | status |
+--------------------------------------+-----------------+------------+--------------+-----------------+
| 76a1aa9d-33bf-45d1-aefa-c6e9ccd2d7cf | k8s-cluster | 1 | 1 | CREATE_COMPLETE |
| 34987730-2e82-4cb1-a90e-5df054e2e2ff | k8s-bug-cluster | 1 | 1 | CREATE_FAILED |
+--------------------------------------+-----------------+------------+--------------+-----------------+
5) Delete cluster and see if exception is thrown:
swatson1@swatson1-linux:/opt/stack/magnum$ magnum cluster-delete k8s-bug-cluster
Request to delete cluster k8s-bug-cluster has been accepted.
swatson1@swatson1-linux:/opt/stack/magnum$ magnum cluster-list
+--------------------------------------+-------------+------------+--------------+-----------------+
| uuid | name | node_count | master_count | status |
+--------------------------------------+-------------+------------+--------------+-----------------+
| 76a1aa9d-33bf-45d1-aefa-c6e9ccd2d7cf | k8s-cluster | 1 | 1 | CREATE_COMPLETE |
+--------------------------------------+-------------+------------+--------------+-----------------+

This was on the latest from master, not from mitaka-1.

[Vijendar Komalla (vijendar-komalla)]

I am not able to reproduce the issue, so marking it as 'Fix Released'. Please feel free to activate this bug or create a new bug if you still see this issue.