Environment is not deleted after successful deployment

[Anastasia Palkina]

"build_id": "2014-07-28_02-01-14",
"ostf_sha": "8c328521b1444f22c50463b9432193e20ed33813",
"build_number": "366",
"auth_required": true,
"api": "1.0",
"nailgun_sha": "83cc9ed44ebc8dd97248483b6d414ebbc4cff3c0",
"production": "docker",
"fuelmain_sha": "9adfbf5a52cedbdd16ec1a74f6c44c5b3419b87c",
"astute_sha": "aa5aed61035a8dc4035ab1619a8bb540a7430a95",
"feature_groups": ["mirantis"],
"release": "5.1",
"fuellib_sha": "d1c7f67b3cf51978d3178c8666ea398f2477dcb5"

1. Create new environment (Ubuntu, simple mode)
2. Choose Ceilometer
3. Add controller+mongo, compute
4. Start deployment. It was successful
5. Delete environment. It is not deleted. No errors in logs.

Controller node-5, compute node-6

[Dima Shulyak (dshulyak)]

2014-07-28T12:50:26 debug: [384] Requeue message because worker is busy: "{\"args\": {\"engine\": {\"url\": \"http://10.20.0.2:80/cobbler_api\", \"username\": \"cobbler\", \"password\": \"2H88eG3P\", \"master_ip\": \"10.20.0.2\"}, \"task_uuid\": \"eac70a67-1451-4bb9-856b-baa481d877f3\", \"nodes\": [{\"slave_name\": \"node-5\", \"id\": 5, \"roles\": [\"controller\", \"mongo\"], \"uid\": 5}, {\"slave_name\": \"node-6\", \"id\": 6, \"roles\": [\"compute\"], \"uid\": 6}]}, \"respond_to\": \"remove_cluster_resp\", \"method\": \"remove_nodes\", \"api_version\": \"1.0\"}"

Message wasnt requeud

[Dima Shulyak (dshulyak)]

I've restarted astute with one worker, and can reproduce next bug quite consistently..

2014-07-28 15:28:35 DEBUG
[492] Successfully delete object at /api/queues/%2F/nailgun
2014-07-28 15:28:35 DEBUG
[492] Successfully delete object at /api/queues/%2F/naily
2014-07-28 15:28:35 DEBUG
[492] Successfully delete object at /api/exchanges/%2F/naily_service
2014-07-28 15:28:35 DEBUG
[492] Successfully delete object at /api/exchanges/%2F/nailgun
2014-07-28 15:28:35 WARNING
[492] Try to remove problem exchanges and queues
2014-07-28 15:28:33 DEBUG
[492] Requeue message because worker is busy: "{\"args\": {\"engine\": {\"url\": \"http://10.108.10.2:80/cobbler_api\", \"username\": \"cobbler\", \"password\": \"WQZpqiT6\", \"master_ip\": \"10.108.10.2\"}, \"task_uuid\": \"b72f2b10-5488-44df-a363-21e50f5460b9\", \"nodes\": [{\"slave_name\": \"node-1\", \"id\": 1, \"roles\": [\"controller\"], \"uid\": 1}]}, \"respond_to\": \"remove_cluster_resp\", \"method\": \"remove_nodes\", \"api_version\": \"1.0\"}"

[Dima Shulyak (dshulyak)]

For some reason we are receiving 406 PRECONDITION FAILURE each time astute tries to requeu message

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/110019

[Vladimir Sharshov (vsharshov)]

Best and fast solution provided by Dima Shulyak (dshulyak). If this patch will unlock this problem: https://bugs.launchpad.net/fuel/+bug/1340704, we should create another solution.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/110019
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=743ec9ecf3581ba1e99d8c8ffd2d171fc1c2be9d
Submitter: Jenkins
Branch: master

commit 743ec9ecf3581ba1e99d8c8ffd2d171fc1c2be9d
Author: Dima Shulyak <email address hidden>
Date: Mon Jul 28 18:29:59 2014 +0300

    Requeu and acknowledge logic for server

    - no_ack should not be used with requeu implementation at same time
    rejecting message when no_ack set leads to:
    PRECONDITION_FAILED - unknown delivery tag

    - metadata.ack will guarantee that message is delivered and started
    processing

    Change-Id: I15d940f635b1d0f80ac1eaefc8f37a3080e7f4d2
    Closes-Bug: 1349399