OpenStack-Ansible

os_cinder : Ensure cinder api is available timeout

Bug #1753125 reported by XiaoRuiguo
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
Critical
Jesse Pretorius

Bug Description

when run openstack-ansible setup-openstack.yml has errors:

TASK [os_cinder : Ensure cinder api is available] **************************************************************************************************************************
Saturday 03 March 2018 08:51:32 +0800 (0:00:00.438) 0:04:42.461 ********
FAILED - RETRYING: Ensure cinder api is available (10 retries left).
FAILED - RETRYING: Ensure cinder api is available (9 retries left).
FAILED - RETRYING: Ensure cinder api is available (8 retries left).
FAILED - RETRYING: Ensure cinder api is available (7 retries left).
FAILED - RETRYING: Ensure cinder api is available (6 retries left).
FAILED - RETRYING: Ensure cinder api is available (5 retries left).
FAILED - RETRYING: Ensure cinder api is available (4 retries left).
FAILED - RETRYING: Ensure cinder api is available (3 retries left).
FAILED - RETRYING: Ensure cinder api is available (2 retries left).
FAILED - RETRYING: Ensure cinder api is available (1 retries left).
fatal: [node226_cinder_api_container-9398c6d1]: FAILED! => {"attempts": 10, "cache_control": "no-cache", "changed": false, "connection": "close", "content": "<html><body><h1>
503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n", "content_type": "text/html", "msg": "Status code was not [200, 300]: HTTP Error 503:
 Service Unavailable", "redirected": false, "status": 503, "url": ...

See original description
Revision history for this message
Jean-Philippe Evrard (jean-philippe-evrard) wrote :

It looks like there is a problem in your deployment, could you give us more details? Like a log of your full play? Maybe your user variables/openstack_user_config?

Changed in openstack-ansible:
status: New → Incomplete
Revision history for this message
XiaoRuiguo (705437650-6) wrote :

problem like: https://serverfault.com/questions/491644/haproxy-responding-with-nosrv-while-backend-is-up

haproxy log:
[07/Mar/2018:22:37:14.736] cinder_api-front-2 cinder_api-back/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 45/0/0/0/0 0/0 "GET / HTTP/1.1"

ansible log:

fatal: [node225_cinder_api_container-20ab0a58]: FAILED! => {"attempts": 10, "cache_control": "no-cache", "changed": false, "connection": "close", "content": "<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n", "content_type": "text/html", "msg": "Status code was not [200, 300]: HTTP Error 503: Service Unavailable", "redirected": false, "status": 503, "url": "http://XX.XX.XX.XX:8776"}

Revision history for this message
XiaoRuiguo (705437650-6) wrote :

x.x.x.x is internalip

# curl -v http://x.x.x.x:8776
* Rebuilt URL to: http://x.x.x.x:8776/
* Trying x.x.x.x...
* Connected to x.x.x.x (x.x.x.x) port 8776 (#0)
> GET / HTTP/1.1
> Host: x.x.x.x:8776
> User-Agent: curl/7.47.0
> Accept: */*
>
 * HTTP 1.0, assume close after body
< HTTP/1.0 503 Service Unavailable
< Cache-Control: no-cache
< Connection: close
< Content-Type: text/html
<
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
* Closing connection 0

cinder-api.log

... ...
2018-03-08 20:00:36.985 35538 INFO cinder.api.openstack.wsgi [req-30118703-748b-455e-b9fc-278a1841abef - - - - -] HEAD http://node225-cinder-api-container-20ab0a58:8776/
2018-03-08 20:00:36.986 35538 INFO cinder.api.openstack.wsgi [req-30118703-748b-455e-b9fc-278a1841abef - - - - -] http://node225-cinder-api-container-20ab0a58:8776/ returned with HTTP 300

... ...

Revision history for this message
XiaoRuiguo (705437650-6) wrote :

other service is right but cinder.

Revision history for this message
XiaoRuiguo (705437650-6) wrote :

stable/queens

Revision history for this message
Fatih Degirmenci (fdegir) wrote :

We seem to be hitting this issue in OPNFV XCI as well.

Here are some details about the deployment.

OSA Version: eee0498dbe21fea565c06c686546c0734c4bb2e3
Ansible Role Requirements: https://gerrit.opnfv.org/gerrit/gitweb?p=releng-xci.git;a=blob;f=xci/installer/osa/files/ansible-role-requirements.yml
OpenStack Services: https://gerrit.opnfv.org/gerrit/gitweb?p=releng-xci.git;a=blob;f=xci/installer/osa/files/openstack_services.yml
Link to the log: http://paste.ubuntu.com/p/CjJ775yWcd/

Please let me know if you need further information.

Revision history for this message
XiaoRuiguo (705437650-6) wrote :

when I got stable/queen lastest commit yesterday.the problem is solved.

Changed in openstack-ansible:
status: Incomplete → Invalid
Revision history for this message
Fatih Degirmenci (fdegir) wrote :

We are still seeing the issue with the stable/queens rc2 with ha deployment. The failure is same.

log: https://hastebin.com/usegajuhit.sql
osa sha: ba77cee3e8ddc55644ab2195cea51554482f13a5 (http://git.openstack.org/cgit/openstack/openstack-ansible/commit/?h=17.0.0.0rc2&id=ba77cee3e8ddc55644ab2195cea51554482f13a5)
arr: https://gerrit.opnfv.org/gerrit/gitweb?p=releng-xci.git;hb=refs%2Fchanges%2F55%2F53655%2F4;f=xci%2Finstaller%2Fosa%2Ffiles%2Fansible-role-requirements.yml
openstack-services: https://gerrit.opnfv.org/gerrit/gitweb?p=releng-xci.git;hb=refs%2Fchanges%2F55%2F53655%2F4;f=xci%2Finstaller%2Fosa%2Ffiles%2Fopenstack_services.yml

Changed in openstack-ansible:
status: Invalid → New
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/552889

Changed in openstack-ansible:
assignee: nobody → Markos Chandras (hwoarang)
status: New → In Progress
OpenStack Infra (hudson-openstack)
Changed in openstack-ansible:
assignee: Markos Chandras (hwoarang) → Jimmy McCrory (jimmy-mccrory)
OpenStack Infra (hudson-openstack)
Changed in openstack-ansible:
assignee: Jimmy McCrory (jimmy-mccrory) → Jesse Pretorius (jesse-pretorius)
Revision history for this message
Jesse Pretorius (jesse-pretorius) wrote :

I've proposed an alternative to the previous approach which I think is simple, back portable and should solve the problem: https://review.openstack.org/556046

Revision history for this message
Jesse Pretorius (jesse-pretorius) wrote :

Given that this has been confirmed by multiple parties, and that it breaks a deployment, marking this as critical.

Changed in openstack-ansible:
importance: Undecided → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-os_cinder (master)

Reviewed: https://review.openstack.org/556046
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_cinder/commit/?id=76e0e8cfa0ac63d4aa5df1e135999b8488d04e0d
Submitter: Zuul
Branch: master

commit 76e0e8cfa0ac63d4aa5df1e135999b8488d04e0d
Author: Jesse Pretorius <email address hidden>
Date: Sat Mar 24 15:10:23 2018 +0000

    Execute cinder_backends on the last API server

    Currently the backend setup executes on the first cinder-api
    host. The playbook to execute the setup of the API is done
    in serial. This creates a bit of a problem on a greenfield
    deployment which uses haproxy because the backend is disabled
    and there are no other API hosts setup yet. This makes the
    CLI commands which require the API to be working fail.

    This patch makes the backend create task set execute on the
    last cinder-api host. This means that when they execute, there
    should already be other API services up and therefore the load
    balancer will be able to handle the API requests.

    Closes-Bug: #1753125
    Change-Id: I335b9f4ef0566d1561275059b90e0a50d16c4992

Changed in openstack-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_cinder (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/556052

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_cinder (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/556072

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_cinder (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/556073

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on openstack-ansible-os_cinder (stable/pike)

Change abandoned by Jesse Pretorius (odyssey4me) (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/556072
Reason: The group here used is cinder-volume, which is different to master/queens. Given that cinder-api is done before cinder-volume, this is unnecessary.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on openstack-ansible-os_cinder (stable/ocata)

Change abandoned by Jesse Pretorius (odyssey4me) (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/556073
Reason: The group here used is cinder-volume, which is different to master/queens. Given that cinder-api is done before cinder-volume, this is unnecessary.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-os_cinder (stable/queens)

Reviewed: https://review.openstack.org/556052
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_cinder/commit/?id=948305aa6bfeeb2abdda7351aa0a8ff292810e56
Submitter: Zuul
Branch: stable/queens

commit 948305aa6bfeeb2abdda7351aa0a8ff292810e56
Author: Jesse Pretorius <email address hidden>
Date: Sat Mar 24 15:10:23 2018 +0000

    Execute cinder_backends on the last API server

    Currently the backend setup executes on the first cinder-api
    host. The playbook to execute the setup of the API is done
    in serial. This creates a bit of a problem on a greenfield
    deployment which uses haproxy because the backend is disabled
    and there are no other API hosts setup yet. This makes the
    CLI commands which require the API to be working fail.

    This patch makes the backend create task set execute on the
    last cinder-api host. This means that when they execute, there
    should already be other API services up and therefore the load
    balancer will be able to handle the API requests.

    Closes-Bug: #1753125
    Change-Id: I335b9f4ef0566d1561275059b90e0a50d16c4992
    (cherry picked from commit 76e0e8cfa0ac63d4aa5df1e135999b8488d04e0d)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/556275

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible (stable/queens)

Reviewed: https://review.openstack.org/556275
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=f72f6925c107d8117da96aeb65f35d301c6078d8
Submitter: Zuul
Branch: stable/queens

commit f72f6925c107d8117da96aeb65f35d301c6078d8
Author: Jesse Pretorius <email address hidden>
Date: Sun Mar 25 18:59:16 2018 +0100

    Update cinder role SHA for cinder_backends fix

    Currently the backend setup executes on the first cinder-api
    host. The playbook to execute the setup of the API is done
    in serial. This creates a bit of a problem on a greenfield
    deployment which uses haproxy because the backend is disabled
    and there are no other API hosts setup yet. This makes the
    CLI commands which require the API to be working fail.

    Given that this is a critical bug, this patch updates the
    cinder role to ensure that the fix for this is included in
    the next release

    Change-Id: I23b966c68b491e51f33c89a1aa196a50c42fe3f4
    Closes-Bug: #1753125

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on openstack-ansible-os_cinder (master)

Change abandoned by Jesse Pretorius (odyssey4me) (<email address hidden>) on branch: master
Review: https://review.openstack.org/552889
Reason: It looks like this isn't necessary any more. Please restore it if you wish to continue with this approach.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible 17.0.1

This issue was fixed in the openstack/openstack-ansible 17.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-os_cinder 17.0.1

This issue was fixed in the openstack/openstack-ansible-os_cinder 17.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-os_cinder 18.0.0.0b1

This issue was fixed in the openstack/openstack-ansible-os_cinder 18.0.0.0b1 development milestone.

Kiluria (kiluria)
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.