Fuel for OpenStack

[systest] We should wait until Nailgun is become ready after resetting master node

Bug #1368537 reported by Artem Panchenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Artem Panchenko
Fuel for OpenStack 5.1
5.1.x
Fix Released
High
Artem Panchenko
Fuel for OpenStack 5.1
6.0.x
Fix Released
High
Fuel QA Team
Fuel for OpenStack 6.0

Bug Description

In system tests we destroy master node virtual machine and then start it if it's not accessible after snapshot revert:

https://github.com/stackforge/fuel-main/blob/master/fuelweb_test/models/environment.py#L355

But then test often fails with '502 error':

http://jenkins:8080/view/0_master_swarm/job/master_fuelmain.system_test.centos.upgrade/33/testReport/junit/%28root%29/upgrade_ha_env/upgrade_ha_env/

Because nailgun isn't ready yet (but nginx container is started already):

(venv-nailgun-tests)apanchenko@srv07-srt:~$ virsh destroy env_base_5_0_CentOS_admin
Domain env_base_5_0_CentOS_admin destroyed

(venv-nailgun-tests)apanchenko@srv07-srt:~$ virsh start env_base_5_0_CentOS_admin
Domain env_base_5_0_CentOS_admin started

(venv-nailgun-tests)apanchenko@srv07-srt:~$ curl 10.108.20.2:8000
curl: (7) Failed to connect to 10.108.20.2 port 8000: Connection refused
(venv-nailgun-tests)apanchenko@srv07-srt:~$ curl 10.108.20.2:8000
curl: (7) Failed to connect to 10.108.20.2 port 8000: Connection refused
(venv-nailgun-tests)apanchenko@srv07-srt:~$ curl 10.108.20.2:8000
curl: (7) Failed to connect to 10.108.20.2 port 8000: Connection refused
(venv-nailgun-tests)apanchenko@srv07-srt:~$ curl 10.108.20.2:8000
curl: (7) Failed to connect to 10.108.20.2 port 8000: Connection refused
(venv-nailgun-tests)apanchenko@srv07-srt:~$ curl 10.108.20.2:8000
curl: (7) Failed to connect to 10.108.20.2 port 8000: Connection refused
(venv-nailgun-tests)apanchenko@srv07-srt:~$ curl 10.108.20.2:8000
curl: (7) Failed to connect to 10.108.20.2 port 8000: Connection refused
(venv-nailgun-tests)apanchenko@srv07-srt:~$ curl 10.108.20.2:8000
curl: (7) Failed to connect to 10.108.20.2 port 8000: Connection refused
(venv-nailgun-tests)apanchenko@srv07-srt:~$ curl 10.108.20.2:8000
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.0.15</center>
</body>
</html>

We should wait some time until Nailgun's API becomes available.

Tags: system-tests
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-main (master)

Fix proposed to branch: master
Review: https://review.openstack.org/120985

Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Artem Panchenko (apanchenko-8)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-main (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/120986

Revision history for this message
Aleksey Kasatkin (alekseyk-ru) wrote :

Doesn't it connected with availability of keystone container? It usually starts several minutes later.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-main (master)

Reviewed: https://review.openstack.org/120985
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=25d6f2e2fb4cab8328c14134ebf2c895cec96190
Submitter: Jenkins
Branch: master

commit 25d6f2e2fb4cab8328c14134ebf2c895cec96190
Author: Artem Panchenko <email address hidden>
Date: Fri Sep 12 09:07:37 2014 +0300

    Wait until Nailgun's API is ready after VM reset

    After resetting Fuel master node we wait until 8000/tcp
    port (nginx) is accepting connections. We also should wait
    up to 2 minutes until Nailgun's API is ready.

    Change-Id: I2360e0dab68ae0a20618f1da64ee5d6b4499e126
    Closes-bug: #1368537

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-main (stable/5.1)

Reviewed: https://review.openstack.org/120986
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=cb7d12287efff8142696615ff423822e97506467
Submitter: Jenkins
Branch: stable/5.1

commit cb7d12287efff8142696615ff423822e97506467
Author: Artem Panchenko <email address hidden>
Date: Fri Sep 12 09:07:37 2014 +0300

    Wait until Nailgun's API is ready after VM reset

    After resetting Fuel master node we wait until 8000/tcp
    port (nginx) is accepting connections. We also should wait
    up to 2 minutes until Nailgun's API is ready.

    Change-Id: I2360e0dab68ae0a20618f1da64ee5d6b4499e126
    Closes-bug: #1368537

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.