Fuel Web does not start, if change subnet

Bug #1285573 reported by Vadim Rovachev on 2014-02-27
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Medium
Matthew Mosesohn

Bug Description

{"build_id": "2014-02-26_17-35-21", "mirantis": "yes", "build_number": "214", "nailgun_sha": "ea08cef3e06a72f47cfaa8cd8fe6d034e2cf722e", "ostf_sha": "8e6681b6d06c7cb20a84c1cc740d5f2492fb9d85", "fuelmain_sha": "6e6f6b31b8880b7600a1af5cc94d48ac01d60a07", "astute_sha": "10cccc87f2ee35510e43c8fa19d2bf916ca1fced", "release": "4.1", "fuellib_sha": "0a2e5bdc01c1e3bb285acb7b39125101e950ac72"}

Steps to reproduce:
1. Start KVM with fuel ISO.
2. Await Fuel Setup.
http://picpaste.com/01-wZSNZlzD.png
Change IP address -> 10.20.1.2 and GW -> 10.20.1.1
click < Apply >
Await 20 sec
http://picpaste.com/02-7VNCh3B6.png
3. Go to "Quit Setup"
Click "Sava and Quit"
4. Await end of install
http://picpaste.com/04-yfgeFLSV.png

5.
Open WebUI(10.20.1.2:8000) in browser
Expected result: openstak WebUI
Actual result: 502 Bad Gateway

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

curl http://10.20.1.2:8000/api/version

Expected:
{"build_id": "2014-02-26_17-35-21", "mirantis": "yes", "build_number": "214", "nailgun_sha": "ea08cef3e06a72f47cfaa8cd8fe6d034e2cf722e", "ostf_sha": "8e6681b6d06c7cb20a84c1cc740d5f2492fb9d85", "fuelmain_sha": "6e6f6b31b8880b7600a1af5cc94d48ac01d60a07", "astute_sha": "10cccc87f2ee35510e43c8fa19d2bf916ca1fced", "release": "4.1", "fuellib_sha": "0a2e5bdc01c1e3bb285acb7b39125101e950ac72"}

Actual:
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.0.15</center>
</body>
</html>

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[root@fuel ~]# grep -r 'error' /var/log/
/var/log/puppet/bootstrap_admin_node.log:notice: /Stage[main]/Nailgun::Cobbler/File[/var/www/cobbler/aux/20-puppet_error_syslog.conf]/ensure: defined content as '{md5}e2436ac978b00d6f8b4b278e7bb53771'
/var/log/anaconda.syslog:08:24:08,667 WARNING NetworkManager: <warn> error requesting auth for org.freedesktop.NetworkManager.enable-disable-network: (26) Remote Exception invoking org.freedesktop.PolicyKit1.Authority.CheckAuthorization() on /org/freedesktop/PolicyKit1/Authority at name org.freedesktop.PolicyKit1: org.freedesktop.DBus.Error.Spawn.ExecFailed: Cannot launch daemon, file not found or permissions invalid
/var/log/anaconda.syslog:08:24:08,667 WARNING NetworkManager: <warn> error requesting auth for org.freedesktop.NetworkManager.sleep-wake: (26) Remote Exception invoking org.freedesktop.PolicyKit1.Authority.CheckAuthorization() on /org/freedesktop/PolicyKit1/Authority at name org.freedesktop.PolicyKit1: org.freedesktop.DBus.Error.Spawn.ExecFailed: Cannot launch daemon, file not found or permissions invalid
/var/log/anaconda.syslog:08:24:08,667 WARNING NetworkManager: <warn> error requesting auth for org.freedesktop.NetworkManager.enable-disable-wifi: (26) Remote Exception invoking org.freedesktop.PolicyKit1.Authority.CheckAuthorization() on /org/freedesktop/PolicyKit1/Authority at name org.freedesktop.PolicyKit1: org.freedesktop.DBus.Error.Spawn.ExecFailed: Cannot launch daemon, file not found or permissions invalid
/var/log/anaconda.syslog:08:24:08,667 WARNING NetworkManager: <warn> error requesting auth for org.freedesktop.NetworkManager.enable-disable-wwan: (26) Remote Exception invoking org.freedesktop.PolicyKit1.Authority.CheckAuthorization() on /org/freedesktop/PolicyKit1/Authority at name org.freedesktop.PolicyKit1: org.freedesktop.DBus.Error.Spawn.ExecFailed: Cannot launch daemon, file not found or permissions invalid
/var/log/anaconda.syslog:08:24:08,667 WARNING NetworkManager: <warn> error requesting auth for org.freedesktop.NetworkManager.use-user-connections: (26) Remote Exception invoking org.freedesktop.PolicyKit1.Authority.CheckAuthorization() on /org/freedesktop/PolicyKit1/Authority at name org.freedesktop.PolicyKit1: org.freedesktop.DBus.Error.Spawn.ExecFailed: Cannot launch daemon, file not found or permissions invalid
/var/log/anaconda.syslog:08:24:08,667 WARNING NetworkManager: <warn> error requesting auth for org.freedesktop.NetworkManager.network-control: (26) Remote Exception invoking org.freedesktop.PolicyKit1.Authority.CheckAuthorization() on /org/freedesktop/PolicyKit1/Authority at name org.freedesktop.PolicyKit1: org.freedesktop.DBus.Error.Spawn.ExecFailed: Cannot launch daemon, file not found or permissions invalid
/var/log/anaconda.syslog:08:24:08,667 WARNING NetworkManager: <warn> error requesting auth for org.freedesktop.network-manager-settings.system.wifi.share.open: (26) Remote Exception invoking org.freedesktop.PolicyKit1.Authority.CheckAuthorization() on /org/freedesktop/PolicyKit1/Authority at name org.freedesktop.PolicyKit1: org.freedesktop.DBus.Error.Spawn.ExecFailed: Cannot launch daemon, file not found or permissions invalid
/var/log/anaconda.syslog:08:24:08,667 WARNING NetworkManager: <warn> error requesting auth for org.freedesktop.network-manager-settings.system.wifi.share.protected: (26) Remote Exception invoking org.freedesktop.PolicyKit1.Authority.CheckAuthorization() on /org/freedesktop/PolicyKit1/Authority at name org.freedesktop.PolicyKit1: org.freedesktop.DBus.Error.Spawn.ExecFailed: Cannot launch daemon, file not found or permissions invalid
/var/log/anaconda.yum.log:[2014-02-27 08:26:30,787] DEBUG : TSINFO: Marking libgpg-error-1.7-4.el6.x86_64 as install for gpgme-1.1.8-3.el6.x86_64
/var/log/fuelmenu.log:You may see errors during provisioning and in system logs. NTP errors are not fatal.
/var/log/fuelmenu.log:2014-02-27 08:50:20,711 INFO No errors found
/var/log/mcollective.log:E, [2014-02-27T09:04:49.259475 #2239] ERROR -- : rabbitmq.rb:30:in `on_miscerr' Unexpected error on connection stomp://mcollective@localhost:61613: es_recv: connection.receive returning EOF as nil - resetting connection.
/var/log/nginx/error_repo.log:2014/02/27 09:14:25 [error] 7597#0: *12 open() "/var/www/nailgun/favicon.ico" failed (2: No such file or directory), client: 10.20.1.200, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "10.20.1.2:8080"
/var/log/nginx/error_nailgun.log:2014/02/27 09:10:01 [error] 7597#0: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:11:04 [error] 7597#0: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:12:35 [error] 7597#0: *6 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:13:22 [error] 7597#0: *8 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:14:09 [error] 7597#0: *10 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:14:29 [error] 7597#0: *14 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.200, server: localhost, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8001/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:15:10 [error] 7597#0: *21 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:16:20 [error] 7597#0: *23 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:17:31 [error] 7597#0: *25 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:18:28 [error] 7597#0: *27 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:19:22 [error] 7597#0: *29 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:20:04 [error] 7597#0: *31 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:21:09 [error] 7597#0: *33 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:22:24 [error] 7597#0: *35 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:23:04 [error] 7597#0: *37 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:24:22 [error] 7597#0: *39 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:25:25 [error] 7597#0: *41 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:26:31 [error] 7597#0: *43 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:27:15 [error] 7597#0: *45 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:28:12 [error] 7597#0: *47 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:29:16 [error] 7597#0: *49 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:30:15 [error] 7597#0: *51 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:31:03 [error] 7597#0: *53 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:32:29 [error] 7597#0: *55 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:33:20 [error] 7597#0: *57 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:34:04 [error] 7597#0: *59 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:35:06 [error] 7597#0: *61 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:36:28 [error] 7597#0: *63 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:37:29 [error] 7597#0: *67 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:38:16 [error] 7597#0: *69 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:38:55 [error] 7597#0: *71 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.200, server: localhost, request: "GET /api/version HTTP/1.1", upstream: "http://127.0.0.1:8001/api/version", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:39:08 [error] 7597#0: *73 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:40:12 [error] 7597#0: *75 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:41:23 [error] 7597#0: *77 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/nginx/error_nailgun.log:2014/02/27 09:42:15 [error] 7597#0: *79 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.1.5, server: localhost, request: "POST /api/nodes/ HTTP/1.1", upstream: "http://127.0.0.1:8001/api/nodes/", host: "10.20.1.2:8000"
/var/log/rabbitmq/startup_log:starting error log relay ...done
/<email address hidden>:STOMP error frame sent:

description: updated
Changed in fuel:
assignee: nobody → Matthew Mosesohn (raytrac3r)
Matthew Mosesohn (raytrac3r) wrote :

Vadim, can you post your /etc/astute.yaml on Fuel master for me? The only thing I would suspect is if you change the master IP address, but skip viewing the PXE Setup pane, it might write the wrong info in /etc/astute.yaml.

Changed in fuel:
milestone: none → 5.0
milestone: 5.0 → 4.1.1
Vadim Rovachev (vrovachev) wrote :

/etc/astute.yaml

HOSTNAME: fuel
DNS_DOMAIN: domain.tld
DNS_SEARCH: domain.tld
DNS_UPSTREAM: 8.8.8.8
NTP1: 0.pool.ntp.org
NTP2: 1.pool.ntp.org
NTP3: 2.pool.ntp.org
ADMIN_NETWORK:
  interface: eth0
  ipaddress: 10.20.1.2
  netmask: 255.255.255.0
  cidr: 10.20.0.0/24
  size: '256'
  static_pool_start: 10.20.1.4
  static_pool_end: 10.20.1.128
  dhcp_pool_start: 10.20.1.129
  dhcp_pool_end: 10.20.1.254
TEST_DNS: www.google.com

.
In
{"build_id": "2014-02-27_19-37-46", "mirantis": "yes", "build_number": "224", "nailgun_sha": "12a7e7a99557f2bc302f0806ad3beef02e94b974", "ostf_sha": "8e6681b6d06c7cb20a84c1cc740d5f2492fb9d85", "fuelmain_sha": "ba019bf15a9597a154e7c1d6ecc840614d21414c", "astute_sha": "10cccc87f2ee35510e43c8fa19d2bf916ca1fced", "release": "4.1", "fuellib_sha": "61d3a150402da3ce1160836c8d659f6d9d1f9640"}
bug is not reproduced

Evgeniy L (rustyrobot) on 2014-02-27
Changed in fuel:
status: New → Incomplete
importance: Undecided → Medium
Matthew Mosesohn (raytrac3r) wrote :

I couldn't reproduce this on 4.1 RC3 ISO using the same steps as reporter. When changing IP and gateway to a different network and then finishing deployment, I can still access Fuel UI with no issues.

Changed in fuel:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers