Deployment fails to start: "Verification failed. failed to detect a valid IP address from u''"

[Alexei Sheplyakov]

Two nodes (controller + compute), no HA, Ubuntu 12.04, nova-network

Deployment stops immediately with the following error message:

Verification failed.
failed to detect a valid IP address from u''

nailgun/app.log contains:

2014-11-14 11:08:35.678 DEBUG [7fd5e4bfa740] (cluster) New pending changes in environment 1: attributes
[pid: 808|app: 0|req: 12/22] 172.17.42.1 () {48 vars in 906 bytes} [Fri Nov 14 11:08:35 2014] PUT /api/clusters/1/attributes => generated 18771 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 217 bytes (3 switches on core 0)
[pid: 809|app: 0|req: 11/23] 172.17.42.1 () {44 vars in 799 bytes} [Fri Nov 14 11:08:35 2014] GET /api/clusters/1?_=1415963273324 => generated 308 bytes in 5 msecs (HTTP/1.1 200) 5 headers in 217 bytes (2 switches on core 0)
[pid: 809|app: 0|req: 12/24] 172.17.42.1 () {44 vars in 805 bytes} [Fri Nov 14 11:08:38 2014] GET /api/notifications?_=1415963273326 => generated 328 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 217 bytes (2 switches on core 0)
[pid: 808|app: 0|req: 13/25] 172.17.42.1 () {44 vars in 823 bytes} [Fri Nov 14 11:08:38 2014] GET /api/nodes/allocation/stats?_=1415963273325 => generated 30 bytes in 16 msecs (HTTP/1.1 200) 5 headers in 217 bytes (2 switches on core 0)
2014-11-14 11:08:46.393 ERROR [7fd5e4bfa740] (manager) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nailgun/task/manager.py", line 70, in _call_silently
    to_return = method(task, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nailgun/task/task.py", line 561, in execute
    checker.check_configuration()
  File "/usr/lib/python2.6/site-packages/nailgun/network/checker.py", line 576, in check_configuration
    self.check_network_addresses_not_match_subnet_and_broadcast()
  File "/usr/lib/python2.6/site-packages/nailgun/network/checker.py", line 527, in check_network_addresses_not_match_subnet_and_broadcast
    ipr = netaddr.IPRange(r[0], r[1])
  File "/usr/lib/python2.6/site-packages/netaddr/ip/__init__.py", line 1344, in __init__
    self._start = IPAddress(start, flags=flags)
  File "/usr/lib/python2.6/site-packages/netaddr/ip/__init__.py", line 307, in __init__
    'address from %r' % addr)
AddrFormatError: failed to detect a valid IP address from u''

2014-11-14 11:08:46.394 DEBUG [7fd5e4bfa740] (task) Updating task: 23b1fb13-86ef-4edc-b0f8-ccc252f286e4
2014-11-14 11:08:46.395 DEBUG [7fd5e4bfa740] (task) Updating cluster status: 23b1fb13-86ef-4edc-b0f8-ccc252f286e4 cluster_id: 1 status: error

Note: the ISO has been built locally with the following commands:

make iso USE_MIRROR=none

[Dima Shulyak (dshulyak)]

Please provide logs

[Aleksey Kasatkin (alekseyk-ru)]

Seems that IP range for some network (Public, I guess) is left empty. Please provide your diagnostic snapshot or networking data.

[Alexei Sheplyakov (asheplyakov)]

2014-11-14 14:28:53.481 DEBUG [7f308956e740] (checker) network: {'name': 'admin (PXE)', 'ip_ranges': [[u'', u'']], 'gateway': None, 'vlan_start': None, 'meta': {u'notation': u'ip_ranges', u'render_type': None, u'assign_vip': False, u'configurable': False, u'unmovable': True, u'use_gateway': True, u'render_addr_mask': None, u'map_priority': 0}, 'release': None, 'cidr': u'10.20.0.0/24', 'group_id': None, 'id': 1}
2014-11-14 14:28:53.481 DEBUG [7f308956e740] (checker) r = [u'', u'']
2014-11-14 14:28:53.482 ERROR [7f308956e740] (manager) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nailgun/task/manager.py", line 70, in _call_silently
    to_return = method(task, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nailgun/task/task.py", line 561, in execute
    checker.check_configuration()
  File "/usr/lib/python2.6/site-packages/nailgun/network/checker.py", line 578, in check_configuration
    self.check_network_addresses_not_match_subnet_and_broadcast()
  File "/usr/lib/python2.6/site-packages/nailgun/network/checker.py", line 529, in check_network_addresses_not_match_subnet_and_broadcast
    ipr = netaddr.IPRange(r[0], r[1])
  File "/usr/lib/python2.6/site-packages/netaddr/ip/__init__.py", line 1344, in __init__
    self._start = IPAddress(start, flags=flags)
  File "/usr/lib/python2.6/site-packages/netaddr/ip/__init__.py", line 307, in __init__
    'address from %r' % addr)
AddrFormatError: failed to detect a valid IP address from u''

[Alexei Sheplyakov (asheplyakov)]

Nope, it's admin network which doesn't have ip range data (which is kind of expected).
Perhaps the check_network_addresses_not_match_subnet_and_broadcast should be skipped for this network since it's not user-configurable?

[Alexei Sheplyakov (asheplyakov)]

 Alternatively some special values of ip_ranges (such as [['', '']]) should be treated as an empty ip_ranges?

[Aleksey Kasatkin (alekseyk-ru)]

What is your Fuel version?
Actually, it is not expected that admin network does not have IP range data.

[Aleksey Kasatkin (alekseyk-ru)]

'meta': {u'notation': u'ip_ranges', ...}
- this means that IP range is required parameter.

AFAIK, admin IP range cannot be empty. It is an issue if this range is empty by default.

[Alexei Sheplyakov (asheplyakov)]

[root@fuel ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - experimental
  production: "docker"
  release: "6.0"
  api: "1.0"
  astute_sha: "0085021fe327f6f910901b3ca55051b1df33a96e"
  fuellib_sha: "8849db90ccac07169a33330acd0cfb2ccd172e10"
  ostf_sha: "82465a94eed4eff1fc8d8e1f2fb7e9993c22f068"
  nailgun_sha: "affd093f83b81e7969d78e80a615b4dbd956d314"
  fuelmain_sha: "e36aaace592dee94873a89b138ccf31489961e7f"

[Aleksey Kasatkin (alekseyk-ru)]

Thank you.

Nailgun cannot work properly without admin IP range data currently.

So, if it was empty by default - it is improper behaviour and should be fixed.
If it was cleared manually - Fuel do not support this case now.
You can file another bug with such request if you need this to be supported.

[Alexei Sheplyakov (asheplyakov)]

> If it was cleared manually - Fuel do not support this case now.

I haven't changed it, in fact, Fuel UI provides no way to specify it.

[Aleksey Kasatkin (alekseyk-ru)]

Right. But it can be changed via Fuel-menu. It should not be allowed to clear range though.

So, the issue can be connected with ISO building.
We should build the ISO following our docs and see the results.

What is your OS where you were building that?
Can you provide your ISO please? And ISO build logs may be?

[Alexei Sheplyakov (asheplyakov)]

http://172.18.66.101/Public/iso/fuel-6.0-e36aaace.iso

[Aleksey Kasatkin (alekseyk-ru)]

Reproduced with your ISO.

[Aleksey Kasatkin (alekseyk-ru)]

Your manifests are out of date:

/etc/puppet/2014.2-6.0/modules/nailgun/examples/nailgun-only.pp :
  admin_network_first => $::fuel_settings['ADMIN_NETWORK']['static_pool_start'],
  admin_network_last => $::fuel_settings['ADMIN_NETWORK']['static_pool_end'],

/etc/fuel/astute.yaml :
ADMIN_NETWORK:
  dhcp_pool_start: 10.20.0.3
  netmask: 255.255.255.0
  mac: 08:00:27:b1:51:d7
  dhcp_pool_end: 10.20.0.254
  interface: eth0
  ipaddress: 10.20.0.2