On host systems with IPv6 enabled, install fails at overcloud_prep_network step due to ipv6 dns server being used for ipv4 neutron network

Bug #1736264 reported by David Peacock
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Medium
Martin André

Bug Description

CentOS 7 host system, with IPv6 enabled
OpenStack environment: Latest upstream via latest tripleo-quickstart devmode.sh

DNS resolver set thus:

[dpeacock@gandalf ~]$ cat /etc/resolv.conf
# Generated by NetworkManager
search davidjpeacock.ca
nameserver 192.168.1.1
nameserver 2001:4860:4860:8888::
nameserver 2001:4860:4860:8844::

tripleo-quickstart kicked off against this host via:

`$ ./devmode.sh --no-gate gandalf`

Ansible failure information:

```
TASK [overcloud-prep-network : Create overcloud-prep-network script] ***********************************
task path: /home/dpeacock/.quickstart/usr/local/share/ansible/roles/overcloud-prep-network/tasks/create-scripts.yml:3
Monday 04 December 2017 16:28:43 -0500 (0:01:09.077) 1:06:14.707 *******
changed: [undercloud] => {"changed": true, "checksum": "ddd8237240fd5f099ca0ed618f8a3af9fda6e94a", "dest": "/home/stack/overcloud-prep-network.sh", "failed": false, "gid": 1001, "group": "stack", "md5sum": "95744bfdb86952408b7ead37d3a07ce1", "mode": "0755", "owner": "stack", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 1450, "src": "/home/stack/.ansible/tmp/ansible-tmp-1512422924.04-183330380951554/source", "state": "file", "uid": 1000}

TASK [overcloud-prep-network : Prepare the network-isolation required networks on the undercloud] ******
task path: /home/dpeacock/.quickstart/usr/local/share/ansible/roles/overcloud-prep-network/tasks/overcloud-prep-network.yml:1
Monday 04 December 2017 16:29:22 -0500 (0:00:38.915) 1:06:53.622 *******
fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "set -o pipefail && /home/stack/overcloud-prep-network.sh 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S |\"), $0; fflush(); }' > /home/stack/overcloud_prep_network.log", "delta": "0:00:10.712405", "end": "2017-12-04 21:29:57.486725", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2017-12-04 21:29:46.774320", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

PLAY RECAP *********************************************************************************************
gandalf : ok=156 changed=75 unreachable=0 failed=0
localhost : ok=17 changed=8 unreachable=0 failed=1
undercloud : ok=49 changed=34 unreachable=0 failed=1

```

Log of failure from undercloud virt:

```
[stack@undercloud ~]$ tail -20 overcloud_prep_network.log
2017-12-04 21:29:46 | DEVICETYPE=ovs
2017-12-04 21:29:46 | TYPE=OVSIntPort
2017-12-04 21:29:46 | BOOTPROTO=static
2017-12-04 21:29:46 | IPADDR=10.0.0.1
2017-12-04 21:29:46 | NETMASK=255.255.255.0
2017-12-04 21:29:46 | OVS_BRIDGE=br-ctlplane
2017-12-04 21:29:46 | OVS_OPTIONS="tag=10"
2017-12-04 21:29:46 | EOF'
2017-12-04 21:29:46 | + sudo ifup ifcfg-vlan10
2017-12-04 21:29:51 | ++ neutron net-list
2017-12-04 21:29:51 | ++ awk '/ctlplane/{print $(NF-2) }'
2017-12-04 21:29:51 | neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
2017-12-04 21:29:54 | ++ cat /etc/resolv.conf
2017-12-04 21:29:54 | ++ grep nameserver
2017-12-04 21:29:54 | ++ awk '{ print $2 }'
2017-12-04 21:29:54 | ++ sed ':a;N;$!ba;s/\n/ --dns-nameserver /g'
2017-12-04 21:29:54 | + neutron subnet-update f115ae1b-7c8a-43a3-b8ab-aee174d459f4 --dns-nameserver 192.168.1.1 --dns-nameserver 2001:4860:4860:8888::
2017-12-04 21:29:55 | neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
2017-12-04 21:29:57 | Invalid input for operation: dns_nameserver '2001:4860:4860:8888::' does not match the ip_version '4'.
2017-12-04 21:29:57 | Neutron server returns request_ids: ['req-b7b17c8a-9776-4eb9-ae98-d22a3395dd28']
```

Expected result:

Correct setup for DNS, and ultimate successful deployment.

Actual result:

Failure due to confusion of dns_nameserver not being an IPv4 address.

Tags: quickstart
Changed in tripleo:
status: New → Triaged
importance: Undecided → Medium
milestone: none → queens-3
summary: On host systems with IPv6 enabled, install fails at
- overcloud_prep_network step
+ overcloud_prep_network step due to ipv6 dns server being used for ipv4
+ neutron network
tags: added: quickstart
Revision history for this message
wes hayutin (weshayutin) wrote :

Are you attempting to deploy the overcloud using ipv6?
If you are passing the following config to devmode [1]

[1] https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset024.yml

Changed in tripleo:
status: Triaged → Incomplete
Revision history for this message
Emilien Macchi (emilienm) wrote :

Wes, I don't think he's trying to deploy TripleO with an ipv6 isolated network, rather on a host that has ipv6 enabled, which is a different case.
I'm re-opening the bug report.

Changed in tripleo:
status: Incomplete → Triaged
Revision history for this message
David Peacock (davidjpeacock) wrote :

Hi Wes,

My apologies for not being more clear; I'm actually attempting to deploy a regular tripleo-quickstart environment via the `devstack.sh` script, basically default vanilla style.

The IPv6 comes into play in that the host machine I am running this on has a dual-stack IPv4 and IPv6 setup.

```
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 44:39:c4:51:63:48 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.210/24 brd 192.168.1.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 2001:1970:4ba8:6500:5bdc:1c5:9b7a:8c3a/64 scope global noprefixroute dynamic
       valid_lft 65395sec preferred_lft 22195sec
    inet6 fe80::3d6c:e07f:8e24:866c/64 scope link
       valid_lft forever preferred_lft forever
```

Please let me know if there are any details, logs, commands, etc, you need me to provide.

Thank you!
David :-)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-quickstart-extras (master)

Fix proposed to branch: master
Review: https://review.openstack.org/534326

Changed in tripleo:
assignee: nobody → Martin André (mandre)
status: Triaged → In Progress
Revision history for this message
Martin André (mandre) wrote :

I faced the same issue today (which is odd because I had IPv6 enabled since always). https://review.openstack.org/#/c/534326/ should fix it.

Changed in tripleo:
milestone: queens-3 → queens-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-quickstart-extras (master)

Reviewed: https://review.openstack.org/534326
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=770ef97a9d36ed8dcc8da227af46fbc255b9b88b
Submitter: Zuul
Branch: master

commit 770ef97a9d36ed8dcc8da227af46fbc255b9b88b
Author: Martin André <email address hidden>
Date: Tue Jan 16 15:20:57 2018 +0100

    Restrict resolvers to IPv4 addresses

    Previously, the overcloud-prep-network.sh script would parse
    /etc/resolv.conf and add all resolvers found in the file to the ctlplane
    neutron subnet unless the overcloud_dns_servers variable is set.

    In the case the host machine has IPv6 enabled, this causes the neutron
    subnet-update command to fail on IPv6 addresses with:

      Invalid input for operation: dns_nameserver '2a01:e00::1' does not
      match the ip_version '4'.

    This commit filters out IPv6 addresses.

    Change-Id: I4f287015b6161b615eb95312b528d6a392a54d90
    Closes-Bug: #1736264

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-quickstart-extras 2.1.1

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.