Deployment fails due to apt-get IPv6 failure

[Iryna Vovk]

Steps to reproduce:

1. Take ISO 323
2. Install ISO with slaves
3. Create cluster with ubuntu, neutron with Vlan segmentation
4. Add 1 node with controller role and one node with base-os role
5. All interfaces separatly
6. Start deploy prosedure

Actual result:
Deployment has failed. Method granular_deploy.
from log
Error: /Stage[main]/Main/L23network::L2::Patch[patch__br-aux--br-prv]/L2_patch[patch__br-aux--br-prv]: Provider ovs is not functional on this host

[Artem Panchenko (apanchenko-8)]

Looks like a duplicate of bug #1446187

[Stanislav Makar (smakar)]

In logs I can not find such kind of errors as
Error: /Stage[main]/Main/L23network::L2::Patch[patch__br-aux--br-prv]

I have find that the root cause of failed deployment was no connection to ubuntu repo, see below

2015-04-20 12:22:53 +0000 Puppet (err): Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install iptables-persistent' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  cloud-guest-utils eatmydata python-cheetah python-configobj
  python-json-pointer python-jsonpatch python-oauth python-serial
  python3-pycurl python3-software-properties software-properties-common
  unattended-upgrades
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
  iptables-persistent
0 upgraded, 1 newly installed, 0 to remove and 63 not upgraded.
Need to get 10.2 kB of archives.
After this operation, 62.5 kB of additional disk space will be used.
Err http://archive.ubuntu.com/ubuntu/ trusty/universe iptables-persistent all 0.5.7
  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::19). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::19 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/i/iptables-persistent/iptables-persistent_0.5.7_all.deb Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::19). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::19 80]

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

[Stanislav Makar (smakar)]

it is very wiered that it tried to connect using IPv6 address

[Vladimir Kuklin (vkuklin)]

It seems that we should apply Acquire::ForceIPv4 "true" to apt-get.conf

[Vladimir Kuklin (vkuklin)]

Lowering bug prio as it is intermittent and deployment can be retried to fix the problem.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/175799

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/175800

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergii Golovatiuk (<email address hidden>) on branch: master
Review: https://review.openstack.org/175799
Reason: Aleksandr Gordeev is going to release patch based on python implementation

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/175848

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/175800
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=2d5d37cbce6a3e262ffdd74e01aa1ac4af84dac2
Submitter: Jenkins
Branch: master

commit 2d5d37cbce6a3e262ffdd74e01aa1ac4af84dac2
Author: Sergii Golovatiuk <email address hidden>
Date: Tue Apr 21 12:22:39 2015 +0200

    Convince apt-get not to use IPv6 method

    Closes-Bug: 1446227
    Change-Id: Idc40f2ccf9fb8494b9b30c58775b38d9487ddcd4
    Signed-off-by: Sergii Golovatiuk <email address hidden>

[Oleksandr Martsyniuk (omartsyniuk)]

The actual reason of this error is that default route via mgmt network is not available on pre-deploy stage. Apt http method cannot connect via IPv4, so fallback to IPv6 is performed.
As a plugin developer, I propose not to add default route via controller VIP on base-os nodes, to let plugin pre-deploy tasks use the default route via admin node.

[Oleksandr Martsyniuk (omartsyniuk)]

This bug reproduces on Fuel 6.1 ISO build 327.
We need this to be resolved ASAP as it is a blocker for contrail plugin and also may affect other plugins which use base-os node role.

[Sergii Golovatiuk (sgolovatiuk)]

The networking behavior is correct. Default gateway is set directly via VIP on controllers allowing to remove master node and not to be single point of failure. Additionally, master node has direct connection with all nodes, so all packages can be obtained via http://IPv4_of_master_node/

[Sean M. Collins (scollins)]

Is it wise for us to completely disable IPv6 support in apt? This is going to have to be revisited when we start deploying dual-stack and v6 only deployments?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/175848
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=9dbdfd045d515148596666ae245692a55f1a11e3
Submitter: Jenkins
Branch: master

commit 9dbdfd045d515148596666ae245692a55f1a11e3
Author: Alexander Gordeev <email address hidden>
Date: Tue Apr 21 15:51:35 2015 +0300

    [IBP] Convince apt-get not to use IPv6 method

    Change-Id: I409e20e8e6246b90789f2454de2458f38bba90a4
    Closes-Bug: 1446227

[Anastasia Palkina (apalkina)]

Verified on ISO #471

"build_id": "2015-05-26_20-59-56", "build_number": "471", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-05-26_20-59-56", "build_number": "471", "api": "1.0", "fuel-library_sha": "8cfeca1a86179ebed1e4e03b2133b49c27350f6f", "nailgun_sha": "f737675091bd1903aace0e36812e855ce47dfec7", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "e19f1b65792f84c4a18b5a9473f85ef3ba172fce", "astute_sha": "0bd72c72369e743376864e8e8dabfe873d40450a", "fuel-ostf_sha": "87819878bc0ca572900e1f6933d9b99e666d6f62", "release": "6.1", "fuelmain_sha": "13b3e9cf074ba1cf1ae06509c55fbab613c73f4e"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "8cfeca1a86179ebed1e4e03b2133b49c27350f6f", "nailgun_sha": "f737675091bd1903aace0e36812e855ce47dfec7", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "e19f1b65792f84c4a18b5a9473f85ef3ba172fce", "astute_sha": "0bd72c72369e743376864e8e8dabfe873d40450a", "fuel-ostf_sha": "87819878bc0ca572900e1f6933d9b99e666d6f62", "release": "6.1", "fuelmain_sha": "13b3e9cf074ba1cf1ae06509c55fbab613c73f4e"