[contrail] Unable to connect to mirror.fuel-infra.org

[Iryna Vovk]

 Scenario:
            1. Revert snapshot "ready_with_5_slaves"
            2. Create cluster
            3. Add 3 nodes with contrail-config, contrail-control, contrail-db
               roles, 1 node with controller role
               and 1 node with compute + cinder role
            4. Enable Contrail plugin
            5. Deploy cluster with plugin
            6. Create net and subnet
            7. Run OSTF tests

Actual result:

Logs are located here:
https://jenkins-tpi.bud.mirantis.net/job/7.0.contrail.3.0.contrail_bvt.juniper/27/console

2015-12-21 19:45:06,313 - INFO fuel_web_client.py:1194 -- Task finished. Took 1444 seconds. {
 "status": "error",
 "name": "deploy",
 "cluster": 1,
 "result": {},
 "progress": 100,
 "message": "Deployment has failed. Method granular_deploy. Deployment failed on nodes 7.\nInspect Astute logs for the details",
 "id": 2,
 "uuid": "05df2476-b834-4650-be37-690980b5bb71"

Traceback (most recent call last):
  File "/home/jenkins/70-venv/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/70-venv/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/70-venv/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/7.0.contrail.3.0.contrail_bvt.juniper/plugin_test/fuel-qa/fuelweb_test/helpers/decorators.py", line 80, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.contrail.3.0.contrail_bvt.juniper/plugin_test/tests/test_smoke_bvt.py", line 140, in contrail_bvt
    openstack.deploy_cluster(self)
  File "/home/jenkins/workspace/7.0.contrail.3.0.contrail_bvt.juniper/plugin_test/helpers/openstack.py", line 51, in deploy_cluster
    obj.env.d_env.get_ssh_to_remote(n['ip']))
  File "/home/jenkins/workspace/7.0.contrail.3.0.contrail_bvt.juniper/plugin_test/fuel-qa/fuelweb_test/helpers/checkers.py", line 1046, in check_repo_managment
    remote.check_call(cmd)
  File "/home/jenkins/70-venv/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 282, in check_call
    ret['stdout'] + ret['stderr'])
DevopsCalledProcessError: Command 'apt-get clean all && apt-get update > /dev/null' returned non-zero exit status 100
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty/Release.gpg Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty-updates/Release.gpg Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty/main/i18n/Translation-en_US Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty/main/i18n/Translation-en Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty/multiverse/i18n/Translation-en_US Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty/multiverse/i18n/Translation-en Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty/universe/i18n/Translation-en_US Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty/universe/i18n/Translation-en Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty-updates/main/i18n/Translation-en_US Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty-updates/main/i18n/Translation-en Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty-updates/multiverse/i18n/Translation-en_US Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty-updates/multiverse/i18n/Translation-en Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty-updates/universe/i18n/Translation-en_US Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty-updates/universe/i18n/Translation-en Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty/main/binary-amd64/Packages Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty/universe/binary-amd64/Packages Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty/multiverse/binary-amd64/Packages Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty-updates/main/binary-amd64/Packages Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty-updates/universe/binary-amd64/Packages Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/dists/trusty-updates/multiverse/binary-amd64/Packages Unable to connect to mirror.seed-cz1.fuel-infra.org:http:

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-holdback/Release.gpg Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-security/Release.gpg Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-updates/Release.gpg Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-holdback/main/i18n/Translation-en_US Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-holdback/main/i18n/Translation-en Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-holdback/restricted/i18n/Translation-en_US Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-holdback/restricted/i18n/Translation-en Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-security/main/i18n/Translation-en_US Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-security/main/i18n/Translation-en Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-security/restricted/i18n/Translation-en_US Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-security/restricted/i18n/Translation-en Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-updates/main/i18n/Translation-en_US Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-updates/main/i18n/Translation-en Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-updates/restricted/i18n/Translation-en_US Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-updates/restricted/i18n/Translation-en Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-holdback/main/binary-amd64/Packages Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-holdback/restricted/binary-amd64/Packages Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-security/main/binary-amd64/Packages Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-security/restricted/binary-amd64/Packages Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-updates/main/binary-amd64/Packages Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-updates/restricted/binary-amd64/Packages Unable to connect to mirror.fuel-infra.org:http: [IP: 5.43.231.47 80]

E: Some index files failed to download. They have been ignored, or old ones used instead.

[Ruslan Khozinov (rkhozinov)]

I've compared mirror of dvs plugin (fuel7.0) and mirror from this issue.
You can see this job http://jenkins-tpi.bud.mirantis.net:8080/view/7.0.dvs/job/7.0.dvs.dvs_vcenter_smoke/71/console, for example.

So, this issue is only for contrail version for fuel7.0.

I'll try to reproduce it.

[Ruslan Khozinov (rkhozinov)]

I've reproduced it.
It seems that it's a fuel env's internal network issue.

Some details from the env:
[root@nailgun ~]# fuel nodes
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|-------------|---------------------------|---------|-------------|-------------------|------------------|---------------|--------|---------
8 | provisioned | slave-07_compute_cinder | 1 | 10.109.0.9 | 64:85:e9:65:de:5c | cinder, compute | | True | 1
6 | discover | Untitled (2d:b0) | None | 10.109.0.10 | 64:7b:55:fb:2d:b0 | | | True | None
7 | discover | Untitled (71:f0) | None | 10.109.0.11 | 64:7a:21:e8:71:f0 | | | True | None
4 | provisioned | slave-01_contrail-config | 1 | 10.109.0.3 | 64:c5:44:6b:a3:b2 | contrail-config | | True | 1
2 | provisioned | slave-03_contrail-db | 1 | 10.109.0.5 | 64:9f:32:d1:11:25 | contrail-db | | True | 1
1 | provisioned | slave-05_contrail-db | 1 | 10.109.0.7 | 64:8e:f7:00:12:c2 | contrail-db | | True | 1
9 | error | slave-06_controller | 1 | 10.109.0.8 | 64:6c:23:88:d2:ca | controller | | True | 1
5 | provisioned | slave-02_contrail-control | 1 | 10.109.0.4 | 64:c8:f8:5f:95:cf | contrail-control | | True | 1
3 | provisioned | slave-04_contrail-db | 1 | 10.109.0.6 | 64:e0:3c:30:2b:a3 | contrail-db | | True | 1

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.109.1.1 0.0.0.0 UG 0 0 0 br-ex
10.109.0.0 0.0.0.0 255.255.255.0 U 0 0 0 br-fw-admin
10.109.1.0 0.0.0.0 255.255.255.224 U 0 0 0 br-ex
10.109.1.32 0.0.0.0 255.255.255.224 U 0 0 0 br-mgmt
10.109.1.64 0.0.0.0 255.255.255.224 U 0 0 0 br-storage
10.109.1.96 0.0.0.0 255.255.255.224 U 0 0 0 br-mesh
169.254.169.254 - 255.255.255.255 !H 0 - 0 -
root@node-9:~# cat /etc/resolv.conf
# re-generated by cloud-init boothook only at the first boot;
search test.domain.local
domain test.domain.local
nameserver 10.109.0.2
root@node-9:~# ping 10.109.0.2
PING 10.109.0.2 (10.109.0.2) 56(84) bytes of data.
64 bytes from 10.109.0.2: icmp_seq=1 ttl=64 time=0.416 ms
64 bytes from 10.109.0.2: icmp_seq=2 ttl=64 time=3.11 ms
^C
--- 10.109.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.416/1.764/3.113/1.349 ms
root@node-9:~# ping ya.ru
PING ya.ru (213.180.204.3) 56(84) bytes of da...

[Ruslan Khozinov (rkhozinov)]

Internet access is on all nodes except controller. So, it's not a CI bug.

[root@nailgun ~]# for host in {3..11}; do ssh 10.109.0.$host "ping -c 1 ya.ru" 2>/dev/null; done
PING ya.ru (93.158.134.3) 56(84) bytes of data.
64 bytes from www.yandex.ru (93.158.134.3): icmp_seq=1 ttl=53 time=62.1 ms

--- ya.ru ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 62.164/62.164/62.164/0.000 ms
PING ya.ru (213.180.193.3) 56(84) bytes of data.
64 bytes from www.yandex.ru (213.180.193.3): icmp_seq=1 ttl=53 time=62.4 ms

--- ya.ru ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 62.426/62.426/62.426/0.000 ms
PING ya.ru (213.180.204.3) 56(84) bytes of data.
64 bytes from www.yandex.ru (213.180.204.3): icmp_seq=1 ttl=53 time=63.0 ms

--- ya.ru ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 63.077/63.077/63.077/0.000 ms
PING ya.ru (93.158.134.3) 56(84) bytes of data.
64 bytes from www.yandex.ru (93.158.134.3): icmp_seq=1 ttl=53 time=66.0 ms

--- ya.ru ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 66.030/66.030/66.030/0.000 ms
PING ya.ru (213.180.193.3) 56(84) bytes of data.
64 bytes from www.yandex.ru (213.180.193.3): icmp_seq=1 ttl=53 time=62.2 ms

--- ya.ru ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 62.255/62.255/62.255/0.000 ms
PING ya.ru (213.180.204.3) 56(84) bytes of data.
From 10.109.1.4 icmp_seq=1 Destination Host Unreachable

--- ya.ru ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

PING ya.ru (93.158.134.3) 56(84) bytes of data.
64 bytes from www.yandex.ru (93.158.134.3): icmp_seq=1 ttl=53 time=62.3 ms

--- ya.ru ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 62.333/62.333/62.333/0.000 ms
PING ya.ru (213.180.193.3) 56(84) bytes of data.
64 bytes from www.yandex.ru (213.180.193.3): icmp_seq=1 ttl=52 time=62.8 ms

--- ya.ru ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 63ms
rtt min/avg/max/mdev = 62.831/62.831/62.831/0.000 ms
PING ya.ru (213.180.204.3) 56(84) bytes of data.
64 bytes from www.yandex.ru (213.180.204.3): icmp_seq=1 ttl=52 time=63.4 ms

--- ya.ru ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 64ms
rtt min/avg/max/mdev = 63.453/63.453/63.453/0.000 ms

[Ruslan Khozinov (rkhozinov)]

It seems, that test has ran with BONDING=true

rkhozinov@tpi75:~$ brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.56847afe9799 no
fuelbr3429 8000.5254002e7b6f yes fuelbr3429-nic
       fuelnet31111
       fuelnet31112
       fuelnet31117
       fuelnet31118
       fuelnet31123
       fuelnet31124
       fuelnet31129
       fuelnet31130
       fuelnet31135
       fuelnet31136
       fuelnet31141
       fuelnet31142
       fuelnet31147
       fuelnet31148
       fuelnet31153
       fuelnet31154
       fuelnet31159
       fuelnet31160
       fuelnet31165
       fuelnet31166
fuelbr3430 8000.525400f6200a yes fuelbr3430-nic
       fuelnet31113
       fuelnet31114
       fuelnet31115
       fuelnet31116
       fuelnet31119
       fuelnet31120
       fuelnet31121
       fuelnet31122
       fuelnet31125
       fuelnet31126
       fuelnet31127
       fuelnet31128
       fuelnet31131
       fuelnet31132
       fuelnet31133
       fuelnet31134
       fuelnet31137
       fuelnet31138
       fuelnet31139
       fuelnet31140
       fuelnet31143
       fuelnet31144
       fuelnet31145
       fuelnet31146
       fuelnet31149
       fuelnet31150
       fuelnet31151
       fuelnet31152
       fuelnet31155
       fuelnet31156
       fuelnet31157
       fuelnet31158
       fuelnet31161
       fuelnet31162
       fuelnet31163
       fuelnet31164
       fuelnet31167
       fuelnet31168
       fuelnet31169
       fuelnet31170

I don't know what's a problem was with bonding+contrail, but I've retriggered CI with the BONDING=false option
And we can see 'common' networks for a fuel env.

bridge name bridge id STP enabled interfaces
docker0 8000.56847afe9799 no
fuelbr3431 8000.52540032543a yes fuelbr3431-nic
       fuelnet31171
fuelbr3432 8000.525400b31a9f yes fuelbr3432-nic
       fuelnet31172
fuelbr3433 8000.525400d9dbaa yes fuelbr3433-nic
       fuelnet31173
fuelbr3434 8000.52540074c803 yes fuelbr3434-nic
       fuelnet31174
fuelbr3435 8000.525400d5c200 yes fuelbr3435-nic
       fuelnet31175

[Ruslan Khozinov (rkhozinov)]

"Today the machines were defeated”

I and Oleksandr Martsyniuk have investigated the issue.

The problem was in repeating of the failed job with illegal parameters for bonding.

So if you want to re-run automated jobs - please use 'Re-trigger' instead of 'Retry'.

http://jenkins-tpi.bud.mirantis.net:8080/view/7.0.contrail.3.0/job/7.0.contrail.3.0.contrail_bvt.juniper/33/