Bug #1259923 “OSTF test “Check network connectivity from instanc...” : Bugs : Fuel for OpenStack

Revision history for this message

Anastasia Palkina (apalkina) wrote on 2013-12-11:

#1

fuel-snapshot-2013-12-11_12-46-57.tgz Edit (3.2 MiB, application/x-tar)

Revision history for this message

Anastasia Palkina (apalkina) wrote on 2013-12-11:

#2

Screenshot from 2013-12-11 16:42:49.png Edit (422.9 KiB, image/png)

Anastasia Palkina (apalkina) on 2013-12-11

description:

updated

Tatyanka (tatyana-leontovich) on 2013-12-11

Changed in fuel:
importance:	Undecided → High

Revision history for this message

Mike Scherbakov (mihgen) wrote on 2013-12-12:

#3

Lowered importance to medium: it affects only one HealthCheck test, right?

Changed in fuel:
importance:	High → Medium

Revision history for this message

Anastasiia Naboikina (anaboikina) wrote on 2013-12-12:

#4

Yes, Mike. And so far I wasn`t able to reproduce this issue.

Revision history for this message

Anastasiia Naboikina (anaboikina) wrote on 2013-12-13:

#5

So far everything works on bare-metal and on virtualbox. So, fix will be provided after there is an env where it`s reproduced.

Changed in fuel:
status:	New → Incomplete

Anastasiia Naboikina (anaboikina) on 2013-12-18

Changed in fuel:
assignee:	Anastasiia Naboikina (anaboikina) → nobody

Revision history for this message

Aleksandr Didenko (adidenko) wrote on 2014-01-10:

#6

Download full text (4.1 KiB)

I get the same problem sporadically on bare-metal.

ISO: {"build_id": "2014-01-08_01-17-41", "ostf_sha": "05b1bfc92fa40728966518992c94176030c25f35", "build_number": "21", "nailgun_sha": "ea5dc470d3b789ad8dc0be2c256b9c7ec6ac8500", "fuelmain_sha": "73fb5d8bba72cb032b7ed4d034827b75042d0c47", "astute_sha": "419e864c77681c355043c21e76b3f5a940bef9ea", "release": "4.1", "fuellib_sha": "f661314923ce584ad84c40c7a99e80a8feb4e713"}

This problem occurs on environments with "neutron" only (with both vlan segmentation and GRE tunnels).

"Check network connectivity from instance via floating IP" OSTF check takes 25-27 seconds to complete on "nova network" configuration. But on "neutron" it takes 85 seconds minimum (average time is 85-100 seconds). And sometimes it hits the retries*timeout limit for SSH connection to the instance via floating IP. In both cases (nova-network and neutron) floating IP assigned to instance is accessible (pingable) in ~25 seconds.

So to summarize:
- nova-network: OSTF "Check network connectivity from instance via floating IP" check takes ~30 seconds (floating IP is "pingable" in ~25 seconds)
- neutron: OSTF "Check network connectivity from instance via floating IP" check takes ~80 seconds minimum (floating IP is "pingable" in ~25 seconds)

So this time difference is not caused by network and/or floating IP configuration, it seems to be caused by delayed SSH service start. It looks like it's happening because of "/etc/rc3.d/S45-cirros-net-ds" start script from Cirros TestVM image. It takes ~45 seconds to complete on neutron environment:

# time /sbin/cirros-ds net
cirros-ds 'net' up at 160.02
checking http://169.254.169.254/2009-04-04/instance-id
successful after 1/20 tries: up 160.03. iid=i-00000009
failed to get http://169.254.169.254/2009-04-04/meta-data// <head>/openssh-key
warning: no ec2 metadata for public-keys
failed to get http://169.254.169.254/2009-04-04/meta-data/instance-id
warning: no ec2 metadata for instance-id
failed to get http://169.254.169.254/2009-04-04/meta-data// <head>/openssh-key
warning: no ec2 metadata for public-keys
failed to get http://169.254.169.254/2009-04-04/meta-data/hostname
warning: no ec2 metadata for hostname
found datasource (ec2, net)
real 0m 45.55s
user 0m 0.00s
sys 0m 0.00s

Same command on instance with nova-network environment takes only few seconds:

# time /sbin/cirros-ds net
cirros-ds 'net' up at 44.18
checking http://169.254.169.254/2009-04-04/instance-id
successful after 1/20 tries: up 44.18. iid=i-0000000a
found datasource (ec2, net)
real 0m 3.45s
user 0m 0.00s
sys 0m 0.00s

Example from /var/log/messages from the cirros instance with neutron, note the time difference between instance start and dropbear (SSH service) run:

Jan 9 04:23:05 cirros kern.info kernel: [ 1.168128] Refined TSC clocksource calibration: 3499.947 MHz.
Jan 9 04:23:15 cirros kern.debug kernel: [ 11.736099] eth0: no IPv6 routers present
Jan 9 04:23:50 cirros auth.notice su: + /dev/console root:cirros
Jan 9 04:23:50 cirros authpriv.info dropbear[289]: Running in background

The same part of /var/log/messages from the cirros instance with nova-network:

Jan 9 06:13:23 cirros kern...

I get the same problem sporadically on bare-metal.

ISO: {"build_id": "2014-01-08_01-17-41", "ostf_sha": "05b1bfc92fa40728966518992c94176030c25f35", "build_number": "21", "nailgun_sha": "ea5dc470d3b789ad8dc0be2c256b9c7ec6ac8500", "fuelmain_sha": "73fb5d8bba72cb032b7ed4d034827b75042d0c47", "astute_sha": "419e864c77681c355043c21e76b3f5a940bef9ea", "release": "4.1", "fuellib_sha": "f661314923ce584ad84c40c7a99e80a8feb4e713"}

This problem occurs on environments with "neutron" only (with both vlan segmentation and GRE tunnels).

"Check network connectivity from instance via floating IP" OSTF check takes 25-27 seconds to complete on "nova network" configuration. But on "neutron" it takes 85 seconds minimum (average time is 85-100 seconds). And sometimes it hits the retries*timeout limit for SSH connection to the instance via floating IP. In both cases (nova-network and neutron) floating IP assigned to instance is accessible (pingable) in ~25 seconds.

So to summarize:
- nova-network: OSTF "Check network connectivity from instance via floating IP" check takes ~30 seconds (floating IP is "pingable" in ~25 seconds)
- neutron:  OSTF "Check network connectivity from instance via floating IP" check takes ~80 seconds minimum (floating IP is "pingable" in ~25 seconds)

So this time difference is not caused by network and/or floating IP configuration, it seems to be caused by delayed SSH service start. It looks like it's happening because of "/etc/rc3.d/S45-cirros-net-ds" start script from Cirros TestVM image. It takes ~45 seconds to complete on neutron environment:

# time /sbin/cirros-ds net
cirros-ds 'net' up at 160.02
checking http://169.254.169.254/2009-04-04/instance-id
successful after 1/20 tries: up 160.03. iid=i-00000009
failed to get http://169.254.169.254/2009-04-04/meta-data// <head>/openssh-key
warning: no ec2 metadata for public-keys
failed to get http://169.254.169.254/2009-04-04/meta-data/instance-id
warning: no ec2 metadata for instance-id
failed to get http://169.254.169.254/2009-04-04/meta-data// <head>/openssh-key
warning: no ec2 metadata for public-keys
failed to get http://169.254.169.254/2009-04-04/meta-data/hostname
warning: no ec2 metadata for hostname
found datasource (ec2, net)
real    0m 45.55s
user    0m 0.00s
sys     0m 0.00s

Same command on instance with nova-network environment takes only few seconds:

# time /sbin/cirros-ds net
cirros-ds 'net' up at 44.18
checking http://169.254.169.254/2009-04-04/instance-id
successful after 1/20 tries: up 44.18. iid=i-0000000a
found datasource (ec2, net)
real    0m 3.45s
user    0m 0.00s
sys     0m 0.00s

Example from /var/log/messages from the cirros instance with neutron, note the time difference between instance start and dropbear (SSH service) run:

Jan  9 04:23:05 cirros kern.info kernel: [    1.168128] Refined TSC clocksource calibration: 3499.947 MHz.
Jan  9 04:23:15 cirros kern.debug kernel: [   11.736099] eth0: no IPv6 routers present
Jan  9 04:23:50 cirros auth.notice su: + /dev/console root:cirros
Jan  9 04:23:50 cirros authpriv.info dropbear[289]: Running in background

The same part of /var/log/messages from the cirros instance with nova-network:

Jan  9 06:13:23 cirros kern.info kernel: [    0.992140] eth0: IPv6 duplicate address fe80::f816:3eff:fe13:b75a detected!
Jan  9 06:13:23 cirros kern.info kernel: [    1.168077] Refined TSC clocksource calibration: 3499.944 MHz.
Jan  9 06:13:26 cirros auth.notice su: + /dev/console root:cirros
Jan  9 06:13:26 cirros authpriv.info dropbear[491]: Running in background

I've modified Cirros TestVM image on my test environment to make it start SSH service before cirros-net-ds (mv /etc/rc3.d/S50-dropbear /etc/rc3.d/S41-dropbear). After that "Check network connectivity from instance via floating IP" OSTF check started to take 55-57 seconds (down from 85-100) on environments with neutron.

I suppose the most easy solution here is to either add a 45-60 seconds sleep or increase number of SSH connection retries for "Check network connectivity from instance via floating IP" OSTF check on environments with "neutron". Or we can update TestVM image and start SSH before cirros-net-ds.

Aleksandr Didenko (adidenko) on 2014-01-10

Changed in fuel:
status:	Incomplete → Confirmed

Revision history for this message

Anastasia Palkina (apalkina) wrote on 2014-01-13:

#7

Also reproduced on ISO #30
"build_id": "2014-01-13_01-17-41",
"ostf_sha": "05b1bfc92fa40728966518992c94176030c25f35",
"build_number": "30",
"nailgun_sha": "b4af350fc72ebaa26310cdb2bde71a466b16d598",
"fuelmain_sha": "4b7628de62b5dc0301ba3553a1aa2437d233e95f",
"astute_sha": "ca787b5b0a3a418e6885b9fd2d795c9fd158ed0a",
"release": "4.1",
"fuellib_sha": "c8673bb9474ccb0a51fb9077910b009ff2d9034b"

From /var/log/messages from the cirros instance with neutron (VLAN)
Jan 13 03:11:05 cirros kern.debug kernel: [ 29.681432] eth0: no IPv6 routers present
Jan 13 03:11:46 cirros auth.notice su: + /dev/console root:cirros
Jan 13 03:12:27 cirros authpriv.info dropbear[495]: Running in background

Changed in fuel:
milestone:	4.0 → 4.1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-24: Fix proposed to fuel-ostf (master)

#8

Fix proposed to branch: master
Review: https://review.openstack.org/68891

Changed in fuel:
assignee:	nobody → Anastasiia Naboikina (anaboikina)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-26: Fix merged to fuel-ostf (master)

#9

Reviewed: https://review.openstack.org/68891
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=338ddf840c229918d1df8c6597588b853d02de4c
Submitter: Jenkins
Branch: master

commit 338ddf840c229918d1df8c6597588b853d02de4c
Author: AnastasiiaNaboikina <email address hidden>
Date: Fri Jan 24 13:19:43 2014 +0200

Add more retries for connectivity check

    Increased number of retries for instance connectivity
    checks in test_nova_create_instance_with_connectivity
    to prevent failures when running tests on Neutron

Change-Id: I429080524cbd04dae2db40cdba7d0feec63e5205
Closes-bug: 1259923

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

Anastasia Palkina (apalkina) wrote on 2014-02-07:

#10

Verified on ISO #102
"build_id": "2014-02-07_13-44-21",
"ostf_sha": "d15d6b5b952e455e3afff383413ffa6d89ee7981",
"build_number": "102",
"nailgun_sha": "2d6aa79b9ed01ba166a0db543e19cd6d1d844503",
"fuelmain_sha": "7b8a9343db8d3c9cc5ba249768d72ea6d15d1a11",
"astute_sha": "d002c3bf626cff96a1d4aec9eb92fc4d5f4542c4",
"release": "4.1",
"fuellib_sha": "c10d3896a6b1b760229cc8ebd93778a48b571ee6"

Changed in fuel:
status:	Fix Committed → Fix Released

Fuel for OpenStack

OSTF test "Check network connectivity from instance via floating IP" on ping 8.8.8.8 from instance

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches