Network verification failed after reboot of cluster with bonded interfaces

Bug #1594287 reported by Vladimir Khlyunev
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Won't Fix
High
Fuel CI
Mitaka
Confirmed
High
Fuel Sustaining

Bug Description

Swarm test failed:
ISO 9.0mos-507
Snapshot: https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.bonding_ha/146/artifact/logs/fail_error_deploy_bonding_neutron_vlan-fuel-snapshot-2016-06-20_07-52-56.tar.gz

Steps to reproduce:
1. Create cluster
2. Add 3 nodes with controller role
3. Add 1 node with compute role and 1 node with cinder role
4. Setup bonding for all interfaces (including admin interface
   bonding)
5. Run network verification
6. Deploy the cluster
7. Run network verification
8. Run OSTF
9. Save network configuration from slave nodes
10. Reboot all environment nodes
11. Verify that network configuration is the same after reboot
12. Run network verification <<<<<< fail

Expected result: all fine

Actual result:
Task 'verify_networks' has incorrect status. error != ready, 'Repo availability verification failed on following nodes slave-04_compute, slave-05_cinder.
 Following repos are not available - http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-06-17-222829/, http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/.

Notice: 3 node(s) were offline during connectivity check so they were skipped from the check.'

After revert I discovered that there is 2 offline nodes and the error mentioned above. I tried to reach an repo from affected node - and repos are accessible: http://paste.openstack.org/show/520570/
However, after several network check offline nodes has became online but slave-01 which was online before this issue failed with https://bugs.launchpad.net/fuel/+bug/1592844 .
Looks like there is multiple issues.

Revision history for this message
Maksim Malchuk (mmalchuk) wrote :

Maybe this bug is related to: https://bugs.launchpad.net/fuel/+bug/1589469

Changed in fuel:
status: New → Confirmed
importance: Undecided → High
milestone: 9.0 → 9.0-updates
tags: added: area-library
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 9.0-updates → 10.0
assignee: Fuel Sustaining (fuel-sustaining-team) → l23network (l23network)
tags: added: l23network
removed: area-library
Revision history for this message
Dmitry Belyaninov (dbelyaninov) wrote :
tags: added: swarm-fail
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: l23network (l23network) → Fuel Sustaining (fuel-sustaining-team)
tags: added: area-library
Revision history for this message
Kyrylo Galanov (kgalanov) wrote :
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Kyrylo Galanov (kgalanov)
Changed in fuel:
status: Confirmed → In Progress
Changed in fuel:
assignee: Kyrylo Galanov (kgalanov) → Fuel Sustaining (fuel-sustaining-team)
status: In Progress → Confirmed
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Oleksiy Molchanov (omolchanov)
status: Confirmed → In Progress
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

Diagnostic snapshot mentioned in report was deleted. Also I didn't manage to find similar case in recent CI builds. Marking this as Incomplete, please reopen as soon as this issue happens again.

Changed in fuel:
status: In Progress → Incomplete
Changed in fuel:
assignee: Oleksiy Molchanov (omolchanov) → Fuel Sustaining (fuel-sustaining-team)
Revision history for this message
Andrey Lavrentyev (alavrentyev) wrote :
Revision history for this message
Alexey. Kalashnikov (akalashnikov) wrote :
Changed in fuel:
status: Incomplete → Confirmed
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

The reason is that all controller nodes are down and non-controller nodes are trying to reach these repos using gateway that is managed by pacemaker on controller nodes.

Devops team, can you check the reason they are offline?

Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Fuel DevOps (fuel-devops)
tags: added: area-qa
removed: area-library
Revision history for this message
Andrey Nikitin (heos) wrote :

Looks like, Fuel CI team can help you.

Changed in fuel:
assignee: Fuel DevOps (fuel-devops) → Fuel CI (fuel-ci)
Roman Vyalov (r0mikiam)
Changed in fuel:
status: Confirmed → Won't Fix
Revision history for this message
Dmitry Kaigarodеsev (dkaiharodsev) wrote :

folks ci-team only can confirm that inside job we're launching system test with needed group and exporting needed environment variables,
the rest of work is under fuel-qa framework and scenario of used test group

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.