Fuel for OpenStack

[vCenter] Wrong default gateway on cinder-vmware and compute-vmware nodes

Bug #1544067 reported by Ilya Bumarskov
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Alexander Arzhanov
Fuel for OpenStack 9.0
7.0.x
Fix Released
Critical
Denis Meltsaykin
Fuel for OpenStack 7.0-mu-7
8.0.x
Fix Released
Critical
Alexander Arzhanov
Fuel for OpenStack 8.0

Bug Description

As default gateway on compute-vmware and cinder-vmware nodes is used ip address of Fuel node (admin network) instead ip address of vrouter (management network).

Fuel 8.0 ISO #529

Enviroment:
Compute: QEMU + vCenter
Networking Setup: Neutron with VMware DVS (ML2 plugin)
Storage backends: default
Additional services: all by default

[root@nailgun ~]# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|----------|------------------|---------|------------|-------------------|-----------------|---------------|--------|---------
1 | ready | Untitled (ce:6c) | 1 | 10.109.0.6 | 64:6e:71:2b:ce:6c | controller | | True | 1
4 | ready | Untitled (d8:c5) | 1 | 10.109.0.3 | 64:6c:a9:9a:d8:c5 | cinder-vmware | | True | 1
2 | ready | Untitled (f6:df) | 1 | 10.109.0.7 | 64:4a:c7:76:f6:df | cinder, compute | | True | 1
3 | ready | Untitled (8b:28) | 1 | 10.109.0.5 | 64:33:16:d4:8b:28 | compute-vmware | | True | 1
5 | ready | Untitled (0f:14) | 1 | 10.109.0.4 | 64:8b:51:4a:0f:14 | cinder-vmware | | False | 1
6 | discover | Untitled (f9:47) | None | 10.109.0.8 | 64:ba:b6:d4:f9:47 | | | True | None

Compute-vmware node:
[root@nailgun ~]# ssh node-3 ip route
Warning: Permanently added 'node-3' (ECDSA) to the list of known hosts.
default via 10.109.0.1 dev br-fw-admin
10.109.0.0/24 dev br-fw-admin proto kernel scope link src 10.109.0.5
10.109.1.0/24 dev br-mgmt proto kernel scope link src 10.109.1.7
10.109.2.0/24 dev br-storage proto kernel scope link src 10.109.2.5
unreachable 169.254.169.254 scope host

Cinder-vmware node:
[root@nailgun ~]# ssh node-4 ip route
Warning: Permanently added 'node-4' (ECDSA) to the list of known hosts.
default via 10.109.0.1 dev br-fw-admin
10.109.0.0/24 dev br-fw-admin proto kernel scope link src 10.109.0.3
10.109.1.0/24 dev br-mgmt proto kernel scope link src 10.109.1.6
10.109.2.0/24 dev br-storage proto kernel scope link src 10.109.2.4

Compute node (for comparison):
[root@nailgun ~]# ssh node-2 ip route
Warning: Permanently added 'node-2' (ECDSA) to the list of known hosts.
default via 10.109.1.2 dev br-mgmt
10.109.0.0/24 dev br-fw-admin proto kernel scope link src 10.109.0.7
10.109.1.0/24 dev br-mgmt proto kernel scope link src 10.109.1.8
10.109.2.0/24 dev br-storage proto kernel scope link src 10.109.2.6
unreachable 169.254.169.254 scope host

See original description
Ilya Bumarskov (ibumarskov)
description: updated
tags: added: area-pce-vcenter vcenter
Changed in fuel:
milestone: none → 9.0
importance: Undecided → Critical
assignee: nobody → Partner Centric Engineering (fuel-partner-engineering)
Revision history for this message
Artem Savinov (asavinov) wrote :

For reconfigure gw to public vip on compute node execute "configure_default_route" task, which run /etc/puppet/modules/osnailyfacter/modular/netconfig/configure_default_route.pp puppet manifest. In nsxv plugin we run custom task, which execute this manifest. Please do not rename or move this manifest, else nsxv plugin will not work.

Alexander Arzhanov (aarzhanov)
Changed in fuel:
status: New → Confirmed
Alexander Arzhanov (aarzhanov)
Changed in fuel:
importance: Critical → High
Revision history for this message
Alexander Arzhanov (aarzhanov) wrote :

I dropped to a "High" importance, because we do not have problems with deployment.
But this is a very nasty bug, because we have incorrectly configured gateway.

I think, we can fix this bug, if we add compute-vmware and cinder-vmware roles on this places(need to test):

for 7.0 https://github.com/openstack/fuel-library/blob/stable/7.0/deployment/puppet/osnailyfacter/modular/astute/tasks.yaml#L96
for 8.0 https://github.com/openstack/fuel-library/blob/stable/8.0/deployment/puppet/osnailyfacter/modular/astute/tasks.yaml#L118
for 9.0 https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/astute/tasks.yaml#L118

These are small changes and they will not affect the product.

Revision history for this message
Alexander Arzhanov (aarzhanov) wrote :

Workaround this problem, you need to do the following on all compute-vmware and separate cinder-vmware nodes or on all compute-vmware+cinder-vmware nodes:

/usr/bin/puppet apply /etc/puppet/modules/osnailyfacter/modular/netconfig/configure_default_route.pp --modulepath=/etc/puppet/modules --logdest syslog --trace --no-report --debug --evaltrace --logdest /var/log/puppet.log

Changed in fuel:
assignee: Partner Centric Engineering (fuel-partner-engineering) → Alexander Arzhanov (aarzhanov)
Alexander Arzhanov (aarzhanov)
Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/279046

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/279050

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/279056

Revision history for this message
Eugene Bogdanov (ebogdanov) wrote :

Promoted to Critical because of bad user experience and large number of users to be potentially affected.

Changed in fuel:
importance: High → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/8.0)

Change abandoned by Alexander Arzhanov (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/279050

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/279631

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/279056
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=54f6f6952f695544fd51f9c8d2f6451af53d27b8
Submitter: Jenkins
Branch: master

commit 54f6f6952f695544fd51f9c8d2f6451af53d27b8
Author: Alexander Arzhanov <email address hidden>
Date: Thu Feb 11 13:30:55 2016 +0000

    Configure correct gateway for cinder-vmware and compute-vmware nodes

    - Add cinder-vmware and compute-vmware roles to task configure_default_route
      like compute and cinder nodes.

    Change-Id: I179d7e6b41ae84529f35975fbd98298380d49e47
    Closes-bug: #1544067

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/8.0)

Reviewed: https://review.openstack.org/279631
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c2a335b5b725f1b994f78d4c78723d29fa44685a
Submitter: Jenkins
Branch: stable/8.0

commit c2a335b5b725f1b994f78d4c78723d29fa44685a
Author: Alexander Arzhanov <email address hidden>
Date: Thu Feb 11 13:30:55 2016 +0000

    Configure correct gateway for cinder-vmware and compute-vmware nodes

    - Add cinder-vmware and compute-vmware roles to task configure_default_route
      like compute and cinder nodes.

    Change-Id: I179d7e6b41ae84529f35975fbd98298380d49e47
    Closes-bug: #1544067

Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :

Verified on Fuel 8.0 #571

[root@nailgun ~]# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|--------|-------------------------|---------|------------|-------------------|-----------------|---------------|--------|---------
6 | ready | slave-04_compute_cinder | 1 | 10.109.0.6 | 64:f2:bc:2d:c0:cc | cinder, compute | | True | 1
1 | ready | slave-02_controller | 1 | 10.109.0.4 | 64:7f:fe:61:47:da | controller | | True | 1
3 | ready | slave-06_compute-vmware | 1 | 10.109.0.8 | 64:d9:72:fb:5a:b0 | compute-vmware | | True | 1
2 | ready | slave-05_cinder-vmware | 1 | 10.109.0.7 | 64:d5:1b:e6:91:bd | cinder-vmware | | True | 1
4 | ready | slave-03_controller | 1 | 10.109.0.5 | 64:8c:c4:2b:fa:2f | controller | | True | 1
5 | ready | slave-01_controller | 1 | 10.109.0.3 | 64:67:20:34:20:ac | controller | | True | 1

[root@nailgun ~]# ssh node-3 ip route
Warning: Permanently added 'node-3' (ECDSA) to the list of known hosts.
default via 10.109.1.2 dev br-mgmt
10.109.0.0/24 dev br-fw-admin proto kernel scope link src 10.109.0.8
10.109.1.0/24 dev br-mgmt proto kernel scope link src 10.109.1.8
10.109.2.0/24 dev br-storage proto kernel scope link src 10.109.2.6
unreachable 169.254.169.254 scope host

[root@nailgun ~]# ssh node-2 ip route
Warning: Permanently added 'node-2' (ECDSA) to the list of known hosts.
default via 10.109.1.2 dev br-mgmt
10.109.0.0/24 dev br-fw-admin proto kernel scope link src 10.109.0.7
10.109.1.0/24 dev br-mgmt proto kernel scope link src 10.109.1.9
10.109.2.0/24 dev br-storage proto kernel scope link src 10.109.2.7
unreachable 169.254.169.254 scope host

Revision history for this message
Ekaterina Khomyakova (ekhomyakova) wrote :

Verified on Fuel 9.0 #133.

[root@nailgun ~]# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|--------|-------------------------|---------|------------|-------------------|-----------------|---------------|--------|---------
2 | ready | slave-03_compute-vmware | 1 | 10.109.0.5 | 64:58:54:21:f3:a5 | compute-vmware | | True | 1
3 | ready | slave-02_cinder-vmware | 1 | 10.109.0.4 | 64:ed:02:e0:32:c3 | cinder-vmware | | True | 1
5 | ready | Untitled (5a:ec) | 1 | 10.109.0.6 | 64:17:17:f6:5a:ec | cinder-vmware | | True | 1
1 | ready | slave-01_controller | 1 | 10.109.0.3 | 64:f7:53:48:79:f2 | controller | | True | 1
6 | ready | Untitled (5b:59) | 1 | 10.109.0.7 | 64:9b:84:8a:5b:59 | cinder, compute | | True | 1

[root@nailgun ~]# ssh node-2 ip route
Warning: Permanently added 'node-2' (ECDSA) to the list of known hosts.
default via 10.109.1.2 dev br-mgmt
10.109.0.0/24 dev br-fw-admin proto kernel scope link src 10.109.0.5
10.109.1.0/24 dev br-mgmt proto kernel scope link src 10.109.1.8
10.109.2.0/24 dev br-storage proto kernel scope link src 10.109.2.6
unreachable 169.254.169.254 scope host

[root@nailgun ~]# ssh node-3 ip route
Warning: Permanently added 'node-3' (ECDSA) to the list of known hosts.
default via 10.109.1.2 dev br-mgmt
10.109.0.0/24 dev br-fw-admin proto kernel scope link src 10.109.0.4
10.109.1.0/24 dev br-mgmt proto kernel scope link src 10.109.1.7
10.109.2.0/24 dev br-storage proto kernel scope link src 10.109.2.5
unreachable 169.254.169.254 scope host

[root@nailgun ~]# ssh node-6 ip route
Warning: Permanently added 'node-6' (ECDSA) to the list of known hosts.
default via 10.109.1.2 dev br-mgmt
10.109.0.0/24 dev br-fw-admin proto kernel scope link src 10.109.0.7
10.109.1.0/24 dev br-mgmt proto kernel scope link src 10.109.1.4
10.109.2.0/24 dev br-storage proto kernel scope link src 10.109.2.2
unreachable 169.254.169.254 scope host

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/7.0)

Change abandoned by Alexander Arzhanov (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/279046

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/7.0)

Reviewed: https://review.openstack.org/279046
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=5ca8cf49d67662299859040e98b17a2fac84fd9c
Submitter: Jenkins
Branch: stable/7.0

commit 5ca8cf49d67662299859040e98b17a2fac84fd9c
Author: Alexander Arzhanov <email address hidden>
Date: Thu Feb 11 13:12:18 2016 +0000

    Configure correct gateway for cinder-vmware and compute-vmware nodes

    - Add cinder-vmware and compute-vmware roles to task configure_default_route
      like compute and cinder nodes.
    - Fix incorrect fixture.

    Change-Id: I0ff62276375de4355cfc00a884f47697a11436f1
    Closes-bug: #1544067

TatyanaGladysheva (tgladysheva)
tags: added: on-verification
Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

Verified on MOS 7.0 + MU7 updates.

Actual results after fix:
[root@nailgun ~]# fuel nodes
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|--------|------------------|---------|-------------|-------------------|-----------------|---------------|--------|---------
5 | ready | Untitled (f8:6e) | 1 | 10.109.10.6 | 64:66:e7:f1:f8:6e | compute-vmware | | True | 1
4 | ready | Untitled (3d:aa) | 1 | 10.109.10.4 | 64:4b:24:73:3d:aa | cinder-vmware | | True | 1
1 | ready | Untitled (0f:ad) | 1 | 10.109.10.3 | 64:0e:e8:7a:0f:ad | controller | | True | 1
2 | ready | Untitled (9f:57) | 1 | 10.109.10.7 | 64:7a:68:57:9f:57 | cinder, compute | | True | 1
3 | ready | Untitled (f3:d5) | 1 | 10.109.10.5 | 64:1f:02:cf:f3:d5 | cinder-vmware | | True | 1
[root@nailgun ~]# ssh node-2 ip route
Warning: Permanently added 'node-2' (RSA) to the list of known hosts.
default via 10.109.11.1 dev br-mgmt
10.109.10.0/24 dev br-fw-admin proto kernel scope link src 10.109.10.7
10.109.11.0/24 dev br-mgmt proto kernel scope link src 10.109.11.5
10.109.12.0/24 dev br-storage proto kernel scope link src 10.109.12.4
unreachable 169.254.169.254 scope host

[root@nailgun ~]# ssh node-3 ip route
Warning: Permanently added 'node-3' (RSA) to the list of known hosts.
default via 10.109.11.1 dev br-mgmt
10.109.10.0/24 dev br-fw-admin proto kernel scope link src 10.109.10.5
10.109.11.0/24 dev br-mgmt proto kernel scope link src 10.109.11.4
10.109.12.0/24 dev br-storage proto kernel scope link src 10.109.12.5
unreachable 169.254.169.254 scope host

[root@nailgun ~]# ssh node-5 ip route
Warning: Permanently added 'node-5' (RSA) to the list of known hosts.
default via 10.109.11.1 dev br-mgmt
10.109.10.0/24 dev br-fw-admin proto kernel scope link src 10.109.10.6
10.109.11.0/24 dev br-mgmt proto kernel scope link src 10.109.11.6
10.109.12.0/24 dev br-storage proto kernel scope link src 10.109.12.3
unreachable 169.254.169.254 scope host
[root@nailgun ~]#

tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.