default route via admin network on nodes with mixed role 'compute' and 'virt'

[Danil Akhmetov]

Customer-fund issue.

Versions of components: Fuel 9.0

Detailed bug description:

When Reduced Footprint is enabled, compute nodes deployed with additional role 'virt' have misconfigured default gateway (via br-fw-admin instead of br-mgmt).

Admin network is designed for another purpose (server bootstrapping only, not for service traffic or user traffic) for this reason it should be secured as much as possible, so it seems to be at least a security issue.

Expected results:

default gateway configured via br-mgmt on compute nodes with virt role (virt role is provided by Reduced Footprint).

Actual result:

default gateway configured via br-fw-admin, seems like misconfiguration.

Impact:

Potential scalability / security issue, no any other impact noticed.

Steps to reproduce:

Enable Reduced Footprint in newly created environment as described in instructions, assign role 'virt' to computes, deploy env.

Additional information:

MOS-7.x and MOS-8.x have default gateway configured via correct interface in the same environment.

[Danil Akhmetov (dinobot)]

sla1 for 9.0-updates

[Oleksiy Molchanov (omolchanov)]

Marking as Incomplete. Please provide diagnostic snapshot.

[Bartosz Kupidura (zynzel)]

I tested this on Fuel 9.0 without success.

After deployment compute+virt node routing looks correct:
root@node-5:~# ip r
default via 10.109.7.1 dev br-ex
10.109.4.0/24 dev br-fw-admin proto kernel scope link src 10.109.4.4
10.109.5.0/24 dev br-mgmt proto kernel scope link src 10.109.5.1
10.109.6.0/24 dev br-storage proto kernel scope link src 10.109.6.1
10.109.7.0/24 dev br-ex proto kernel scope link src 10.109.7.2
10.109.8.0/24 dev br-mesh proto kernel scope link src 10.109.8.1
unreachable 169.254.169.254 scope host

Default GW is set to br-ex (and not br-mgmt as in standard compute), this is correct behavior as on 'virt' node we can run controllers in VM, which requires access to br-ex.

Please provide reproduction steps.

Also if network-templates are in use, please verify this without any custom network-templates.

[slava valyavskiy (slava-val-al)]

List of tasks performed on compute + virt role:
http://pastebin.com/5xUqaGDW

'netconfig.pp' task was not re-triggered after execution of 'hiera_default_route.pp'. So, override file has been generated, but, these changes have not been considered.

root@node-1:~# cat /etc/hiera/override/configuration/default_route.yaml
# Created by puppet, please do not edit
network_scheme:
  endpoints:
    br-fw-admin:
      gateway: ""
    br-mgmt:
      gateway: "10.109.1.8"

It should be fixed in yaql expression for netconfig task or some other magic should be added to make 'netconfig.pp' task be restarted after 'hiera_default_route.pp' task.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/354832

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/355208

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/355208
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f951f86716f1df6eb3a954da0a979286e90354e2
Submitter: Jenkins
Branch: stable/mitaka

commit f951f86716f1df6eb3a954da0a979286e90354e2
Author: slava <email address hidden>
Date: Fri Aug 12 19:35:36 2016 +0300

    Restart netconfig task when vips were changed

    Reduced Footprint workflow:
    1) add node with 'virt' role to cluster
    2) set RAM/CPU limits for VM what will be spawned on
       'virt' node
    3) run 'Provision VM' task
         - 'virt' node will be provisioned
         - set of tasks will be run on the node(including netconfig
           task and task for VM's creation)
         - created VM will be registered in the nailgun as this VM
           is connected to admin network
    4) add registered node with 'controller' node to the cluster
    5) run 'Deploy changes' task

    We have no acting 'vrouter' on step 3.2 and executing of 'hiera_default_route'
    will lead to failed deployment due to broken routing('hiera_default_route' is
    not included into SpawnVMs graph in nailgun).
    'hiera_default_route' is executing on step 5 for 'virt' nodes, but, 'netconfig'
    task wiil not be re-executed after it due to yaql condition.
    I have add extra condition to yaql expressions for both tasks to consider
    changes in 'vips' field.

    Change-Id: I932faa7bb931bfad12af1a15785dd2f1b6e8eb7b
    Closes-Bug: #1610236

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/354832
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4ae0535615fc8005780c64687abf14d704bd94b5
Submitter: Jenkins
Branch: master

commit 4ae0535615fc8005780c64687abf14d704bd94b5
Author: slava <email address hidden>
Date: Fri Aug 12 19:35:36 2016 +0300

    Restart netconfig task when vips were changed

    Reduced Footprint workflow:
    1) add node with 'virt' role to cluster
    2) set RAM/CPU limits for VM what will be spawned on
       'virt' node
    3) run 'Provision VM' task
         - 'virt' node will be provisioned
         - set of tasks will be run on the node(including netconfig
           task and task for VM's creation)
         - created VM will be registered in the nailgun as this VM
           is connected to admin network
    4) add registered node with 'controller' node to the cluster
    5) run 'Deploy changes' task

    We have no acting 'vrouter' on step 3.2 and executing of 'hiera_default_route'
    will lead to failed deployment due to broken routing('hiera_default_route' is
    not included into SpawnVMs graph in nailgun).
    'hiera_default_route' is executing on step 5 for 'virt' nodes, but, 'netconfig'
    task wiil not be re-executed after it due to yaql condition.
    I have add extra condition to yaql expressions for both tasks to consider
    changes in 'vips' field.

    Change-Id: I932faa7bb931bfad12af1a15785dd2f1b6e8eb7b
    Closes-Bug: #1610236

[Alexander Zatserklyany (zatserklyany)]

Verified:

root@node-4:~# ip r
default via 10.109.6.1 dev br-mgmt
10.109.5.0/24 dev br-fw-admin proto kernel scope link src 10.109.5.4
10.109.6.0/24 dev br-mgmt proto kernel scope link src 10.109.6.5
10.109.7.0/24 dev br-storage proto kernel scope link src 10.109.7.3
unreachable 169.254.169.254 scope host

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0 release.

[Sergey Novikov (snovikov)]

Verified on MOS 10.0 (RC #2)

Actually default gateway for node with mixed roles "compute+virt" is set via br-ex. As I know, it is correct. Task "netconfig" was re-triggered after execution of "hiera_default_route.pp"

2017-04-18 09:33:46 +0000 Scope(Class[Osnailyfacter::Generate_vms::Generate_vms]) (notice): MODULAR: generate_vms/generate_vms.pp
2017-04-18 09:58:11 +0000 Scope(Class[main]) (notice): MODULAR: rsync_core_puppet
2017-04-18 09:58:40 +0000 Scope(Class[Osnailyfacter::Globals::Globals]) (notice): MODULAR: globals/globals.pp
2017-04-18 09:58:45 +0000 Scope(Class[Osnailyfacter::Limits::Limits]) (notice): MODULAR: limits/limits.pp
2017-04-18 09:58:49 +0000 Scope(Class[Osnailyfacter::Netconfig::Hiera_default_route]) (notice): MODULAR: netconfig/hiera_default_route.pp
2017-04-18 10:03:40 +0000 Scope(Class[Osnailyfacter::Netconfig::Netconfig]) (notice): MODULAR: netconfig/netconfig.pp