fuel-devops 2.9.13 does not create postrouting rules for admin network

[Alex Schultz]

When using fuel-devops 2.9.13, the post routing rules are not being created via libvirt resulting in an admin node with no internet connectivity.

When creating an environment with 2.9.12, the following lines are present in the POSTROUTING configuration for iptables:

-A POSTROUTING -s 10.122.20.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 10.122.20.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 10.122.20.0/24 ! -d 10.122.20.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.122.20.0/24 ! -d 10.122.20.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.122.20.0/24 ! -d 10.122.20.0/24 -j MASQUERADE

In 2.9.13, no such rules are created and internet access is does not work from the fuel master.

[Dennis Dmitriev (ddmitriev)]

Could you please provide steps to reproduce:
1) Which environment variables were set,
2) How the environment was created (by command line, by a system test with some group name, using a YAML devops template or original approach) ?
3) Please provide the content of XML for admin network:
     virsh net-dumpxml ${ENV_NAME}_admin
4) Please provide the version of libvirt package

[Roman Prykhodchenko (romcheg)]

Setting to Incomplete until the requested information is provided

[Alex Schultz (alex-schultz)]

Using the following method since I'm running into https://bugs.launchpad.net/fuel/+bug/1521520 when trying to create environments via dos.py.

export ISO_PATH=/home/fuel/arch/fuel-8.0-264-2015-12-07_01-42-00.iso
export NODES_COUNT=6
export ENV_NAME=as-8.0-264-13
export VENV_PATH=~/venv-nailgun-tests-2.9
./utils/jenkins/system_tests.sh -t test -w $(pwd) -V $VENV_PATH -j testing -o --group=setup -i $ISO_PATH -k -K

(venv-nailgun-tests-2.9)fuel@qa-01:~/fuel-qa$ virsh net-dumpxml as-8.0-264-13_admin
<network connections='1'>
  <name>as-8.0-264-13_admin</name>
  <uuid>ca836b4c-ffa3-4c8e-a18a-13a4efd0c05f</uuid>
  <forward mode='route'/>
  <bridge name='fuelbr390' stp='on' delay='0'/>
  <mac address='52:54:00:e4:60:cc'/>
  <ip address='10.122.30.1' prefix='24'>
  </ip>
</network>

In comparison, here's the same from a working environment created via 2.9.12.
(venv-nailgun-tests-2.9)fuel@qa-01:~/fuel-qa$ virsh net-dumpxml as-8.0-264_admin
<network connections='6'>
  <name>as-8.0-264_admin</name>
  <uuid>ce3bb931-caa5-4cf8-86f7-aadd29663d84</uuid>
  <forward mode='route'/>
  <bridge name='fuelbr385' stp='on' delay='0'/>
  <mac address='52:54:00:17:98:02'/>
  <ip address='10.122.20.1' prefix='24'>
  </ip>
</network>

Version of libvirt:
ii libsys-virt-perl 1.2.15-1 amd64 Perl module providing an extension for the libvirt library
ii libvirt-bin 1.2.15-2 amd64 programs for the libvirt library
ii libvirt-clients 1.2.15-2 amd64 programs for the libvirt library
ii libvirt-daemon 1.2.15-2 amd64 programs for the libvirt library
ii libvirt-daemon-system 1.2.15-2 amd64 Libvirt daemon configuration files
ii libvirt-dev 1.2.15-2 amd64 development files for the libvirt library
ii libvirt0 1.2.15-2 amd64 library for interfacing with different virtualization systems
ii python-libvirt 1.2.9-1 amd64 libvirt Python bindings

[Igor Shishkin (teran)]

Infra DevOps team is not maintaining python-devops, assigning to fuel-qa.

[Dennis Dmitriev (ddmitriev)]

Alex, looks like you are re-using already created environment (with option '-k' ), so the test is not re-created networks.

According to your XMLs from networks, admin and public networks were created in 'route' mode instead of 'nat' mode.

Please check environment variables ADMIN_FORWARD, PUBLIC_FORWARD and FORWARD_DEFAULT.

Here you can find defaults: https://github.com/openstack/fuel-devops/blob/master/devops/settings.py#L129-L131

Moving to invalid until more details provided.

The bug wasn't reproduced on my local workstation:

$ dos.py create --iso-path /home/images/iso/fuel-8.0-270-2015-12-08_07-42-00.iso --node-count 9 --vcpu 2 --ram 3072 --admin-vcpu 2 --admin-ram 3072 --net-pool 10.155.0.0/16:24 fuelweb_test_system_test

$ sudo iptables -t nat -S | grep 10.155.0.0
-A POSTROUTING -s 10.155.0.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 10.155.0.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 10.155.0.0/24 ! -d 10.155.0.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.155.0.0/24 ! -d 10.155.0.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.155.0.0/24 ! -d 10.155.0.0/24 -j MASQUERADE

$ dos.py version
2.9.13

[Alex Schultz (alex-schultz)]

So the create method works fine (since I fixed it), the method via the system tests does not. Additionally even thought the -k was provided I was not reusing an environment. Please recheck with the method provided as the end results differs between 2.9.12 and 2.9.13 given the exact same setup. I did not specify any other environment variables other than what is in this bug.

[Dennis Dmitriev (ddmitriev)]

Bug is not reproduced on CI, where fuel-devops 2.9.16 is used.
Looks like the issue was solved in this version.