VMs in 192.168.122.0/24 subnet have broken networking

[Erik Swanson]

If I create a 192.168.122.0/24 subnet, VMs in that subnet are unable to reach the external world.

Symptoms:

As seen while pinging an external IP, there is an unwanted/incorrect SNAT applied to the packets before they leave the compute node. (When observing the private network with tcpdump, the pings will have a source IP that is *not* the VM.)
This causes the packets to not match the right (or any) SNAT rule inside the qrouter on the controller.

Other subnets with identical router and security group configurations work fine.

Suspected cause:

Failure of Fuel to remove a presumably-unneeded default libvirt network. (Observation/suggestion by "Sam-I-Am" in #openstack, who was helping me diagnose this.)

[Erik Swanson (erik-swanson)]

Cluster details: Fuel 6.0, HA, on Ubuntu. Networking: Neutron with VLANs.

[Vladimir Kuklin (vkuklin)]

Erik, can you choose other network than this one? This is a very specific issue that can be easily worked around.

[Dmitry Ilyin (idv1985)]

do you have these values in sysctl?
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1

Try to set them to 0

[Dmitry Ilyin (idv1985)]

On your host node

[Erik Swanson (erik-swanson)]

My cluster has 16 compute nodes and 3 controllers.

Is this change you want me to try safe for a production environment?
If so, is there a way to have the change rolled out automatically to the appropriate nodes?

[Vladimir Kuklin (vkuklin)]

The issue here is pretty simple - there is a default network created by libvirt. Workaround is also pretty simple - if you want to use this network, just undefine default libvirt network on all compute nodes.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/209508

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/209508
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=4e5e4cfe3a7be1d0b4c9830865bb7b6eeee9f0a3
Submitter: Jenkins
Branch: master

commit 4e5e4cfe3a7be1d0b4c9830865bb7b6eeee9f0a3
Author: Alexey Deryugin <email address hidden>
Date: Wed Aug 5 16:04:48 2015 +0300

    Delete default libvirt network definition

    Remove default libvirt network due to possible using
    the same network as VMs subnet in production cluster.

    Co-Authored-By: Denis Egorenko <email address hidden>

    Change-Id: Ic8720b37e924c4fbbe243f7a5ea9a94113d054fd
    Closes-Bug: #1437410

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/222658
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=18e039a858e9e7d8846f55dc0bf5ff193d1d8ac2
Submitter: Jenkins
Branch: master

commit 18e039a858e9e7d8846f55dc0bf5ff193d1d8ac2
Author: Alexander Adamov <email address hidden>
Date: Fri Sep 11 18:02:12 2015 +0300

    [RN 7.0]Fuel install&deploy issues

    Adds resolved and known issues:
    LP1491725, LP1437410,
    LP1477903

    Change-Id: I87fcb333d632de6faa7071713f88e8519bccf8d7
    Related-Bug: #1491725
    Related-Bug: #1437410
    Related-Bug: #1477903

[Dmitriy Kruglov (dkruglov)]

Verified on MOS 7.0, custom ISO. The issue is not reproduced.

ISO info:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "1260"
  build_id: "2015-10-09_12-02-12"
  nailgun_sha: "edbae54d510edbaa1d379e9523febe5a0e5acd41"
  python-fuelclient_sha: "486bde57cda1badb68f915f66c61b544108606f3"
  fuel-agent_sha: "50e90af6e3d560e9085ff71d2950cfbcca91af67"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "6c5b73f93e24cc781c809db9159927655ced5012"
  fuel-library_sha: "713698e88c6e1e4ed9ebad759a21266890898d57"
  fuel-ostf_sha: "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c"
  fuelmain_sha: "a65d453215edb0284a2e4761be7a156bb5627677"