VMs in subnet have broken networking

Bug #1437410 reported by Erik Swanson on 2015-03-27
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Denis Egorenko
MOS Maintenance
MOS Maintenance
Denis Egorenko

Bug Description

If I create a subnet, VMs in that subnet are unable to reach the external world.


As seen while pinging an external IP, there is an unwanted/incorrect SNAT applied to the packets before they leave the compute node. (When observing the private network with tcpdump, the pings will have a source IP that is *not* the VM.)
This causes the packets to not match the right (or any) SNAT rule inside the qrouter on the controller.

Other subnets with identical router and security group configurations work fine.

Suspected cause:

Failure of Fuel to remove a presumably-unneeded default libvirt network. (Observation/suggestion by "Sam-I-Am" in #openstack, who was helping me diagnose this.)

Erik Swanson (erik-swanson) wrote :

Cluster details: Fuel 6.0, HA, on Ubuntu. Networking: Neutron with VLANs.

Changed in fuel:
milestone: none → 6.1
assignee: nobody → Fuel Library Team (fuel-library)
Vladimir Kuklin (vkuklin) wrote :

Erik, can you choose other network than this one? This is a very specific issue that can be easily worked around.

Changed in fuel:
status: New → Confirmed
importance: Undecided → Medium
Changed in fuel:
status: Confirmed → Won't Fix
tags: added: qa-agree-7.0 release-notes
Dmitry Ilyin (idv1985) wrote :

do you have these values in sysctl?
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1

Try to set them to 0

Dmitry Ilyin (idv1985) wrote :

On your host node

Erik Swanson (erik-swanson) wrote :

My cluster has 16 compute nodes and 3 controllers.

Is this change you want me to try safe for a production environment?
If so, is there a way to have the change rolled out automatically to the appropriate nodes?

Vladimir Kuklin (vkuklin) wrote :

The issue here is pretty simple - there is a default network created by libvirt. Workaround is also pretty simple - if you want to use this network, just undefine default libvirt network on all compute nodes.

Fix proposed to branch: master
Review: https://review.openstack.org/209508

Changed in fuel:
status: Won't Fix → In Progress
Changed in fuel:
assignee: Alexey Deryugin (velovec) → Denis Egorenko (degorenko)

Reviewed: https://review.openstack.org/209508
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=4e5e4cfe3a7be1d0b4c9830865bb7b6eeee9f0a3
Submitter: Jenkins
Branch: master

commit 4e5e4cfe3a7be1d0b4c9830865bb7b6eeee9f0a3
Author: Alexey Deryugin <email address hidden>
Date: Wed Aug 5 16:04:48 2015 +0300

    Delete default libvirt network definition

    Remove default libvirt network due to possible using
    the same network as VMs subnet in production cluster.

    Co-Authored-By: Denis Egorenko <email address hidden>

    Change-Id: Ic8720b37e924c4fbbe243f7a5ea9a94113d054fd
    Closes-Bug: #1437410

Changed in fuel:
status: In Progress → Fix Committed
tags: added: release-notes-done
removed: release-notes
tags: added: rn7.0

Reviewed: https://review.openstack.org/222658
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=18e039a858e9e7d8846f55dc0bf5ff193d1d8ac2
Submitter: Jenkins
Branch: master

commit 18e039a858e9e7d8846f55dc0bf5ff193d1d8ac2
Author: Alexander Adamov <email address hidden>
Date: Fri Sep 11 18:02:12 2015 +0300

    [RN 7.0]Fuel install&deploy issues

    Adds resolved and known issues:
    LP1491725, LP1437410,

    Change-Id: I87fcb333d632de6faa7071713f88e8519bccf8d7
    Related-Bug: #1491725
    Related-Bug: #1437410
    Related-Bug: #1477903

tags: added: on-verification
Dmitriy Kruglov (dkruglov) wrote :

Verified on MOS 7.0, custom ISO. The issue is not reproduced.

ISO info:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "1260"
  build_id: "2015-10-09_12-02-12"
  nailgun_sha: "edbae54d510edbaa1d379e9523febe5a0e5acd41"
  python-fuelclient_sha: "486bde57cda1badb68f915f66c61b544108606f3"
  fuel-agent_sha: "50e90af6e3d560e9085ff71d2950cfbcca91af67"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "6c5b73f93e24cc781c809db9159927655ced5012"
  fuel-library_sha: "713698e88c6e1e4ed9ebad759a21266890898d57"
  fuel-ostf_sha: "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c"
  fuelmain_sha: "a65d453215edb0284a2e4761be7a156bb5627677"

tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers