| 2013-11-19 23:49:48 |
Thiago Martins |
bug |
|
|
added bug |
| 2013-11-20 00:00:56 |
Thiago Martins |
description |
Hello!
Currently, Havana L3 Router have a serious issue. Which makes it almost useless (sorry, I do not want to be rude but instead, trying to bring more attention to this problem).
When the tenant network traffic pass trough the L3 Router (Namespace at the Network Node), it becomes very, very slow and intermittent. The issue also affects the traffic that hit a "Floating IP", going into the Tenant subnet.
The affected topology is: "Per-Tenant Router with Private Networks".
As a reference, I'm using the following Grizzly guide for my Havana deployment:
https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst
Extra info:
http://docs.openstack.org/havana/install-guide/install/apt/content/section_networking-routers-with-private-networks.html
The symptoms are:
1- "Slow connection to Canonical or when browsing the web from within a tenant subnet"
aptitude update ; aptitude safe-upgrade
From within a Tenant instance, it will take about 1 hour to finish, on a link capable of finishing it in 2~3 minutes.
2- SSH connection using Floating IPs froze 10 times per minute.
Connecting from the outside world, into a Instance using its Floating IP address, it is a pain.
We're talking about this issue at the OpenStack mail list, here is the related thread: http://lists.openstack.org/pipermail/openstack/2013-November/002705.html
Also, I made a video about it, watch it here: http://www.youtube.com/watch?v=jVjiphMuuzM
Tested versions:
* OpenStack Havana on top of Ubuntu 12.04.3 using Ubuntu Cloud Archive
* Tested with Open vSwitch versions:
1.10.2 from UCA
1.11.0 compiled for Ubuntu 12.04.3 using "dpkg-buildpackage"
1.9.0 from Ubuntu package "openvswitch-datapath-lts-raring-dkms"
* Not tested:
Havana with Ubuntu 12.04.1 + OVS 1.4.0 (does not support VXLAN).
* Tenant subnet tested types:
VXLAN
GRE
VLAN
It does not matter the subnet type you choose, it will be always slow.
Apparently, if you upgrade your Grizzly from Ubuntu 12.04.1 + OVS 1.4.0, to Ubuntu 12.04.3 with OVS 1.9.0, it will trigger this problem when with Grizzly too. So, I think that this problem might be related to Open vSwitch itself. But I need more time to check this.
My private cloud computing based on Havana is open for you guys to debug it, just ask for an access! =)
My current plan it to test Havana with OVS 1.4.0 but, I don't have too much time this week to do this job.
I'm not sure if the problem is with OVS or not, I'll try to test it this week.
Also, at my video, you guys can see how I "fixed" it, by starting a Squid proxy-cache server within the Tenant Namespece Router, proving that the problem appear ONLY when you try to establish a connection from a tenant subnet, directly to the External network.
I mean, the connection between a tenant and its router is okay, from its router to the Internet, is also okay but, from a tenant to the Internet, is not. So, Squid was a perfect choice to verify this theory at the Namespace router...
Best!
Thiago |
Hello!
Currently, Havana L3 Router have a serious issue. Which makes it almost useless (sorry, I do not want to be rude but instead, trying to bring more attention to this problem).
When the tenant network traffic pass trough the L3 Router (Namespace at the Network Node), it becomes very, very slow and intermittent. The issue also affects the traffic that hit a "Floating IP", going into the Tenant subnet.
The affected topology is: "Per-Tenant Router with Private Networks".
As a reference, I'm using the following Grizzly guide for my Havana deployment:
https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst
Extra info:
http://docs.openstack.org/havana/install-guide/install/apt/content/section_networking-routers-with-private-networks.html
The symptoms are:
1- "Slow connection to Canonical or when browsing the web from within a tenant subnet"
aptitude update ; aptitude safe-upgrade
From within a Tenant instance, it will take about 1 hour to finish, on a link capable of finishing it in 2~3 minutes.
2- SSH connection using Floating IPs froze 10 times per minute.
Connecting from the outside world, into a Instance using its Floating IP address, it is a pain.
We're talking about this issue at the OpenStack mail list, here is the related thread: http://lists.openstack.org/pipermail/openstack/2013-November/002705.html
Also, I made a video about it, watch it here: http://www.youtube.com/watch?v=jVjiphMuuzM
Tested versions:
* OpenStack Havana on top of Ubuntu 12.04.3 using Ubuntu Cloud Archive
* Tested with Open vSwitch versions (none of it works):
1.10.2 from UCA
1.11.0 compiled for Ubuntu 12.04.3 using "dpkg-buildpackage"
1.9.0 from Ubuntu package "openvswitch-datapath-lts-raring-dkms"
* Not tested (maybe it will work):
Havana with Ubuntu 12.04.1 + OVS 1.4.0 (does not support VXLAN).
* Tenant subnet tested types:
VXLAN
GRE
VLAN
It does not matter the subnet type you choose, it will be always slow.
Apparently, if you upgrade your Grizzly from Ubuntu 12.04.1 + OVS 1.4.0, to Ubuntu 12.04.3 with OVS 1.9.0, it will trigger this problem when with Grizzly too. So, I think that this problem might be related to Open vSwitch itself. But I need more time to check this.
My private cloud computing based on Havana is open for you guys to debug it, just ask for an access! =)
My current plan it to test Havana with OVS 1.4.0 but, I don't have too much time this week to do this job.
I'm not sure if the problem is with OVS or not, I'll try to test it this week.
Also, at my video, you guys can see how I "fixed" it, by starting a Squid proxy-cache server within the Tenant Namespece Router, proving that the problem appear ONLY when you try to establish a connection from a tenant subnet, directly to the External network.
I mean, the connection between a tenant and its router is okay, from its router to the Internet, is also okay but, from a tenant to the Internet, is not. So, Squid was a perfect choice to verify this theory at the Namespace router...
Please, let me know what configuration files do you guys will need to be able to reproduce this problem.
Best!
Thiago |
|
| 2013-11-20 00:15:13 |
Thiago Martins |
description |
Hello!
Currently, Havana L3 Router have a serious issue. Which makes it almost useless (sorry, I do not want to be rude but instead, trying to bring more attention to this problem).
When the tenant network traffic pass trough the L3 Router (Namespace at the Network Node), it becomes very, very slow and intermittent. The issue also affects the traffic that hit a "Floating IP", going into the Tenant subnet.
The affected topology is: "Per-Tenant Router with Private Networks".
As a reference, I'm using the following Grizzly guide for my Havana deployment:
https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst
Extra info:
http://docs.openstack.org/havana/install-guide/install/apt/content/section_networking-routers-with-private-networks.html
The symptoms are:
1- "Slow connection to Canonical or when browsing the web from within a tenant subnet"
aptitude update ; aptitude safe-upgrade
From within a Tenant instance, it will take about 1 hour to finish, on a link capable of finishing it in 2~3 minutes.
2- SSH connection using Floating IPs froze 10 times per minute.
Connecting from the outside world, into a Instance using its Floating IP address, it is a pain.
We're talking about this issue at the OpenStack mail list, here is the related thread: http://lists.openstack.org/pipermail/openstack/2013-November/002705.html
Also, I made a video about it, watch it here: http://www.youtube.com/watch?v=jVjiphMuuzM
Tested versions:
* OpenStack Havana on top of Ubuntu 12.04.3 using Ubuntu Cloud Archive
* Tested with Open vSwitch versions (none of it works):
1.10.2 from UCA
1.11.0 compiled for Ubuntu 12.04.3 using "dpkg-buildpackage"
1.9.0 from Ubuntu package "openvswitch-datapath-lts-raring-dkms"
* Not tested (maybe it will work):
Havana with Ubuntu 12.04.1 + OVS 1.4.0 (does not support VXLAN).
* Tenant subnet tested types:
VXLAN
GRE
VLAN
It does not matter the subnet type you choose, it will be always slow.
Apparently, if you upgrade your Grizzly from Ubuntu 12.04.1 + OVS 1.4.0, to Ubuntu 12.04.3 with OVS 1.9.0, it will trigger this problem when with Grizzly too. So, I think that this problem might be related to Open vSwitch itself. But I need more time to check this.
My private cloud computing based on Havana is open for you guys to debug it, just ask for an access! =)
My current plan it to test Havana with OVS 1.4.0 but, I don't have too much time this week to do this job.
I'm not sure if the problem is with OVS or not, I'll try to test it this week.
Also, at my video, you guys can see how I "fixed" it, by starting a Squid proxy-cache server within the Tenant Namespece Router, proving that the problem appear ONLY when you try to establish a connection from a tenant subnet, directly to the External network.
I mean, the connection between a tenant and its router is okay, from its router to the Internet, is also okay but, from a tenant to the Internet, is not. So, Squid was a perfect choice to verify this theory at the Namespace router...
Please, let me know what configuration files do you guys will need to be able to reproduce this problem.
Best!
Thiago |
Hello!
Currently, Havana L3 Router have a serious issue. Which makes it almost useless (sorry, I do not want to be rude but instead, trying to bring more attention to this problem).
When the tenant network traffic pass trough the L3 Router (Namespace at the Network Node), it becomes very, very slow and intermittent. The issue also affects the traffic that hit a "Floating IP", going into the Tenant subnet.
The affected topology is: "Per-Tenant Router with Private Networks".
As a reference, I'm using the following Grizzly guide for my Havana deployment:
https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst
Extra info:
http://docs.openstack.org/havana/install-guide/install/apt/content/section_networking-routers-with-private-networks.html
The symptoms are:
1- "Slow connection to Canonical or when browsing the web from within a tenant subnet"
aptitude update ; aptitude safe-upgrade
From within a Tenant instance, it will take about 1 hour to finish, on a link capable of finishing it in 2~3 minutes.
2- SSH connection using Floating IPs froze 10 times per minute.
Connecting from the outside world, into a Instance using its Floating IP address, is a pain.
We're talking about this issue at the OpenStack mail list, here is the related thread: http://lists.openstack.org/pipermail/openstack/2013-November/002705.html
Also, I made a video about it, watch it here: http://www.youtube.com/watch?v=jVjiphMuuzM
Tested versions:
* OpenStack Havana on top of Ubuntu 12.04.3 using Ubuntu Cloud Archive
* Tested with Open vSwitch versions (none of it works):
1.10.2 from UCA
1.11.0 compiled for Ubuntu 12.04.3 using "dpkg-buildpackage"
1.9.0 from Ubuntu package "openvswitch-datapath-lts-raring-dkms"
* Not tested (maybe it will work):
Havana with Ubuntu 12.04.1 + OVS 1.4.0 (does not support VXLAN).
* Tenant subnet tested types:
VXLAN
GRE
VLAN
It does not matter the subnet type you choose, it will be always slow.
Apparently, if you upgrade your Grizzly from Ubuntu 12.04.1 + OVS 1.4.0, to Ubuntu 12.04.3 with OVS 1.9.0, it will trigger this problem when with Grizzly too. So, I think that this problem might be related to Open vSwitch itself. But I need more time to check this.
My private cloud computing based on Havana is open for you guys to debug it, just ask for an access! =)
My current plan it to test Havana with OVS 1.4.0 but, I don't have too much time this week to do this job.
I'm not sure if the problem is with OVS or not, I'll try to test it this week.
Also, at my video, you guys can see how I "fixed" it, by starting a Squid proxy-cache server within the Tenant Namespece Router, proving that the problem appear ONLY when you try to establish a connection from a tenant subnet, directly to the External network.
I mean, the connection between a tenant and its router is okay, from its router to the Internet, is also okay but, from a tenant to the Internet, is not. So, Squid was a perfect choice to verify this theory at the Namespace router...
Please, let me know what configuration files do you guys will need to be able to reproduce this problem.
Best!
Thiago |
|
| 2013-11-20 00:19:30 |
Thiago Martins |
bug task added |
|
openvswitch |
|
| 2013-11-20 00:20:14 |
Thiago Martins |
tags |
|
l3 namespace neutron openstack openvswitch |
|
| 2013-11-20 00:46:26 |
Geraint Jones |
bug task added |
|
ubuntu |
|
| 2013-11-20 00:52:27 |
Geraint Jones |
bug |
|
|
added subscriber Geraint Jones |
| 2013-11-20 01:55:26 |
Koji Iida |
bug |
|
|
added subscriber Koji Iida |
| 2013-11-20 06:05:01 |
Thiago Martins |
description |
Hello!
Currently, Havana L3 Router have a serious issue. Which makes it almost useless (sorry, I do not want to be rude but instead, trying to bring more attention to this problem).
When the tenant network traffic pass trough the L3 Router (Namespace at the Network Node), it becomes very, very slow and intermittent. The issue also affects the traffic that hit a "Floating IP", going into the Tenant subnet.
The affected topology is: "Per-Tenant Router with Private Networks".
As a reference, I'm using the following Grizzly guide for my Havana deployment:
https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst
Extra info:
http://docs.openstack.org/havana/install-guide/install/apt/content/section_networking-routers-with-private-networks.html
The symptoms are:
1- "Slow connection to Canonical or when browsing the web from within a tenant subnet"
aptitude update ; aptitude safe-upgrade
From within a Tenant instance, it will take about 1 hour to finish, on a link capable of finishing it in 2~3 minutes.
2- SSH connection using Floating IPs froze 10 times per minute.
Connecting from the outside world, into a Instance using its Floating IP address, is a pain.
We're talking about this issue at the OpenStack mail list, here is the related thread: http://lists.openstack.org/pipermail/openstack/2013-November/002705.html
Also, I made a video about it, watch it here: http://www.youtube.com/watch?v=jVjiphMuuzM
Tested versions:
* OpenStack Havana on top of Ubuntu 12.04.3 using Ubuntu Cloud Archive
* Tested with Open vSwitch versions (none of it works):
1.10.2 from UCA
1.11.0 compiled for Ubuntu 12.04.3 using "dpkg-buildpackage"
1.9.0 from Ubuntu package "openvswitch-datapath-lts-raring-dkms"
* Not tested (maybe it will work):
Havana with Ubuntu 12.04.1 + OVS 1.4.0 (does not support VXLAN).
* Tenant subnet tested types:
VXLAN
GRE
VLAN
It does not matter the subnet type you choose, it will be always slow.
Apparently, if you upgrade your Grizzly from Ubuntu 12.04.1 + OVS 1.4.0, to Ubuntu 12.04.3 with OVS 1.9.0, it will trigger this problem when with Grizzly too. So, I think that this problem might be related to Open vSwitch itself. But I need more time to check this.
My private cloud computing based on Havana is open for you guys to debug it, just ask for an access! =)
My current plan it to test Havana with OVS 1.4.0 but, I don't have too much time this week to do this job.
I'm not sure if the problem is with OVS or not, I'll try to test it this week.
Also, at my video, you guys can see how I "fixed" it, by starting a Squid proxy-cache server within the Tenant Namespece Router, proving that the problem appear ONLY when you try to establish a connection from a tenant subnet, directly to the External network.
I mean, the connection between a tenant and its router is okay, from its router to the Internet, is also okay but, from a tenant to the Internet, is not. So, Squid was a perfect choice to verify this theory at the Namespace router...
Please, let me know what configuration files do you guys will need to be able to reproduce this problem.
Best!
Thiago |
Hello!
Currently, Havana L3 Router have a serious issue. Which makes it almost useless (sorry, I do not want to be rude but instead, trying to bring more attention to this problem).
When the tenant network traffic pass trough the L3 Router (Namespace at the Network Node), it becomes very, very slow and intermittent. The issue also affects the traffic that hit a "Floating IP", going into the Tenant subnet.
The affected topology is: "Per-Tenant Router with Private Networks".
As a reference, I'm using the following Grizzly guide for my Havana deployment:
https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst
Extra info:
http://docs.openstack.org/havana/install-guide/install/apt/content/section_networking-routers-with-private-networks.html
The symptoms are:
1- "Slow connection to Canonical or when browsing the web from within a tenant subnet"
aptitude update ; aptitude safe-upgrade
From within a Tenant instance, it will take about 1 hour to finish, on a link capable of finishing it in 2~3 minutes.
2- SSH connection using Floating IPs froze 10 times per minute.
Connecting from the outside world, into a Instance using its Floating IP address, is a pain.
We're talking about this issue at the OpenStack mail list, here is the related thread: http://lists.openstack.org/pipermail/openstack/2013-November/002705.html
Also, I made a video about it, watch it here: http://www.youtube.com/watch?v=jVjiphMuuzM
Tested versions:
* OpenStack Havana on top of Ubuntu 12.04.3 using Ubuntu Cloud Archive
* Tested with Open vSwitch versions (none of it works):
1.10.2 from UCA
1.11.0 compiled for Ubuntu 12.04.3 using "dpkg-buildpackage"
1.9.0 from Ubuntu package "openvswitch-datapath-lts-raring-dkms"
* Not tested (maybe it will work):
Havana with Ubuntu 12.04.1 + OVS 1.4.0 (does not support VXLAN).
* Tenant subnet tested types:
VXLAN
GRE
VLAN
It does not matter the subnet type you choose, it will be always slow.
Apparently, if you upgrade your Grizzly from Ubuntu 12.04.1 + OVS 1.4.0, to Ubuntu 12.04.3 with OVS 1.9.0, it will trigger this problem when with Grizzly too. So, I think that this problem might be related to Open vSwitch itself. But I need more time to check this.
My private cloud computing based on Havana is open for you guys to debug it, just ask for an access! =)
My current plan it to test Havana with OVS 1.4.0 but, I don't have too much time this week to do this job.
I'm not sure if the problem is with OVS or not, I'll try to test it this week.
Also, at my video, you guys can see how I "fixed" it, by starting a Squid proxy-cache server within the Tenant Namespece Router, proving that the problem appear ONLY when you try to establish a connection from a tenant subnet, directly to the External network.
I mean, the connection between a tenant and its router is okay, from its router to the Internet, is also okay but, from a tenant to the Internet, is not. So, Squid was a perfect choice to verify this theory at the Namespace router... And Voialá! "There I fixed it"! =P
Please, let me know what configuration files do you guys will need to be able to reproduce this problem.
Best!
Thiago |
|
| 2013-11-20 20:02:41 |
Launchpad Janitor |
ubuntu: status |
New |
Confirmed |
|
| 2013-11-20 20:03:01 |
Daniel Speichert |
bug |
|
|
added subscriber Daniel Speichert |
| 2013-11-21 08:52:56 |
Darragh O'Reilly |
bug |
|
|
added subscriber Darragh O'Reilly |
| 2013-11-21 10:26:56 |
Édouard Thuleau |
bug |
|
|
added subscriber Édouard Thuleau |
| 2013-11-22 22:21:24 |
Michael H Wilson |
neutron: status |
New |
Confirmed |
|
| 2013-11-23 07:26:20 |
Tomoko Inoue |
bug |
|
|
added subscriber Tomoko Inoue |
| 2013-11-24 21:21:27 |
Thiago Martins |
attachment added |
|
Tarball file from tcpdump of qr- and qg- Neutron interfaces during the network outage https://bugs.launchpad.net/neutron/+bug/1252900/+attachment/3916391/+files/qg-qr-tcpdump-data.tar |
|
| 2013-11-25 16:13:51 |
Ronan Lanore |
bug |
|
|
added subscriber Ronan Lanore |
| 2013-11-25 21:07:27 |
KaZeR |
bug |
|
|
added subscriber KaZeR |
| 2013-11-26 10:44:25 |
Darragh O'Reilly |
bug task added |
|
openstack-manuals |
|
| 2013-11-26 10:44:37 |
Darragh O'Reilly |
openstack-manuals: assignee |
|
Darragh O'Reilly (darragh-oreilly) |
|
| 2013-11-26 14:45:45 |
GMi |
bug |
|
|
added subscriber GMi |
| 2013-11-26 19:12:13 |
Darragh O'Reilly |
openstack-manuals: status |
New |
In Progress |
|
| 2013-11-27 01:08:36 |
OpenStack Infra |
openstack-manuals: status |
In Progress |
Fix Released |
|
| 2013-11-27 07:48:59 |
OpenStack Infra |
tags |
l3 namespace neutron openstack openvswitch |
in-stable-havana l3 namespace neutron openstack openvswitch |
|
| 2013-12-03 17:58:23 |
Jean-Sebastien Mouret |
bug |
|
|
added subscriber Jean-Sebastien Mouret |
| 2013-12-16 12:58:42 |
Alan Pevec |
tags |
in-stable-havana l3 namespace neutron openstack openvswitch |
l3 namespace neutron openstack openvswitch |
|
| 2014-01-03 10:07:02 |
Lei Wang |
bug |
|
|
added subscriber Ray Wang |
| 2014-01-31 09:35:12 |
Pellaeon Lin |
bug |
|
|
added subscriber Pellaeon Lin |
| 2014-03-06 06:22:40 |
Xiang Hui |
bug |
|
|
added subscriber Xiang Hui |
| 2014-05-06 11:57:09 |
Ghe Rivero |
bug |
|
|
added subscriber Ghe Rivero |
| 2014-09-02 04:33:13 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto MURATA |
| 2015-03-25 20:37:09 |
Stephen Gordon |
bug |
|
|
added subscriber Stephen Gordon |
| 2016-03-12 01:30:04 |
Armando Migliaccio |
neutron: status |
Confirmed |
Incomplete |
|
| 2018-07-03 04:36:44 |
Lei Wang |
removed subscriber Ray Wang |
|
|
|
| 2020-02-07 18:52:12 |
Brian Haley |
neutron: status |
Incomplete |
Won't Fix |
|
