container addressability: lxc/lxd units are behind NAT on manual and openstack providers

Bug #1614364 reported by Ryan Beisner on 2016-08-18
76
This bug affects 14 people
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Critical
Unassigned
juju
High
John A Meinel
2.1
High
Unassigned
2.2
Undecided
Unassigned
juju-core
Critical
Unassigned
1.25
Critical
Unassigned

Bug Description

1.25.6: Charm applications deployed to lxc units on multiple manual machines with the manual provider are guaranteed to fail by default.

This is because the lxc units sit behind a NAT bridge interface on each manual machine. The lxc units are not reachable from the controller, and lxc units on a manual machine cannot communicate with lxc units on another manual machine.

An over-simplification of what I'm seeing:

### One Simple Network
192.168.100.0/24

### Bastion (bootstrapped here) - 16.04
This could be your laptop.
192.168.100.10/24

### Machine 1 - 16.04
192.168.100.11/24

  1/lxc/0:
  10.0.3.12/24

  1/lxc/1:
  10.0.3.13/24

### Machine 2 - 16.04
192.168.100.12/24

  2/lxc/0:
  10.0.3.12/24

  2/lxc/1:
  10.0.3.15/24

### Machine 3 - 16.04
192.168.100.13/24

  3/lxc/0:
  10.0.3.13/24

  3/lxc/1:
  10.0.3.22/24

I think a more sane default behavior for the manual provider would be to configure the bridge as a pure L2 ('transparent') bridge, similar to what the Juju MAAS provider creates.

This would require that the user have pre-existing DHCP and DNS services ready on the network in advance. But I think that is in line with the spirit of the manual provider, and that can be documented accordingly.

If this turns out not to be something that is addressed, the docs should be updated to indicate --to lxc:foo is not supported with the manual provider in a default machine configuration.

Changed in juju-core:
status: New → Triaged
importance: Undecided → High
milestone: none → 2.0-beta17
Ryan Beisner (1chb1n) wrote :

For a real example, manual reproducer, juju status and connectivity checks, see the attached file.

It's two machines, 10 containers on each machine.

From the "LAN," all containers are unreachable, metal hosts are reachable.

From a metal host, containers on other metal hosts are unreachable, containers on that metal host are reachable.

From any given container, containers on other metal hosts are unreachable, containers on that metal host are reachable.

This is all as expected given the L3 NAT.

affects: juju-core → juju
Changed in juju:
milestone: 2.0-beta17 → none
milestone: none → 2.0-beta17
Ryan Beisner (1chb1n) wrote :

Tagging with s390x for tracking purposes, although this also impacts amd64.

tags: added: amd64 s390x
Changed in juju-core:
importance: Undecided → Critical
status: New → Triaged
Ryan Beisner (1chb1n) wrote :

Workaround:

Pre-configure the bridge on each host as a true L2 (transparent) bridge before adding the machines into the model/environment. Units deployed to lxc/lxd will then be on the same L2 network as their host, and will obtain IP addresses via DHCP. This assumes that there is a DHCP server on the host's connected broadcast domain which is present, ready and willing to serve. :)

Changed in juju:
status: Triaged → Invalid
Changed in juju:
milestone: 2.0-beta17 → none
Changed in juju-core:
status: Triaged → Won't Fix
Andrew Cloke (andrew-cloke) wrote :

As Juju manual provider is currently the only option for IBM z, this issue gives openstack deployments on z a less than ideal user experience. My understanding is that the workaround is not something that would be advised outside of a development environment.

Could you expand on the rationale for the decision not to fix?

Just to be clear, this bug has been marked invalid for 2.0 as it lxc + 1.25; it is still marked and considered critical for 1.25.7

Ryan Beisner (1chb1n) on 2016-09-09
description: updated
Lou Peers (louie-pe) wrote :

Is there a different bug we should be tracking? Also can we have an update to indicate when this might be looked at?
Thank you!

Anastasia (anastasia-macmood) wrote :

We are planning to address this bug for the next release of 1.25 - 1.25.7.

Renamed to be about lxd as well. The idea is that containers on providers that don't support spaces don't have the ability to get a DHCP address on the host network.

This requires changes to the way 1.25 works which is in critical only support and so marked wontfix for 1.25.

However, for 2.0 we need to support networking to allow the user to let Juju know there's something on the network that can provide host-level addresses for containers. This would then work across any provider.

summary: - manual provider lxc units are behind NAT, fail by default
+ manual provider lxc/lxd units are behind NAT, fail by default
Changed in juju:
status: Invalid → Triaged
milestone: none → 2.2.0
Ryan Beisner (1chb1n) wrote :

Ok. So imho, the 2.0-equivalent of this should be just as critical (a s390x Multi-LPAR OpenStack deploy blocker). Unless a human goes and manually configures bridges on all machines before the Juju manual provider is in the picture, the same containers-behind-NAT situation exists.

tags: added: repeatability
Changed in juju:
importance: High → Critical
milestone: 2.2.0 → 2.0.3
assignee: nobody → Tim Penhey (thumper)
Changed in juju:
assignee: Tim Penhey (thumper) → nobody
importance: Critical → High
milestone: 2.0.3 → 2.2.0
John A Meinel (jameinel) wrote :

Here are my thoughts
This can't be a critical as we clearly haven't stopped the line. And there is a reasonably simple workaround (juju run lxd init) and reconfigure the bridge. It's fully script able and if you are doing the manual provider I would be surprised if you have that many machines without scripting around adding them anyway.

That said, bridging to the host nic seems the more sane default. If you don't have dhcp available you'll get unusable containers but they are unusable anyway if they are hidden.

But just doing that really doesn't solve manual issues, because we haven't really gotten them into the model. What happens if you add a machine but when you get there you find there are 2 NICs

Even if you have added the 2 NIC machine we don't have a way to describe where you want workloads to listen.

We also don't have anything like a description of storage on the machine. Where to get more IP addresses, etc. All the things that really bring a machine into the Juju Model.

Now if there is a stakeholder escalation then we can have this critical and focus on it, but standard bug triage wouldn't put it there.

Andrew Cloke (andrew-cloke) wrote :

I would like to make a stakeholder escalation. This bug impacts running openstack on IBM z using a single or multiple LPARs and LXD containers. The automation required around openstack installation makes that the workload around impracticable. It is not something that we would want to recommend.

cargonza (cargonza) wrote :

Hi, any decision on the path for this bug? A stakeholder escalation has been requested and we would like to get an update on the next steps. Thank you!

Richard Harding (rharding) wrote :

At the moment we're in the state that John notes. We have a work around and we've got work in flight to improve container networking that the team is focused on. Unfortunately, this isn't a single "patch a bug" type of fix. Our goal is to get improved container networking, primarily at correcting issues on MAAS, for the 2.1 release before the holiday break.

We will take this bug and attempt to provide a tasteful path forward as part of that work.

Changed in juju:
milestone: 2.2.0 → 2.1.0
assignee: nobody → Richard Harding (rharding)
tags: added: openstack-ibm
Ryan Beisner (1chb1n) wrote :

Confirmed s390x multi-lpar blocker @ juju 2.1~beta4-0ubuntu1~16.04.1~juju1.

FWIW, we were able to work around this with Juju 1 by carefully crafting the bridge in advance to not do NAT.

For Juju2, we've not found a successful workaround yet. See bugs raised in attempts to work around this issue in Juju2:

https://bugs.launchpad.net/juju/+bug/1655224
https://bugs.launchpad.net/juju/+bug/1655229
https://bugs.launchpad.net/juju/+bug/1575676

tags: added: multi-lpar
Ryan Beisner (1chb1n) on 2017-01-11
Changed in ubuntu-z-systems:
status: New → Confirmed
summary: - manual provider lxc/lxd units are behind NAT, fail by default
+ juju1 and juju2 - manual provider lxc/lxd units are behind NAT, fail by
+ default

To be clear, this affects s390x, but is not specific to s390x. The behavior is reproducible on any arch that I have tried.

Anastasia (anastasia-macmood) wrote :

Changed to 'Critical' as attempts to workaround this issue cause even more issues as per Ryan's comment # 16.

Changed in juju:
importance: High → Critical
Changed in ubuntu-z-systems:
importance: Undecided → Critical
status: Confirmed → Triaged
Changed in juju:
assignee: Richard Harding (rharding) → John A Meinel (jameinel)
Changed in juju:
milestone: 2.1.0 → 2.1-rc1
Changed in juju:
milestone: 2.1-rc1 → 2.2.0-alpha1
Anastasia (anastasia-macmood) wrote :

As this is not likely to be addressed in 2.1, we are removing it from current milestone.

Curtis Hovey (sinzui) on 2017-02-16
summary: - juju1 and juju2 - manual provider lxc/lxd units are behind NAT, fail by
- default
+ container addresability: manual provider lxc/lxd units are behind NAT,
+ fail by default on juju1 and juju2
summary: - container addresability: manual provider lxc/lxd units are behind NAT,
+ container addressability: manual provider lxc/lxd units are behind NAT,
fail by default on juju1 and juju2

Updating description as the corresponding Juju openstack-provider bug [1] was marked as a duplicate.
https://bugs.launchpad.net/juju/+bug/1615917

tags: added: openstack-provider
summary: - container addressability: manual provider lxc/lxd units are behind NAT,
- fail by default on juju1 and juju2
+ container addressability: lxc/lxd units are behind NAT on manual and
+ openstack providers
John A Meinel (jameinel) wrote :

At this point, I won't be able to address this for 2.1.1, I should, however, be able to get it for 2.2. Doing this correctly is going to need a fair bit of testing, and it isn't reasonable to put it directly into a stable 2.1 series without that testing. As 2.2 is likely to be hot on the heels of 2.1.1 anyway, it doesn't really make sense to target a 2.1.X series, but *if* we've proven the work in 2.2, I'll likely make sure to do it in a way that we *could* backport it to 2.1 if there is reason to do so.

Anastasia (anastasia-macmood) wrote :

Marking as Fix Committed for 2.1.1 as we have put in a solution deemed sufficient by stakeholders. The fix requires to set the bridge up in advance.

A follow-on fix is being worked on as part of 2.2, comment # 21, that will not require the bridge in advance.

Ryan Beisner (1chb1n) wrote :

As a stakeholder in this issue, I'm not in agreement with #22. 'Wont-fix' is a more accurate triage level for 2.1 based on the stated timeline.

Anastasia (anastasia-macmood) wrote :

@Ryan Beisner (1chb1n),
Stakeholder endorsement I referred to was from an internal conversation with a different group, not Openstack/OSCI.
Are you saying that the fix that went in into 2.1-beta5, that required bridge to be set up prior and outside of Juju, does not address this issue for you?

We are saying that we have tackled it partially: the behavior and the workaround are improved. This got some of the people affected un-stuck enough - they can proceed. We are working on a cleaner solution for later releases.

Ryan Beisner (1chb1n) wrote :

The essence of this bug is that containers are behind NAT by default on the manual provider and the openstack provider. As far as I'm aware, there have been no commits to change that behavior.

Anastasia (anastasia-macmood) wrote :

Marking as Won't Fix for 2.1 as per recommendation and feedback around workaround.

Anastasia (anastasia-macmood) wrote :

Due to time and resource constraints, moving this issue into next release.

Changed in juju:
milestone: 2.2-alpha1 → 2.3.0
Curtis Hovey (sinzui) on 2017-04-08
Changed in juju:
importance: Critical → High
Tim Penhey (thumper) wrote :

John, is this now solved with the FAN config?

Changed in juju:
milestone: 2.3.0 → 2.3-beta3
John A Meinel (jameinel) on 2017-11-09
Changed in juju:
status: Triaged → Fix Committed
status: Fix Committed → Triaged
Changed in juju:
milestone: 2.3-beta3 → none
Frank Heimes (frank-heimes) wrote :

bump

Is there a revised plan to provide this improved container addressability?

In 2.3 you should be able to set 'container-networking-method=provider' and
then we will try to allocate IP addresses for containers via DHCP. It has
not been tested very rigorously by us, so there may be issues that we are
missing. (manual machines and spaces are generally not tracked very well,
so determining what network device needs to be bridged into the containers
could easily be a missing linkage.)
Theoretically with manual bootstrap and provisioning you would be able to
"juju add-space NAME CIDR" to declare a space for one of the network
devices.

On Tue, Feb 27, 2018 at 6:55 PM, Frank Heimes <email address hidden>
wrote:

> bump
>
> Is there a revised plan to provide this improved container
> addressability?
>
> --
> You received this bug notification because you are a bug assignee.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1614364
>
> Title:
> container addressability: lxc/lxd units are behind NAT on manual and
> openstack providers
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1614364/+subscriptions
>

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers