split network environment

[Paolo de Rosa]

In some environments traffic segregation is needed due to security reasons, it would be good if kafka charm let us to decide where to bind kafka.
It would be useful to have an option in the charm where to select on which interface/network we can bind the service without relying on "unit-get private-address".

[Pen Gale (pengale)]

Zookeeper should do this, too.

[Cory Johns (johnsca)]

New development focus for big data is on the Apache Bigtop version of the charms (since that better aligns with community best practices, eliminates a lot of redundant work / code, and ensures we are using version of services and deployment logic that ensure that the components all work together).

For Kakfa, this means you should instead use the charm at https://jujucharms.com/kafka/ but this should function exactly the same as the apache-kafka charm, so the transition should be trivial.

The work to add the option for controlling the bind host.name is currently being reviewed in the following PRs, and should be available as soon as they are signed off on and the charm store is updated (I expect first thing next week):

* https://github.com/juju-solutions/bigtop/pull/28
* https://github.com/juju-solutions/bigtop/pull/27
* https://github.com/juju-solutions/layer-apache-bigtop-base/pull/32

[Lakdar Benasroune (lakdar-benasroune)]

Thx Cory , the same for Zookeeper right ?

https://bugs.launchpad.net/charms/+source/zookeeper/+bug/1603007
https://bugs.launchpad.net/charms/+source/zookeeper/+bug/1604448

On Fri, Jul 22, 2016 at 4:23 PM, Cory Johns <email address hidden>
wrote:

> New development focus for big data is on the Apache Bigtop version of
> the charms (since that better aligns with community best practices,
> eliminates a lot of redundant work / code, and ensures we are using
> version of services and deployment logic that ensure that the components
> all work together).
>
> For Kakfa, this means you should instead use the charm at
> https://jujucharms.com/kafka/ but this should function exactly the same
> as the apache-kafka charm, so the transition should be trivial.
>
> The work to add the option for controlling the bind host.name is
> currently being reviewed in the following PRs, and should be available
> as soon as they are signed off on and the charm store is updated (I
> expect first thing next week):
>
> * https://github.com/juju-solutions/bigtop/pull/28
> * https://github.com/juju-solutions/bigtop/pull/27
> * https://github.com/juju-solutions/layer-apache-bigtop-base/pull/32
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1602666
>
> Title:
> split network environment
>
> Status in apache-kafka package in Juju Charms Collection:
> New
>
> Bug description:
> In some environments traffic segregation is needed due to security
> reasons, it would be good if kafka charm let us to decide where to bind
> kafka.
> It would be useful to have an option in the charm where to select on
> which interface/network we can bind the service without relying on
> "unit-get private-address".
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/charms/+source/apache-kafka/+bug/1602666/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: distribution=charms; sourcepackage=apache-kafka;
> component=None; status=New; importance=Undecided; assignee=None;
> Launchpad-Bug-Tags: 4010
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: johnsca paolo-de-rosa petevg
> Launchpad-Bug-Reporter: Paolo de Rosa (paolo-de-rosa)
> Launchpad-Bug-Modifier: Cory Johns (johnsca)
> Launchpad-Message-Rationale: Subscriber
> Launchpad-Message-For: lakdar-benasroune
>

--
Lakdar Benasroune
Canonical Ltd. Germany
fon +49 151.22.90.78.10 <http://[phone]/>
email <email address hidden>

[Cory Johns (johnsca)]

https://jujucharms.com/kafka/trusty/4 is now available with the bind_addr option.

Zookeeper is currently being worked on and should be available later this week. However, I will be out the rest of this week and part of next, so petevg and lazyPower will be the points of contact
for that work.

[Kevin W Monroe (kwmonroe)]

Reopening this. We cannot use an IP address for the bind_addr config option. Config is applied across all units; IPs are unique. We'll need to refactor this (and the zk implementation) so bind_addr takes a network (10.0.0.0/8) or interface name (eth0).

The charm will then need to detect whether or not it has an IP in the given range (or on the given interface) and do the kafka config accordingly.

[Kevin W Monroe (kwmonroe)]

Apologies for the delayed response here, but the issue raised in comment #5 was addressed in the following PRs:

https://github.com/juju-solutions/bigtop/pull/32
https://github.com/juju-solutions/bigtop/pull/34

Version 5 of the bigtop kafka charm supports binding kafka to a specific ip or interface:

https://jujucharms.com/kafka/trusty/5

Note!!
There are 2 Kafka charms in play here. "kafka" provides the Bigtop kafka package, but will not work in a network restricted environment. "apache-kafka" provides the vanilla upstream kafka package, and does have a mechanism for installing in a net-restricted env.

Only "kafka" contains this interface binding functionality. I'll leave this bug opened to track the progress of bringing this feature into the "apache-kafka" charm.

[Pen Gale (pengale)]

PR with changes to the apache-kafka charm here: https://github.com/juju-solutions/layer-apache-kafka/pull/8

[Cory Johns (johnsca)]

The charm has been updated at https://jujucharms.com/apache-kafka/ (cs:trusty/apache-kafka-5) with this support.

[Pen Gale (pengale)]

New apache-kafka charm published, with the fix: cs:trusty/apache-kafka-5

[Pen Gale (pengale)]

This issue should probably re-opened, as there are some issues w/ the way that the revised kafka charm handles bindings.

I just pushed an updated kafka charm to bigdata-dev: cs:~bigdata-dev/trusty/apache-kafka-18

The kafka relation should now contain the bound ip address, and the charm should handle ipv6 local addresses without throwing an Exception.

Kafka is picky about how you talk to it, though. It wants clients to use the advertised hostname, so clients would need to map the ip address they got from the relation to the kafka advertised hostname in order to talk to Kafka successfully. This is still either a manual process, or a process that requires dev work on the charm that you're relating to Kafka.

[Pen Gale (pengale)]

I just published another dev version of this charm, with updated spot fixes: cs:~bigdata-dev/trusty/apache-zookeeper-23