Support for multiple L3 networks

[Ante Karamatić]

Ceph should support deployments on different L3 networks. At the moment, one can only specify one network for both ceph-cluster-network and ceph-public-network. Since Ceph OSDs can be deployed in different L3 networks, one should be able to configure these two config options in a way to support multiple networks. Something like:

ceph-public-network: 172.16.1.0/24 172.16.2.0/24
ceph-cluster-network: 192.168.1.0/24 192.168.2.0/24

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/295510

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/295510
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-mon/commit/?id=b497f4de1ea2658df3ba39e979ddae014a75980d
Submitter: Jenkins
Branch: master

commit b497f4de1ea2658df3ba39e979ddae014a75980d
Author: Chris MacNaughton <email address hidden>
Date: Mon Mar 21 16:51:09 2016 -0400

    Merge ceph charm into ceph-mon

    Squashed commit of the following:

    commit 9b832d9391f9fea2d1491d01da6101585930fc75
    Merge: e2432c4 7b36210
    Author: Chris MacNaughton <email address hidden>
    Date: Mon Mar 21 16:40:54 2016 -0400

        Merge branch 'master' of github.com:openstack/charm-ceph into charm-ceph-mon

        Change-Id: I42cfe6f1e5887627981f8ce4beff164803cc3957

    commit 7b36210bac5bef3bacae2614995e123ef926453f
    Author: Chris Holcombe <email address hidden>
    Date: Fri Mar 18 15:37:06 2016 -0700

        Add ceph-osd to ceph

        This change adds ceph-osd back into ceph for amulet testing.

        Change-Id: Ice4aaf7739e8c839189313d3f6175a834cf64219

    commit e87e0b7bd22fe5ccae2aafcf6bd30f145405e01b
    Author: Ryan Beisner <email address hidden>
    Date: Wed Mar 16 17:33:48 2016 +0000

        Update amulet test to include a non-existent osd-devices value

        The osd-devices charm config option is a whitelist, and the
        charm needs to gracefully handle items in that whitelist which
        may not exist.

        Change-Id: I5f9c6c1e4519fd671d6d36b415c9c8f763495dad

    commit ffce15d52333de4063d04b808cfbca5d890fb996
    Merge: fe8bf6e 9614896
    Author: Jenkins <email address hidden>
    Date: Wed Mar 16 17:45:25 2016 +0000

        Merge "Revert "Make 'blocked' status when node have no storage device""

    commit 961489609d85851bd63c6825339a296bdf74e320
    Author: Chris Holcombe <email address hidden>
    Date: Wed Mar 16 16:55:02 2016 +0000

        Revert "Make 'blocked' status when node have no storage device"

        This reverts commit fc04dd0fff33639b812627d04645134dd7d4d3de.

        Change-Id: I9efbf623fc9aa6096725a15e53df426739ac16ff

    commit fe8bf6e4a5cb466a5efc6403c215e7aece2c6b9c
    Author: Billy Olsen <email address hidden>
    Date: Tue Mar 15 20:08:20 2016 -0700

        Use tox in Makefile targets

        Modify the Makefile to point at the appropriate tox targets
        so that tox and Make output can be equivalent. This involves
        mapping the lint target to the pep8 target and the test target
        to the py27 target.

        Change-Id: I99761d2fdf120bacff58d0aa5c2e584382c2e72b

    commit fc04dd0fff33639b812627d04645134dd7d4d3de
    Author: Seyeong Kim <email address hidden>
    Date: Fri Mar 11 06:07:52 2016 +0000

        Make 'blocked' status when node have no storage device

        Currently there is an msg for no storage status on
        ceph node. But it doesn't make this charm state
        'blocked'.

        is_storage_fine function has been created to check
        storage devices on ceph_hooks.py and using it on
        assess_status.

        Change-Id...

[Bruno Carvalho (brunowcs)]

Hi guys, I`m using ceph-osd charm xenial rev 31, but my multi network do not work.

Juju charm accept this configuration with multiple networks, but not apply in ceph.conf only first network is configured in ceph.conf

Question: it can happen because my new network yet not is configured in interface linux?

Tks

[Bruno Carvalho (brunowcs)]

I´m resolved change this code, it´s working. You can create this official charm fix?

def get_networks(config_opt='ceph-public-network'):
    """Get all configured networks from provided config option.

    If public network(s) are provided, go through them and return those for
    which we have an address configured.
    """
    networks = config(config_opt)
    if networks:
        - networks = networks.split()
        + networks = networks.split(',')
        return [n for n in networks if get_address_in_network(n)]

    return []

def get_network_addrs(config_opt):
    """Get all configured public networks addresses.

    If public network(s) are provided, go through them and return the
    addresses we have configured on any of those networks.
    """
    addrs = []
    networks = config(config_opt)
    if networks:
        - networks = networks.split()
        + networks = networks.split(',')
        addrs = [get_address_in_network(n) for n in networks]
        addrs = [a for a in addrs if a]

[Dmitrii Shcherbakov (dmitriis)]

Hi Bruno,

If you are using MAAS you should use Juju network spaces instead of charm config.

https://docs.jujucharms.com/2.4/en/network-spaces

Network spaces in Juju allow you to bind endpoints defined in metadata.yaml to network spaces defined in MAAS ("public" and "cluster" endpoints in case of ceph). Network spaces aggregate VLANs (with their assigned subnets) into a single logical construct which is similar to a routing domain or VRF. In other words, one network space == one L3 network.

https://github.com/openstack/charm-ceph-osd/blob/stable/18.08/metadata.yaml#L24-L26

When you deploy, in a bundle you would specify something like this:

applications:
  ceph-osd:
  # ...
  bindings:
    # <endpoint name in metadata.yaml> : <space name from MAAS>
    public: ceph-public-space
    cluster: ceph-cluster-space

  ceph-mon:
  # ...
  bindings:
    # <endpoint name in metadata.yaml> : <space name from MAAS>
    public: ceph-public-space
    cluster: ceph-cluster-space

Ceph charms use network-get hook tool to retrieve an "ingress-address" for each unit. ingress-address is taken from a node and nodes are allocated by MAAS and receive relevant addresses based on subnets and VLANs they have network ports on. So, if you have a multi-subnet setup in MAAS network-get will always give you the right address which will be put into public_addr or cluster_addr options in ceph.conf by charms. Ceph has a number of options for a multi-network setup and a complex logic (*) to select among the different config options. Ceph charms rely on public_addr and cluster_addr when network spaces are used and you need to avoid using config options, otherwise that Ceph logic may inadvertently pick an incorrect address.

https://github.com/openstack/charm-ceph-osd/blob/stable/18.08/hooks/utils.py#L106-L129
        return network_get_primary_address('public')

        return network_get_primary_address('cluster')

https://docs.jujucharms.com/2.4/en/developer-network-primitives#network-get

https://git.launchpad.net/ubuntu/+source/ceph/tree/src/common/options.cc?h=ubuntu/bionic-updates&id=import/12.2.4-0ubuntu1.1#n167
https://git.launchpad.net/ubuntu/+source/ceph/tree/src/common/pick_address.cc?h=ubuntu/bionic-updates&id=import/12.2.4-0ubuntu1.1#n157

I hope that helps.

[Bruno Carvalho (brunowcs)]

Hi Dmitrii, tks for your answer.

I understand about spaces and I´m using with MaaS, but my did problem in this configuration at multi network

ceph-public-network: 172.16.1.0/24 172.16.2.0/24
ceph-cluster-network: 192.168.1.0/24 192.168.2.0/24

This configuration not working in my envrioment with charm default, it work after change some line in ceph_hooks.py.

I want know if this a bug or multi network only work with spaces?

[Dmitrii Shcherbakov (dmitriis)]

Bruno,

Based on the code it should split the config just fine, unless you use commas to separate networks via config.

.split() without arguments properly handles white space but not commas:

In [1]: networks_opt = '172.16.1.0/24 172.16.2.0/24'

In [2]: networks = networks_opt.split()

In [3]: networks
Out[3]: ['172.16.1.0/24', '172.16.2.0/24']

In [4]: networks_opt = '172.16.1.0/24,172.16.2.0/24'

In [5]: networks = networks_opt.split()

In [6]: networks
Out[6]: ['172.16.1.0/24,172.16.2.0/24']

[Bruno Carvalho (brunowcs)]

Dmitrii I agree with you, but it's not working this way, can you add two networks to your charm and take that test? because it would simulate the same scenario as my.

make sure your ceph.conf is changed

[Bruno Carvalho (brunowcs)]

Hi Dmitrii,

I think the problem is in the second split of the function get_address_in_network

2018-09-27 21:40:44 DEBUG juju-log ['10.254.21.0/24', '10.254.25.0/24'] -> network.split()
2018-09-27 21:46:12 DEBUG juju-log ['10.254.22.0/24 10.254.26.0/24'] -> network.split(',')

charm/hooks/charmhelpers/contrib/network/ip.py

def get_address_in_network(network, fallback=None, fatal=False):
    """Get an IPv4 or IPv6 address within the network from the host.

    :param network (str): CIDR presentation format. For example,
        '192.168.1.0/24'. Supports multiple networks as a space-delimited list.
    :param fallback (str): If no address is found, return fallback.
    :param fatal (boolean): If no address is found, fallback is not
        set and fatal is True then exit(1).
    """
    if network is None:
        if fallback is not None:
            return fallback

        if fatal:
            no_ip_found_error_out(network)
        else:
            return None

networks = network.split() or [network]
    for network in networks:
        _validate_cidr(network)
        network = netaddr.IPNetwork(network)
        for iface in netifaces.interfaces():
            try:
                addresses = netifaces.ifaddresses(iface)
            except ValueError:
                # If an instance was deleted between
                # netifaces.interfaces() run and now, its interfaces are gone
                continue
            if network.version == 4 and netifaces.AF_INET in addresses:
                for addr in addresses[netifaces.AF_INET]:
                    cidr = netaddr.IPNetwork("%s/%s" % (addr['addr'],
                                                        addr['netmask']))
                    if cidr in network:
                        return str(cidr.ip)

            if network.version == 6 and netifaces.AF_INET6 in addresses:
                for addr in addresses[netifaces.AF_INET6]:
                    cidr = _get_ipv6_network_from_address(addr)
                    if cidr and cidr in network:
                        return str(cidr.ip)

    if fallback is not None:
        return fallback

    if fatal:
        no_ip_found_error_out(network)

    return None

[Bruno Carvalho (brunowcs)]

I´m add log debug in code, we can see when it calls the function get_address_in_network, this only list adress one subnetwork (cidr ip IPAddress) cidr ip IPAddress('10.254.21.7') don´t list adress second network ['10.254.25.0/24'] and ['10.254.26.0/24']

2018-09-27 22:28:29 INFO juju-log network func ip ['10.254.21.0/24']
2018-09-27 22:28:30 DEBUG worker.uniter.jujuc server.go:178 running hook tool "juju-log"
2018-09-27 22:28:30 INFO juju-log cidr ip IPAddress('10.254.21.7')
2018-09-27 22:28:30 DEBUG worker.uniter.jujuc server.go:178 running hook tool "juju-log"
2018-09-27 22:28:30 INFO juju-log network func ip ['10.254.25.0/24']
2018-09-27 22:28:30 DEBUG worker.uniter.jujuc server.go:178 running hook tool "juju-log"
2018-09-27 22:28:30 INFO juju-log network func ip ['10.254.26.0/24']
2018-09-27 22:28:30 DEBUG worker.uniter.jujuc server.go:178 running hook tool "juju-log"
2018-09-27 22:28:30 INFO juju-log network func ip ['10.254.22.0/24']
2018-09-27 22:28:30 DEBUG worker.uniter.jujuc server.go:178 running hook tool "juju-log"
2018-09-27 22:28:30 INFO juju-log cidr ip IPAddress('10.254.22.71')
2018-09-27 22:28:30 DEBUG worker.uniter.jujuc server.go:178 running hook tool "relation-get"
2018-09-27 22:28:30 DEBUG worker.uniter.jujuc server.go:178 running hook tool "relation-get"
2018-09-27 22:28:33 DEBUG juju.worker.uniter.remotestate watcher.go:346 got config change: ok=true
2018-09-27 22:28:35 DEBUG worker.uniter.jujuc server.go:178 running hook tool "juju-log"
2018-09-27 22:28:35 INFO juju-log network func ip ['10.254.21.0/24']
2018-09-27 22:28:35 DEBUG worker.uniter.jujuc server.go:178 running hook tool "juju-log"
2018-09-27 22:28:35 INFO juju-log cidr ip IPAddress('10.254.21.7')
2018-09-27 22:28:35 DEBUG worker.uniter.jujuc server.go:178 running hook tool "juju-log"
2018-09-27 22:28:35 INFO juju-log network func ip ['10.254.25.0/24']
2018-09-27 22:28:35 DEBUG worker.uniter.jujuc server.go:178 running hook tool "juju-log"
2018-09-27 22:28:35 INFO juju-log network func ip ['10.254.26.0/24']
2018-09-27 22:28:35 DEBUG worker.uniter.jujuc server.go:178 running hook tool "juju-log"
2018-09-27 22:28:35 INFO juju-log network func ip ['10.254.22.0/24']
2018-09-27 22:28:35 DEBUG worker.uniter.jujuc server.go:178 running hook tool "juju-log"
2018-09-27 22:28:35 INFO juju-log cidr ip IPAddress('10.254.22.71')

[Bruno Carvalho (brunowcs)]

Hi Dmitrii,

My setup worked only with configured subnetwork and not both

ex: ceph.conf monitor public network = 10.254.21.0/24

Just a network, would that be the normal behavior?

Does it only take the network that is configured on the server interface?

[Dmitrii Shcherbakov (dmitriis)]

Hi Bruno,

You should get only one subnet in the config based on the interfaces a given machine has:

https://github.com/openstack/charm-ceph-osd/blob/stable/18.08/hooks/utils.py#L132-L143

    networks = config(config_opt)
    if networks:
        networks = networks.split()
        return [n for n in networks if get_address_in_network(n)]

# ^^^ filters out networks for which there is no corresponding IP configured on a given machine

It then joins filtered networks in get_ceph_context but by this time you have only one left:
https://github.com/openstack/charm-ceph-osd/blob/stable/18.08/hooks/ceph_hooks.py#L316-L331

This should be fine with ceph because it uses that information on a particular machine to create listening sockets.

[Bruno Carvalho (brunowcs)]

Hello, I'm getting the following msg in the log in my L3 environment, this usually occurs because in ceph.conf declare the two subnet.

Ex:

public network = 10.254.25.0/24 10.254.21.0/24
cluster network = 10.254.26.0/24 10.254.22.0/24

The charm can not configure the two subnet in ceph.conf, I think this should be fixed. I believe ceph does not work well this way

Nov 14 19:59:52 uat-l-stor-15 ceph-osd: 2018-11-14 19:59:52.273627 7f9db37a6700 -1 osd.28 148505 heartbeat_check: no reply from 10.254.21.4:6809 osd.2 ever on either front or back, first ping sent 2018-11-14 19:33:39.245718 (cutoff 2018-11-14 19:59:32.273624)
Nov 14 19:59:52 uat-l-stor-15 ceph-osd[24110]: 2018-11-14 19:59:52.273627 7f9db37a6700 -1 osd.28 148505 heartbeat_check: no reply from 10.254.21.4:6809 osd.2 ever on either front or back, first ping sent 2018-11-14 19:33:39.245718 (cutoff 2018-11-14 19:59:32.273624)
Nov 14 19:59:53 uat-l-stor-15 ceph-osd: 2018-11-14 19:59:53.273742 7f9db37a6700 -1 osd.28 148507 heartbeat_check: no reply from 10.254.21.4:6809 osd.2 ever on either front or back, first ping sent 2018-11-14 19:33:39.245718 (cutoff 2018-11-14 19:59:33.273740)
Nov 14 19:59:53 uat-l-stor-15 ceph-osd[24110]: 2018-11-14 19:59:53.273742 7f9db37a6700 -1 osd.28 148507 heartbeat_check: no reply from 10.254.21.4:6809 osd.2 ever on either front or back, first ping sent 2018-11-14 19:33:39.245718 (cutoff 2018-11-14 19:59:33.273740)
Nov 14 19:59:54 uat-l-stor-15 ceph-osd: 2018-11-14 19:59:54.273852 7f9db37a6700 -1 osd.28 148507 heartbeat_check: no reply from 10.254.21.4:6809 osd.2 ever on either front or back, first ping sent 2018-11-14 19:33:39.245718 (cutoff 2018-11-14 19:59:34.273850)
Nov 14 19:59:54 uat-l-stor-15 ceph-osd[24110]: 2018-11-14 19:59:54.273852 7f9db37a6700 -1 osd.28 148507 heartbeat_check: no reply from 10.254.21.4:6809 osd.2 ever on either front or back, first ping sent 2018-11-14 19:33:39.245718 (cutoff 2018-11-14 19:59:34.273850)
Nov 14 19:59:55 uat-l-stor-15 ceph-osd: 2018-11-14 19:59:55.273969 7f9db37a6700 -1 osd.28 148507 heartbeat_check: no reply from 10.254.21.4:6809 osd.2 ever on either front or back, first ping sent 2018-11-14 19:33:39.245718 (cutoff 2018-11-14 19:59:35.273967)
Nov 14 19:59:55 uat-l-stor-15 ceph-osd[24110]: 2018-11-14 19:59:55.273969 7f9db37a6700 -1 osd.28 148507 heartbeat_check: no reply from 10.254.21.4:6809 osd.2 ever on either front or back, first ping sent 2018-11-14 19:33:39.245718 (cutoff 2018-11-14 19:59:35.273967)
Nov 14 19:59:56 uat-l-stor-15 ceph-osd: 2018-11-14 19:59:56.274080 7f9db37a6700 -1 osd.28 148507 heartbeat_check: no reply from 10.254.21.4:6809 osd.2 ever on either front or back, first ping sent 2018-11-14 19:33:39.245718 (cutoff 2018-11-14 19:59:36.274078)
Nov 14 19:59:56 uat-l-stor-15 ceph-osd[24110]: 2018-11-14 19:59:56.274080 7f9db37a6700 -1 osd.28 148507 heartbeat_check: no reply from 10.254.21.4:6809 osd.2 ever on either front or back, first ping sent 2018-11-14 19:33:39.245718 (cutoff 2018-11-14 19:59:36.274078)

Nov 14 20:13:52 uat-l-stor-15 ceph-osd: 2018-11-14 20:13:52.784963 7f6340bfd700 0 -- 10.254.26.5:6808/53263 >> 10.254.26.4:6800/389525450 conn(0x5598af3ca800 :6808 s=STATE_ACCEPTING_WAIT_CONNEC...