contrail-charms: determine vhost0 gateway for multi-interface setup

[Bernhard Koessler]

currently, the contrail-charms for Contrail 4.1 use the default gateway when charm parameter vhost-gateway is set to auto. This is not feasible for a multi-interface setup where all routes should be checked for valid gateways for the specified vrouter interface.

Default GW only is checked for getting gw IP address:
https://github.com/Juniper/contrail-charms/blob/13da977aff7efd67f0e1e7b2f16ecdfc6f445e9c/contrail-agent/hooks/contrail_agent_utils.py#L106
https://github.com/Juniper/contrail-charms/blob/13da977aff7efd67f0e1e7b2f16ecdfc6f445e9c/contrail-agent/hooks/contrail_agent_utils.py#L109

instead of checking the default GW, the function could do somthing like the below to check all installed routes:

def _get_iface_gateway_ip(iface):
    for line in check_output(["route", "-n"]).splitlines()[2:]:
        l = line.split()
        if "G" in l[3] and ( l[7] == VROUTER_INTERFACE or l[7] == iface ):
            log("Found gateway {} for interface {}".format(l[1], l[7]))
            return l[1]
    log("No gateway could be determined from routing table")
    return None

Also, the same routes have to be installed on vhost0 as were present for the original vrouter interface before vhost0 installation. As this depends on the host configuration it might be difficult to consider all possibilities for a persitent configuration but the routes should be installed at least for the provisioning to work.

[Bernhard Koessler (bkoessler)]

to solve vourter provisioning in a multi-interface setup with specific routes I did the following hacks which leads to a working vrouter:

--- ../contrail_agent_utils.py.orig 2018-06-13 12:11:10.645817961 +0000
+++ contrail-charms/contrail-agent/hooks/contrail_agent_utils.py 2018-06-15 12:22:05.564341326 +0000
@@ -102,12 +102,33 @@

 def _get_iface_gateway_ip(iface):
- if hasattr(netifaces, "gateways"):
- data = netifaces.gateways()["default"][netifaces.AF_INET]
- return data[0] if data[1] == iface else None
-
- data = check_output("ip route | grep ^default", shell=True).split()
- return data[2] if data[4] == iface else None
+ for line in check_output(["route", "-n"]).splitlines()[2:]:
+ l = line.split()
+ if "G" in l[3] and ( l[7] == VROUTER_INTERFACE or l[7] == iface ):
+ log("Found gateway {} for interface {}".format(l[1], l[7]))
+ return l[1]
+ log("No gateway could be determined from routing table")
+ return None
+
+
+def _get_routes(iface):
+ routes = []
+ for line in check_output(["route", "-n"]).splitlines()[2:]:
+ route = []
+ l = line.split()
+ if "G" in l[3] and ( l[7] == VROUTER_INTERFACE or l[7] == iface ):
+ log("Found route {} {} via gw {} for interface {}".format(l[0], l[2], l[1], l[7]))
+ route = [ l[0], l[1], l[2] ]
+ routes.append(route)
+ return routes
+
+def _set_routes(routes):
+ for route in routes:
+ args = [ "route", "add", "-net", route[0], "netmask", route[2], "gw", route[1] ]
+ try:
+ check_call(args)
+ except Exception as e:
+ log("Could not add route {} {} via {}, error: {}".format(route[0], route[2], route[1], str(e)))

 def _vhost_cidr(iface):
@@ -139,6 +160,8 @@
         gateway_ip = _get_iface_gateway_ip(iface)
     config["vhost-gateway-ip"] = gateway_ip

+ routes = _get_routes(iface)
+
     if config["dpdk"]:
         fs = os.path.realpath("/sys/class/net/" + iface).split("/")
         # NOTE: why it's not an error?
@@ -154,6 +177,8 @@
         render("agent_param", "/etc/contrail/agent_param",
                {"interface": iface})

+ _set_routes(routes)
+

 def drop_caches():
     """Clears OS pagecache"""
@@ -184,8 +209,7 @@
     ready = (
         config.get("api_port")
         and (config.get("api_ip") or config.get("api_vip"))
- and config.get("analytics_servers")
- and info.get("cloud_orchestrator"))
+ and config.get("analytics_servers"))
     if config.get("vrouter-expected-provision-state"):
         if ready and not config.get("vrouter-provisioned"):
             try:

[Andrey Pavlov (apavlov-e)]

gateway detection https://github.com/Juniper/contrail-charms/commit/3e5f8a76dbbf26b7b9d0ea15cb7afcdf53bf8a35

[Bernhard Koessler (bkoessler)]

regarding routing, the suggestion is to move the route entries in the interfaces file from the vrouter interface to the vhost0 interface in the vrouter installation script.

Currently, the vrouter.cfg file looks like this for a multi-interface, L3 fabric environment after vrouter installation:

root@ic-skbrat2-cz37486c8s:~# cat /etc/network/interfaces.d/vrouter.cfg
###############################################################################
# [ WARNING ]
# Configuration file maintained by Juju. Local changes may be overwritten.
###############################################################################

auto bondB
iface bondB inet manual
    bond-lacp-rate fast
    bond-miimon 100
    bond-mode 802.3ad
    bond-num-grat-arp 1
    bond-slaves none
    bond-xmit-hash-policy layer2
    hwaddress 48:df:37:21:ed:98
    mtu 9134
    post-up route add -net 100.80.39.0 netmask 255.255.255.0 gw 100.80.49.254 metric 0 || true
    pre-down route del -net 100.80.39.0 netmask 255.255.255.0 gw 100.80.49.254 metric 0 || true
    post-up route add -net 100.80.59.0 netmask 255.255.255.0 gw 100.80.49.254 metric 0 || true
    pre-down route del -net 100.80.59.0 netmask 255.255.255.0 gw 100.80.49.254 metric 0 || true
    post-up route add -net 100.80.69.0 netmask 255.255.255.0 gw 100.80.49.254 metric 0 || true
    pre-down route del -net 100.80.69.0 netmask 255.255.255.0 gw 100.80.49.254 metric 0 || true
    post-up route add -net 100.80.38.0 netmask 255.255.255.0 gw 100.80.49.254 metric 0 || true
    pre-down route del -net 100.80.38.0 netmask 255.255.255.0 gw 100.80.49.254 metric 0 || true
    post-up route add -net 100.80.48.0 netmask 255.255.255.0 gw 100.80.49.254 metric 0 || true
    pre-down route del -net 100.80.48.0 netmask 255.255.255.0 gw 100.80.49.254 metric 0 || true
    post-up route add -net 100.80.58.0 netmask 255.255.255.0 gw 100.80.49.254 metric 0 || true
    pre-down route del -net 100.80.58.0 netmask 255.255.255.0 gw 100.80.49.254 metric 0 || true
    post-up route add -net 100.80.68.0 netmask 255.255.255.0 gw 100.80.49.254 metric 0 || true
    pre-down route del -net 100.80.68.0 netmask 255.255.255.0 gw 100.80.49.254 metric 0 || true

auto vhost0
iface vhost0 inet static
    address 100.80.49.1/24
    pre-up ip link add address $(cat /sys/class/net/bondB/address) type vhost
    pre-up vif --add bondB --mac $(cat /sys/class/net/bondB/address) --vrf 0 --vhost-phys --type physical
    pre-up vif --add vhost0 --mac $(cat /sys/class/net/bondB/address) --vrf 0 --type vhost --xconnect bondB
    post-down vif --list | awk '/^vif.*OS: vhost0/ {split($1, arr, "\/"); print arr[2];}' | xargs vif --delete
    post-down vif --list | awk '/^vif.*OS: bondB/ {split($1, arr, "\/"); print arr[2];}' | xargs vif --delete
    post-down ip link delete vhost0

The file should look like this after vrouter installation - making sure the configuration is persitent across reboots etc:

root@ic-skbrat2-cz37486c8s:~# cat /etc/network/interfaces.d/vrouter.cfg
###############################################################################
# [ WARNING ]
# Configuration file maintained by Juju. Local changes may be overwritten.
##################################...