Amphora agent fails to start in a Centos based amphora

[Nir Magnezi]

Running the Amphora agent on a Centos (Cooked image with the diskimage-create script) instance, I got the following error:
IOError: [Errno 2] No such file or directory: '/etc/network/interfaces.d/eth1.cfg'
--> See attached log for full Trace

Looks like the amphora agent is not aware of the OS flavor it's running on.
Interface file path and names indicate that the code assumes it is running on Ubuntu.

agent_server_network_dir (octavia.conf --> [amphora_agent]) is /etc/network/interfaces.d/ by default.
This by itself is not a bug since it is configurable (Although it would have been nicer if it was able to auto-adapt itself to the OS flavor).
Yet, if you change it to the proper path (/etc/sysconfig/network-scripts) it will still fail.

Looking at octavia.amphorae.backends.agent.api_server.util:

def get_network_interface_file(interface):
    if CONF.amphora_agent.agent_server_network_file:
        return CONF.amphora_agent.agent_server_network_file
    return os.path.join(CONF.amphora_agent.agent_server_network_dir,
                        interface + '.cfg')

It shows that the name NIC name convention is hardcoded to ethX.cfg, whereas in Centos, Fedora and RHEL it's ifcfg-ethX --> Issue #1.

To workaround this, I changed agent_server_network_file to /etc/sysconfig/network-scripts/ifcfg-eth0.
This raised Issue #2:
DEBUG octavia.amphorae.backends.agent.api_server.plug [-] Failed to if up Command '['ifup', 'eth1']' returned non-zero exit status 1 _bring_if_up /usr/lib/python2.7/site-packages/octavia/amphorae/backends/agent/api_server/plug.py:147

The amphora agent did not create ifcfg-eth1, hence it failed to start the interface.
Moreover, The ifcfg-eth0 was misconfigured by the template /ctavia/amphorae/backends/agent/api_server/templates/plug_vip_ethX.conf.j2

# Generated by Octavia agent
auto eth1 eth1:0
iface eth1 inet dhcp
iface eth1:0 inet static
address 10.0.0.19
broadcast 10.0.0.255
netmask 255.255.255.0

This template fits Ubuntu, not Centos.
The agent should use a different Template when it is running on Centos, Fedora or RHEL.

[Nir Magnezi (nmagnezi)]

I will not get to this bug for at least two week.
Removing myself from 'Assigned' at the moment in case someone else wish to submit a fix.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/346728

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/346728
Committed: https://git.openstack.org/cgit/openstack/octavia/commit/?id=12d2a0f01b2dfb16ed919d41c7b2fa41a32e0259
Submitter: Jenkins
Branch: master

commit 12d2a0f01b2dfb16ed919d41c7b2fa41a32e0259
Author: Nir Magnezi <email address hidden>
Date: Fri Aug 5 09:45:02 2016 +0300

    Amphora agent refactor to classes

    This patch is a prep work needed for Id99948aec64656a0532afc68e146f0610bff1378
    which comes to Fix the amphora-agent support for RH based Linux flavors.

    This is a pure refactor. Functions were gathered under classes (making
    them methods) so state, such as the operating system flavor, can be preserved
    throughout the entire amphora agent process lifecycle.

    Related-Bug: #1548070

    Change-Id: Ic149211dba8ea78e08cb06b6e1f65da00a6571c7

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/415681

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415681
Committed: https://git.openstack.org/cgit/openstack/octavia/commit/?id=e75c8ecc0908477537bc6474b11c46fddd7b8e4b
Submitter: Jenkins
Branch: master

commit e75c8ecc0908477537bc6474b11c46fddd7b8e4b
Author: Nir Magnezi <email address hidden>
Date: Thu Dec 29 13:39:18 2016 +0200

    Adds user_group option under haproxy_amphora

    This fix adds the user_group configuration option under the haproxy_amphora
    section, which is currently available in the neutron-lbaas code base.

    The incentive for the above-mentioned addition is that in CentOS, Fedora and
    RHEL based amphora images, the 'nogroup' user group does not exist by default.
    Therefore users who wish to use those type of amphora images should have the
    option to configure a different user group such as 'haproxy'.

    The default value for user group is 'nogroup' so the default behavior
    remains intact.

    Related-Bug #1548070
    Change-Id: Ifac59889fa8120d974840bae2913587292f474c1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/331841
Committed: https://git.openstack.org/cgit/openstack/octavia/commit/?id=c00488143daf8e9306194ddcda636910bab532b6
Submitter: Jenkins
Branch: master

commit c00488143daf8e9306194ddcda636910bab532b6
Author: Nir Magnezi <email address hidden>
Date: Thu Jan 12 11:32:50 2017 +0200

    Fix the amphora image support for RH Linux flavors

    Not all Linux flavors accept the same type of configuration to manage
    NICs. The amphora-agent must be able to distinguish between different
    Linux flavors and choose the appropriate type of jinja2 NIC
    configuration template for each one, respectively.

    Up until now, The amphora-agent had no notion of the operating system
    it is running on, therefore it used NIC configuration templates that
    only match Debian based Linux flavors (mostly Ubuntu). Making it
    unusable for flavors such as RHEL, Fedora and CentOS.

    This fix enhances how the amphora-agent is handling NIC hot plugs.
    It will use the appropriate jinja2 template by checking the Amphora
    distribution name when needed.

    Co-Authored-By: Michael Johnson <email address hidden>

    Closes-Bug #1548070

    Change-Id: Id99948aec64656a0532afc68e146f0610bff1378

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/429398

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/431576

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/431576
Committed: https://git.openstack.org/cgit/openstack/octavia/commit/?id=b2494dd531cd0b5692301c5b6f5186f0df76d402
Submitter: Jenkins
Branch: master

commit b2494dd531cd0b5692301c5b6f5186f0df76d402
Author: Nir Magnezi <email address hidden>
Date: Thu Feb 9 16:39:43 2017 +0200

    Adds devstack plugin support for all amp base OS

    Set OCTAVIA_AMP_BASE_OS to pass any option accepted by the
    diskimage-create.sh script: ubuntu | fedora | centos | rhel

    Set OCTAVIA_AMP_DISTRIBUTION_RELEASE_ID to pass release name for the
    selected (either by OCTAVIA_AMP_BASE_OS or by default) operating
    system.

    Set OCTAVIA_AMP_IMAGE_SIZE to pass the image size (by GB).

    If unset, the above mentioned options are not used in the script call
    and thus, the script will use its default values.

    Related-Bug #1548070

    Change-Id: If1139594cf23d9de413ce2603aaa5c1e012c7790

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/429398
Committed: https://git.openstack.org/cgit/openstack/octavia/commit/?id=26a55415abe75e74e22272758a04a2d3000d29eb
Submitter: Jenkins
Branch: master

commit 26a55415abe75e74e22272758a04a2d3000d29eb
Author: Nir Magnezi <email address hidden>
Date: Wed Feb 8 16:02:44 2017 +0200

    Auto-detect haproxy user_group

    As a followup to Id99948aec64656a0532afc68e146f0610bff1378, adding auto
    detection to haproxy_amphora.user_group

    haproxy is capable[1] handling a list of configuration files.
    This patch leverages that capability by simply providing haproxy with an
    additional configuration file, which is baked in the amphora image via a
    diskimage-builder element.

    The above-mentioned element will specify the following values for user group:
    Ubuntu: 'nogroup'
    RHEL/CentOS/Fedora: 'haproxy'

    The amphora-agent will parse and remove any user_group configuration provided
    by Octavia controller worker.
    This is in order to maintain amphora-agent backward compatibility to old
    Octavia workers, who still provide user_group to the amphora-agent.
    Octavia Workers that include this patch will no longer provide user_group
    configuration to the amphora-agent.

    [1] https://cbonte.github.io/haproxy-dconv/1.7/management.html#3

    Related-Bug #1548070

    Change-Id: Ia8fede9d7da4709a48661d1fc595a16d04fcbfa9