amphora-agent package, its init script and configuration file

[Frode Nordahl]

1)
At present the ``amphora-agent`` package has a init script that points the amphora-agent at ``/etc/octavia/octavia.conf``. However, the upstream Octavia project will tell cloud-init to write the agent configuration to ``/etc/octavia/amphora-agent.conf``.

Upstream reference:
https://opendev.org/openstack/octavia/src/branch/master/octavia/controller/worker/tasks/compute_tasks.py#L89

Config-drive contents:
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:~# mount /dev/sr0 /mnt
mount: /mnt: WARNING: device write-protected, mounted read-only.
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:~# cd /mnt
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt# ls -l
total 4
dr-xr-xr-x 4 root root 2048 Jun 4 13:03 ec2
dr-xr-xr-x 12 root root 2048 Jun 4 13:03 openstack
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt# cd openstack/
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack# ls -l
total 20
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2012-08-10
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2013-04-04
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2013-10-17
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2015-10-15
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2016-06-30
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2016-10-06
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2017-02-22
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2018-08-27
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 content
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 latest
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack# cd latest
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack/latest# ls -l
total 4
-r--r--r-- 1 root root 2136 Jun 4 13:03 meta_data.json
-r--r--r-- 1 root root 506 Jun 4 13:03 network_data.json
-r--r--r-- 1 root root 2 Jun 4 13:03 vendor_data.json
-r--r--r-- 1 root root 14 Jun 4 13:03 vendor_data2.json
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack/latest# cat meta_data.json
{"uuid": "0aedb1b1-46a1-4734-aa1c-f1aed62f427f", "files": [{"path": "/etc/octavia/amphora-agent.conf", "content_path": "/content/0000"}, {"path": "/etc/octavia/certs/client_ca.pem", "content_path": "/content/0001"}, {"path": "/etc/octavia/certs/server.pem", "content_path": "/content/0002"}], "admin_pass": "boHNY7N32jAb", "public_keys": {"test": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCi1Mwv77hDyu/68syJflcPO6oKGkUBzjBIcPwC1hNdWvVHyaBI2lm6millJIBVQdLYHh/MJe6FRz6iOOZ7eBnnzd63D8GOTS6UnpX+F7BmT8ICH1CBqbnVAn+LJxCfzs8+Q706jTXqkiU3Rgl7A6v8+Tqqu6VsI+4/lODj43LXhlN4+tt3jvbNCXXo0d4jRBEysEnINjsmxcNce3sCeAaDY1trC8DsGAFUJxoPi8We2qYPVvHbWj7yLgDzJULf05TEIxR1IvbaU/2K+eT2ITyPdTgsSCP7E2DAK8WtKiOWJRBiOwE46yT6Os/rAjoIrzv9IqPDnp97uxf1Ia1jD4Ah ubuntu@test\n"}, "keys": [{"name": "test", "type": "ssh", "data": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCi1Mwv77hDyu/68syJflcPO6oKGkUBzjBIcPwC1hNdWvVHyaBI2lm6millJIBVQdLYHh/MJe6FRz6iOOZ7eBnnzd63D8GOTS6UnpX+F7BmT8ICH1CBqbnVAn+LJxCfzs8+Q706jTXqkiU3Rgl7A6v8+Tqqu6VsI+4/lODj43LXhlN4+tt3jvbNCXXo0d4jRBEysEnINjsmxcNce3sCeAaDY1trC8DsGAFUJxoPi8We2qYPVvHbWj7yLgDzJULf05TEIxR1IvbaU/2K+eT2ITyPdTgsSCP7E2DAK8WtKiOWJRBiOwE46yT6Os/rAjoIrzv9IqPDnp97uxf1Ia1jD4Ah ubuntu@test\n"}], "hostname": "amphora-82c4877a-d932-49af-b590-7859c6edb84c.novalocal", "name": "amphora-82c4877a-d932-49af-b590-7859c6edb84c", "launch_index": 0, "availability_zone": "nova", "random_seed": "TbGbfew/Ie7ZwKL2CIOYTiJ5mMAEddIsbzAK3Tg8aa6pitSZiY67oLXgadY7TUhNGon5ZsW7hnUCmAweHbNMXsYOs5jkMljURsuh55ihJb7+Uz57kNTQaMaLi6mLy0PmHXdhpzWs3PJRxddQGtzLa4k1j7dX4O4otViD4kHwVp8K/ZfDORVd8Ot+xOP9oYkvTCf76kjGl/z8Jnwhm83YlOGvjcYxKVEKAy87QAb7k0qxzcai8as1zwck3nyHe6Frl/lHelywoTadSelXkWUCqxfbN/594bTjQobMZPp2O4iixGjeQMzU0utOGJM+AYG9VjZvI9uZ9T3WW+eERn9vg+TOk0mF7bCTNgtzdd2K5zBthTf+g1+3rBNWp4/iSqbHrA+PHK2CEKwiM/5SIarW4/raoBrsDMep47P2ax4XMcZKuTXKZjcnht1hOblDRKtW65fgELkv5/aixCJmD4k5ks9u4FDJgWxdGRZzTNSpWbmnz+qPOk2R+/n3KkIeevQHR3U341Xq4Oyxff1d/9+unRNU6ujC7o7HyDynlnTYcjOHlUcg1KW5Wl7PZ7qredWVmNgwTdotNPJJQe6E9TZjjeVxXBzTxHGoZuvzAKkew5r60Ky2LyUAm+sZiv5Y9pp/P55Hejntg5U9JvVq870m+dqVbLeqdRWYT3G3ysLX/HI=", "project_id": "964a97ca99c6489b817dd8fb8b6ee740", "devices": []}root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack/latest# cd ..
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack# cd content/
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack/content# ls -l
total 6
-r--r--r-- 1 root root 727 Jun 4 13:03 0000
-r--r--r-- 1 root root 1135 Jun 4 13:03 0001
-r--r--r-- 1 root root 2875 Jun 4 13:03 0002
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack/content# cat 0000

[DEFAULT]
debug = True

[haproxy_amphora]
base_cert_dir = /var/lib/octavia/certs
base_path = /var/lib/octavia
bind_host = ::
bind_port = 9443
haproxy_cmd = /usr/sbin/haproxy
respawn_count = 2
respawn_interval = 2
use_upstart = True

[health_manager]
controller_ip_port_list = fc00:4f4a:d87f:9a6c:f816:3eff:fe58:3dd0:5555, fc00:4f4a:d87f:9a6c:f816:3eff:fee0:a2d9:5555, fc00:4f4a:d87f:9a6c:f816:3eff:fee4:669b:5555
heartbeat_interval = 10
heartbeat_key = e211b7a6-f98a-42be-a4c6-b0d5893fb566

[amphora_agent]
agent_server_ca = /etc/octavia/certs/client_ca.pem
agent_server_cert = /etc/octavia/certs/server.pem
agent_request_read_timeout = 120
amphora_id = 82c4877a-d932-49af-b590-7859c6edb84c
amphora_udp_driver = keepalived_lvsroot@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack/content# cd /etc/octavia/
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/etc/octavia# ls -l
total 84
-rw-rw---- 1 root root 727 Jun 4 13:03 amphora-agent.conf
drwxr-xr-x 2 root root 4096 Jun 4 13:03 certs
-rw-r----- 1 root octavia 68324 Jun 4 12:05 octavia.conf
-rw-r----- 1 root octavia 5290 Jun 4 12:05 policy.json

Due to how the init script is laid out it is not simple to override this through ``/etc/default/amphora-agent`` or ``/etc/default/openstack`` as those files are read in too late.

Can we either fix the location of the configuration file in the init script or expose the ``CONFIG_FILE`` variable from ``/etc/default/amphora-agent``?

2)
The package creates the ``/etc/octavia`` directory with 0o750 permissions, subsequently when cloud-init writes out the ``/etc/octavia/amphora-agent.conf`` from the config drive the Octavia service provides it will get 0o640 permissions.

Given the file is owned by root:root the amphora-agent service does not have access to read this configuration file.

Could we either make ``/etc/octavia`` mode 0o755 or 2o750 (group sticky bit) to alleviate this?

3)
For some reason the amphora-agent attempts to write to ``/var/log/amphora-agent.log`` although the init script does pass --log-file=/ver/log/octavia/amphora-agent.log.

Still investigating this, some bits:
# systemctl status amphora-agent
● amphora-agent.service - OpenStack Octavia Amphora Agent (amphora-agent)
   Loaded: loaded (/lib/systemd/system/amphora-agent.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-06-04 16:26:53 UTC; 324ms ago
 Main PID: 1591 (/usr/bin/python)
    Tasks: 1 (limit: 614)
   CGroup: /system.slice/amphora-agent.service
           └─1591 /usr/bin/python3.6 /usr/bin/amphora-agent --config-file=/etc/octavia/amphora-agent.conf --log-file=/var/log/octavia/amphora-agent.log

-- The start-up result is RESULT.
Jun 04 16:26:56 amphora-f4d05447-0603-4f51-9d35-9f302dc4665f amphora-agent[1649]: tcmsg: [Errno 2] No such file or directory: '/proc/net/psched'
Jun 04 16:26:56 amphora-f4d05447-0603-4f51-9d35-9f302dc4665f amphora-agent[1649]: the tc subsystem functionality is limited
Jun 04 16:26:56 amphora-f4d05447-0603-4f51-9d35-9f302dc4665f amphora-agent[1649]: Error: Error: '/var/log/amphora-agent.log' isn't writable [PermissionError(13, 'Permission denied')]
Jun 04 16:26:56 amphora-f4d05447-0603-4f51-9d35-9f302dc4665f systemd[1]: amphora-agent.service: Main process exited, code=exited, status=1/FAILURE
Jun 04 16:26:56 amphora-f4d05447-0603-4f51-9d35-9f302dc4665f systemd[1]: amphora-agent.service: Failed with result 'exit-code'.

For some of these we have limited control over how the Octavia service instructs the amphora instance to run things as it is in full control over config-drive contents etc. If not change our defaults, we should at least make it easily configurable so a Octavia Amphora can be built using packages from UCA.

I'll keep investigating this and put forward a patch as soon as all the pieces necessary for a working amphora from packages is in place.

[Frode Nordahl (fnordahl)]

For 3) the upstream project has the ``/var/log/amphora-agent.log`` path hard-coded, ref: https://opendev.org/openstack/octavia/src/branch/master/octavia/cmd/agent.py#L80

[Frode Nordahl (fnordahl)]

The ``amphora-agent`` run in the load-balancer instances expects to run as root. This is what the images built with octavia from source does.

This also explains how the hard coded path to ``/var/log/amphora-agent.log`` works with the built-from-source images.

[uosci-testing-bot (uosci-testing-bot)]

Hi Frode,

1) This should be a simple change in the package for the amphora-agent init to use /etc/octavia/amphora-agent.conf.

2) These permissions have changed since this bug was reported. /etc/octavia now gets:

root@g1:~# ls -al /etc/octavia
total 36
drwxr-x--- 2 root octavia 4 Jul 17 17:51 .
drwxr-xr-x 100 root root 191 Jul 17 17:51 ..
-rw-r----- 1 root octavia 61659 Jul 17 17:51 octavia.conf
-rw-r----- 1 root octavia 7184 Jul 17 17:51 policy.json

So perhaps that is now solved?

3) Use of /var/log/amphora-agent.log feels non-standard but I'm not finding proof to back that argument up. Packages generally put log files in /var/log/<package>/, ie. /var/log/octavia/amphora-agent.log. I did a quick search of debian standards and filesystem hierarchy standards but didn't find anything. I'm wondering if this should get changed upstream or if we could carry a patch in ubuntu to patch the code to use /var/log/octavia/amphora-agent.log (unless a patch upstream makes sense).

Thanks,
Corey

[uosci-testing-bot (uosci-testing-bot)]

Will switch status while waiting for further info.

[Corey Bryant (corey.bryant)]

Sorry was logged in as the wrong user, it's me!

[Corey Bryant (corey.bryant)]

Actually, for 1), I was thinking the upstream source might ship with a sample or have the ability to generate amphora-agent.conf but it doesn't appear to. So I don't know that the package can install in /etc/octavia/amphora-agent.conf. Maybe the current permissions will allow the project source to write what it needs?

[Corey Bryant (corey.bryant)]

But even in that case, it seems odd for the package to ship an init file that uses a conf file that isn't shipped by the package.