Amphora agent reload fails with socket in use

[Michael Johnson]

When the amphora agent reloads due to a file change (certificate update in this case) the agent fails to start up with a socket in use error. When looking at "netstat -anp" we see that the agent socket, 9443, is now owned by dhclient.

This means that if a file update occurs the agent will fail and an amphora fail over will be triggered.

Log from the amphora:

2015-09-16 20:21:28.418 417 INFO werkzeug [-] 172.24.4.1 - - [16/Sep/2015 20:21:28] "PUT /0.5/certificate HTTP/1.1" 202 -
2015-09-16 20:21:28.485 417 INFO werkzeug [-] * Detected change in '/etc/octavia/certs/server.pem', reloading
2015-09-16 20:21:28.568 180 INFO werkzeug [-] * Restarting with stat
2015-09-16 20:21:28.987 3704 INFO octavia.common.config [-] Logging enabled!
2015-09-16 20:21:28.988 3704 CRITICAL octavia [-] error: [Errno 98] Address already in use
2015-09-16 20:21:28.988 3704 ERROR octavia Traceback (most recent call last):
2015-09-16 20:21:28.988 3704 ERROR octavia File "/usr/local/bin/amphora-agent", line 10, in <module>
2015-09-16 20:21:28.988 3704 ERROR octavia sys.exit(main())
2015-09-16 20:21:28.988 3704 ERROR octavia File "/usr/local/lib/python2.7/dist-packages/octavia/cmd/agent.py", line 91, in main
2015-09-16 20:21:28.988 3704 ERROR octavia extra_files=[CONF.amphora_agent.agent_server_cert])
2015-09-16 20:21:28.988 3704 ERROR octavia File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 618, in run_simple
2015-09-16 20:21:28.988 3704 ERROR octavia test_socket.bind((hostname, port))
2015-09-16 20:21:28.988 3704 ERROR octavia File "/usr/lib/python2.7/socket.py", line 224, in meth
2015-09-16 20:21:28.988 3704 ERROR octavia return getattr(self._sock,name)(*args)
2015-09-16 20:21:28.988 3704 ERROR octavia error: [Errno 98] Address already in use
2015-09-16 20:21:28.988 3704 ERROR octavia

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/227912

[OpenStack Infra (hudson-openstack)]

Change abandoned by German Eichberger (<email address hidden>) on branch: master
Review: https://review.openstack.org/227912
Reason: there is a patch outstanding which werkzeug which will take care of that

[Brandon Logan (brandon-logan)]

German, any word on that werkzeug patch? Could you link something here?

[German Eichberger (german-eichberger)]

I tried things on a fresh devstack and it successfully reloaded :

stack@Octavia:~/devstack$ neutron lbaas-loadbalancer-create private-subnet Created a new loadbalancer:
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| admin_state_up | True |
| description | |
| id | 0a806ca4-224d-4a77-a83f-604d0789b604 |
| listeners | |
| name | |
| operating_status | OFFLINE |
| provider | octavia |
| provisioning_status | PENDING_CREATE |
| tenant_id | 80f55e09b76e4b91903bc9b466c33970 |
| vip_address | 10.0.0.6 |
| vip_port_id | 9236a715-6720-46d9-8a16-c1a76e51fff1 |
| vip_subnet_id | f6668e1c-7a16-4e84-af86-3e7b697895ec |
+---------------------+--------------------------------------+
stack@Octavia:~/devstack$ nova list
+--------------------------------------+----------------------------------------------+--------+------------+-------------+---------------------------------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------------------------------------------+--------+------------+-------------+---------------------------------------------------------------------------------+
| 2e3c9ad9-cd97-4c83-a2db-45b89f1693c1 | amphora-095403de-fcf6-42dd-8942-368c4cd60f42 | ACTIVE | - | Running | lb-mgmt-net=192.168.0.5; private=fd1d:3c05:fb0d:0:f816:3eff:fe15:9a72, 10.0.0.7 |
+--------------------------------------+----------------------------------------------+--------+------------+-------------+---------------------------------------------------------------------------------+
stack@Octavia:~/devstack$ curl -k -v --cert /etc/octavia/certs/client.pem https://192.168.0.5:9443/0.5/info
* Hostname was NOT found in DNS cache
* Trying 192.168.0.5...
* Connected to 192.168.0.5 (192.168.0.5) port 9443 (#0)
* successfully set certificate verify locations:
* CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS handshake, CERT verify (15):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-GCM-SHA384
* Server certificate:
* subject: CN=095403de-fcf6-42dd-8942-368c4cd60f42
* start date:...

[German Eichberger (german-eichberger)]

installing Werkzeug >=0.11.2 fixes this issues as seen above