l2gateway-agent dies on py36 with "TypeError: a bytes-like object is required, not 'str'"

[Lajos Katona]

l2gateway-agent throws exception during connection towards ovsdb under py36:

Mar 15 04:56:49 l2gwcont neutron-l2gateway-agent[4273]: ERROR networking_l2gw.services.l2gateway.agent.ovsdb.base_connection [None req-92ed38a7-47ee-41e4-a7d1-ccda98db06c5 demo admin] Exception [a bytes-like object is required, not 'str'] occurred while sending message to the OVSDB server: TypeError: a bytes-like object is required, not 'str'
Mar 15 04:56:49 l2gwcont neutron-l2gateway-agent[4273]: ERROR networking_l2gw.services.l2gateway.agent.ovsdb.base_connection Traceback (most recent call last):
Mar 15 04:56:49 l2gwcont neutron-l2gateway-agent[4273]: ERROR networking_l2gw.services.l2gateway.agent.ovsdb.base_connection File "/opt/stack/networking-l2gw/networking_l2gw/services/l2gateway/agent/ovsdb/base_connection.py", line 274, in send
Mar 15 04:56:49 l2gwcont neutron-l2gateway-agent[4273]: ERROR networking_l2gw.services.l2gateway.agent.ovsdb.base_connection bytes_sent = self.socket.send(jsonutils.dumps(message))
Mar 15 04:56:49 l2gwcont neutron-l2gateway-agent[4273]: ERROR networking_l2gw.services.l2gateway.agent.ovsdb.base_connection File "/usr/local/lib/python3.6/dist-packages/eventlet/greenio/base.py", line 395, in send
Mar 15 04:56:49 l2gwcont neutron-l2gateway-agent[4273]: ERROR networking_l2gw.services.l2gateway.agent.ovsdb.base_connection return self._send_loop(self.fd.send, data, flags)
Mar 15 04:56:49 l2gwcont neutron-l2gateway-agent[4273]: ERROR networking_l2gw.services.l2gateway.agent.ovsdb.base_connection File "/usr/local/lib/python3.6/dist-packages/eventlet/greenio/base.py", line 382, in _send_loop
Mar 15 04:56:49 l2gwcont neutron-l2gateway-agent[4273]: ERROR networking_l2gw.services.l2gateway.agent.ovsdb.base_connection return send_method(data, *args)
Mar 15 04:56:49 l2gwcont neutron-l2gateway-agent[4273]: ERROR networking_l2gw.services.l2gateway.agent.ovsdb.base_connection TypeError: a bytes-like object is required, not 'str'

/usr/bin/python3.6 /usr/local/bin/neutron-l3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/l3_agent.ini

Environment:
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

neutron: 03d0e43c3206384bd70a91a19d791d695c84a90e
devstack: e86a6aa9243ce908c71b2131686dbdeab61e57d9
networking-l2gw: f9308feb5585d0a4c30b110b6437234affe5e516

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/666019

[Kees Meijs (keesmeijs)]

Just downloaded the patch from Gerrit. Will try if it solves the problem in my case and will report back.

[Kees Meijs (keesmeijs)]

The patch seems to have corrected the TypeError:

2019-08-10 15:56:53.352 976 DEBUG networking_l2gw.services.l2gateway.agent.ovsdb.base_connection [-] Connected to OVSDB server 10.128.29.66 __init__ /usr/lib/python3/dist-packages/networking_l2gw/services/l2gateway/agent/ovsdb/base_connection.py:91
2019-08-10 15:56:53.352 976 DEBUG networking_l2gw.services.l2gateway.agent.ovsdb.manager [-] Calling notify_ovsdb_states _connect_to_ovsdb_server /usr/lib/python3/dist-packages/networking_l2gw/services/l2gateway/agent/ovsdb/manager.py:161
2019-08-10 15:57:13.353 976 DEBUG networking_l2gw.services.l2gateway.agent.ovsdb.manager [req-1c97a14c-bb25-45fc-a8d7-9e0fca07127c - - - - -] Calling notify_ovsdb_states _connect_to_ovsdb_server /usr/lib/python3/dist-packages/networking_l2gw/services/l2gateway/agent/ovsdb/manager.py:161
2019-08-10 15:57:33.355 976 DEBUG networking_l2gw.services.l2gateway.agent.ovsdb.manager [req-1c97a14c-bb25-45fc-a8d7-9e0fca07127c - - - - -] Calling notify_ovsdb_states _connect_to_ovsdb_server /usr/lib/python3/dist-packages/networking_l2gw/services/l2gateway/agent/ovsdb/manager.py:161
2019-08-10 15:57:53.360 976 DEBUG networking_l2gw.services.l2gateway.agent.ovsdb.manager [req-1c97a14c-bb25-45fc-a8d7-9e0fca07127c - - - - -] Calling notify_ovsdb_states _connect_to_ovsdb_server /usr/lib/python3/dist-packages/networking_l2gw/services/l2gateway/agent/ovsdb/manager.py:161
2019-08-10 15:58:13.365 976 DEBUG networking_l2gw.services.l2gateway.agent.ovsdb.manager [req-1c97a14c-bb25-45fc-a8d7-9e0fca07127c - - - - -] Calling notify_ovsdb_states _connect_to_ovsdb_server /usr/lib/python3/dist-packages/networking_l2gw/services/l2gateway/agent/ovsdb/manager.py:161

[Kees Meijs (keesmeijs)]

The RPC part still doesn't work in Neutron itself:

2019-08-11 11:47:36.278 10925 WARNING stevedore.named [req-704021c0-d112-4101-9688-f9d7525213c8 - - - - -] Could not load networking_l2gw.services.l2gateway.service_drivers.rpc_l2gw.L2gwRpcDriver

Tried this with Rocky on Ubuntu 18.04, Debian 10 (both with Python3) and on CentOS with Python2.

[Lajos Katona (lajos-katona)]

Thanks Kees for checking out, just arrived back from vacation, will check your comments.

[Lajos Katona (lajos-katona)]

Hi, I checked with latest master (neutron is master and https://review.opendev.org/666019 for networking-l2gw).
I see the warning you referenced for stevedore not able to load L2gwRpcDriver, but a few lines later the driver is loaded:

Aug 26 13:08:04 layer2cont neutron-server[12579]: DEBUG neutron.services.service_base [None req-c39646ae-32ff-46a8-a587-9fafb2d3aaa9 None None] Loaded 'networking_l2gw.services.l2gateway.service_drivers.rpc_l2gw.L2gwRpcDriver' provider for service L2GW {{(pid=12579) load_drivers /opt/stack/neutron/neutron/services/service_base.py:52}}

I am not sure why stevedore tries and fails to load the driver, but I think it is unrelated to the original python3 problem of ovsdb connection.

[Kees Meijs (keesmeijs)]

Ah, didn't spot that.

Although the driver does load after all: there's no two-way RPC communication between Neutron and the L2 agent. As as result messages time out and the agent gets marked as down.

Very soon after restarting the agent it's not down yet so I tried to issue commands to create a L2 gateway in the time window.

Some entries do end up in the Neutron SQL tables but the agent itself doesn't do any work. Besides database clutter, no sugar I'm afraid.

How can we proceed further? How can I be of additional help?

[Lajos Katona (lajos-katona)]

Could you check please if you have this patch in your environment:
https://review.opendev.org/665995
By your description I have seen similar issues with the agent, and this fixed for me.

[Kees Meijs (keesmeijs)]

Thank you very much. I'll check as soon as possible and will report back.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/666019
Committed: https://git.openstack.org/cgit/openstack/networking-l2gw/commit/?id=852c8098ca50a2879776cb5437db1271c9a9d8e5
Submitter: Zuul
Branch: master

commit 852c8098ca50a2879776cb5437db1271c9a9d8e5
Author: elajkat <email address hidden>
Date: Tue Jun 18 15:32:43 2019 +0200

    py3: encode messages on py3

    On python3 send fails with TypeError: a bytes-like object is required,
    not 'str', this patch solves this issue.

    Closes-Bug: #1820154
    Change-Id: I700462a9a5bcc66b2edf05e4e0396556e381a8c1

[Kees Meijs (keesmeijs)]

Good morning,

I'm sorry I didn't respond earlier. Other matters overtook priority I'm
afraid.

The patch is not included in the code we're using. Do you recommend just
applying the patch as well?

Maybe an interesting fact: we're justing stock Debian Buster packages.

Regards,
Kees

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/680366

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/680366
Committed: https://git.openstack.org/cgit/openstack/networking-l2gw/commit/?id=30326d75aa68657c433d6eb9f9696cff7ca5d0c1
Submitter: Zuul
Branch: stable/stein

commit 30326d75aa68657c433d6eb9f9696cff7ca5d0c1
Author: elajkat <email address hidden>
Date: Tue Jun 18 15:32:43 2019 +0200

    py3: encode messages on py3

    On python3 send fails with TypeError: a bytes-like object is required,
    not 'str', this patch solves this issue.

    Closes-Bug: #1820154
    Change-Id: I700462a9a5bcc66b2edf05e4e0396556e381a8c1
    (cherry picked from commit 852c8098ca50a2879776cb5437db1271c9a9d8e5)