oslo.service binding same port for multiple processes

Bug #1810280 reported by venkata anil on 2019-01-02
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.service
Undecided
venkata anil

Bug Description

In oslo.service, backdoor_port [1] allows processes spawned by oslo.service for a service to listen on unique port [2] if we provide a port range. Code block [2] i.e

def _listen(host, start_port, end_port, listen_func):
    try_port = start_port
    while True:
        try:
            return listen_func((host, try_port))
        except socket.error as exc:
            if (exc.errno != errno.EADDRINUSE or
               try_port >= end_port):
                raise
            try_port += 1

will always try with first port in the provided range and will try the next only if the listen fails. But because of commit [3] in eventlet (which went in version v0.20.0), which supports listening on same port, our code will always listen on same port for all the processes. Openstack cinder team has already noticed this issue long back [4] for their CI failures and proposed this change [5], i.e replace eventlet.listen() with directly calling socket.listen [6]

We should also do the same change in oslo.service i.e [6].

    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.bind(("127.0.0.1", 0))
    sock.listen(50)

[1] https://github.com/openstack/oslo.service/blob/master/oslo_service/_options.py#L26
[2] https://github.com/openstack/oslo.service/blob/master/oslo_service/eventlet_backdoor.py#L124-L133
[3] https://github.com/eventlet/eventlet/commit/f9a3074a3b75f17f76cc04a693dc48a367b99861
[4] https://github.com/eventlet/eventlet/issues/411
[5] https://review.openstack.org/#/c/463876/2
[6] https://review.openstack.org/#/c/463876/2/test/__init__.py

Changed in oslo.service:
assignee: nobody → venkata anil (anil-venkata)

Fix proposed to branch: master
Review: https://review.openstack.org/627952

Changed in oslo.service:
status: New → In Progress

Reviewed: https://review.openstack.org/627952
Committed: https://git.openstack.org/cgit/openstack/oslo.service/commit/?id=811650783d32accd2b49f907ddc520955146962c
Submitter: Zuul
Branch: master

commit 811650783d32accd2b49f907ddc520955146962c
Author: venkata anil <email address hidden>
Date: Wed Jan 2 05:11:12 2019 -0500

    Avoid eventlet_backdoor listing on same port

    Oslo.service is binding to same port when we provide a port range
    as eventlet is internally setting SO_REUSEPORT flag (starting from
    eventlet version v0.20). And there is a flag (reuse_port)
    introduced in v0.22 to give control to user to avoid SO_REUSEPORT.
    In this patch, first we try passing reuse_port=False, if this fails
    then directly open socket and listen instead of eventlist.listen.

    Closes-Bug: #1810280
    Change-Id: Idc842acc7e430199c76fe12785b0bf0e7a58e121

Changed in oslo.service:
status: In Progress → Fix Released

This issue was fixed in the openstack/oslo.service 1.36.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers