etcd3 driver failing with "Deadline Exceeded"

[Ben Nemec]

This may be related to, but is separate from https://bugs.launchpad.net/python-tooz/+bug/1808036

It is currently blocking the unit test jobs in tooz with etcd3. Unlike the "not capable" error though, I don't yet understand why this is happening. Even simple calls like client.status() are timing out.

[Ben Nemec (bnemec)]

It looks like this is a problem with grpcio-1.16.0+. If I add a version cap to keep it below 1.16.0 then everything works as expected.

Also, since I forgot to include the traceback in the original report, here it is:

2018-12-11 00:21:10.259089 | ubuntu-xenial | Captured traceback:
2018-12-11 00:21:10.259155 | ubuntu-xenial | ~~~~~~~~~~~~~~~~~~~
2018-12-11 00:21:10.259260 | ubuntu-xenial | b'Traceback (most recent call last):'
2018-12-11 00:21:10.259565 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/tooz/.tox/py35-etcd3/lib/python3.5/site-packages/etcd3/client.py", line 46, in handler'
2018-12-11 00:21:10.259666 | ubuntu-xenial | b' return f(*args, **kwargs)'
2018-12-11 00:21:10.259943 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/tooz/.tox/py35-etcd3/lib/python3.5/site-packages/etcd3/client.py", line 679, in lease'
2018-12-11 00:21:10.260044 | ubuntu-xenial | b' metadata=self.metadata'
2018-12-11 00:21:10.260346 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/tooz/.tox/py35-etcd3/lib/python3.5/site-packages/grpc/_channel.py", line 547, in __call__'
2018-12-11 00:21:10.260518 | ubuntu-xenial | b' return _end_unary_response_blocking(state, call, False, None)'
2018-12-11 00:21:10.260855 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/tooz/.tox/py35-etcd3/lib/python3.5/site-packages/grpc/_channel.py", line 466, in _end_unary_response_blocking'
2018-12-11 00:21:10.260984 | ubuntu-xenial | b' raise _Rendezvous(state, None, None, deadline)'
2018-12-11 00:21:10.261137 | ubuntu-xenial | b'grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:'
2018-12-11 00:21:10.261248 | ubuntu-xenial | b'\tstatus = StatusCode.DEADLINE_EXCEEDED'
2018-12-11 00:21:10.261346 | ubuntu-xenial | b'\tdetails = "Deadline Exceeded"'
2018-12-11 00:21:10.262327 | ubuntu-xenial | b'\tdebug_error_string = "{"created":"@1544487511.159561332","description":"Failed to create subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2706,"referenced_errors":[{"created":"@1544487511.159555724","description":"Pick Cancelled","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":221,"referenced_errors":[{"created":"@1544487511.159549203","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4}]}]}"'
2018-12-11 00:21:10.262378 | ubuntu-xenial | b'>'
2018-12-11 00:21:10.262420 | ubuntu-xenial | b''
2018-12-11 00:21:10.262571 | ubuntu-xenial | b'During handling of the above exception, another exception occurred:'
2018-12-11 00:21:10.262617 | ubuntu-xenial | b''
2018-12-11 00:21:10.262719 | ubuntu-xenial | b'Traceback (most recent call last):'
2018-12-11 00:21:10.262960 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/tooz/tooz/tests/__init__.py", line 36, in skip_if_not_implemented'
2018-12-11 00:21:10.263058 | ubuntu-xenial | b' return func(*args, **kwargs)'
2018-12-11 00:21:10.263296 | ubuntu-xenial | b' File "/home/zuul/src/g...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/624508

[Ben Nemec (bnemec)]

I should also note that the problem seems to be cumulative. If I run individual tests then they will pass, but those same tests when run as part of a full test run hit the timeout error.

[Ben Nemec (bnemec)]

Issue submitted to grpc: https://github.com/grpc/grpc/issues/17478

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/624508
Committed: https://git.openstack.org/cgit/openstack/tooz/commit/?id=70f144abde14e07d7f9620a2abb563ed16ef8c63
Submitter: Zuul
Branch: master

commit 70f144abde14e07d7f9620a2abb563ed16ef8c63
Author: Ben Nemec <email address hidden>
Date: Tue Dec 11 21:59:24 2018 +0000

    Unblock tooz gate

    We have a couple of issues blocking tooz test jobs right now. This
    is a combination of two commits that fix the problem.

    Cap grpcio<1.16.0

    1.16.0 introduces a bug that is causing our etcd3 tests to fail on
    a consistent basis. Until that is resolved we can't use any higher
    versions of grpcio.

    1.0.0 is used as the lower bound on the assumption that any version
    of the current major release will work. Since we weren't setting a
    bound before that seems relatively safe.

    Partial-Bug: 1808046

    Fix requirements

    The requirements check job is failing on a couple of the entries in
    this project. enum34 needs a lower bound, and futurist has an
    exclusion that isn't in global-requirements.

    enum34 is given the same lower bound as oslo.upgradecheck, and
    futurist's lower bound is increased to the last global lower bound
    (before lower bounds were removed), which eliminates the need for
    the exclusion.

    Change-Id: I1b279e138325b596f5ec2c61717029d2f7bdd075
    Depends-On: https://review.openstack.org/#/c/624759/

[Ben Nemec (bnemec)]

Since we capped grpcio this is no longer a critical blocker.

[Ben Nemec (bnemec)]

Fixed in grpcio 1.18.0. Will submit a requirements patch to exclude the broken versions.