Ubuntu 22.04 python3-grpcio causes some apps to use 100% of the CPU

[Gustavo Iñiguez Goya]

Hi!

After the latest Ubuntu release 22.04, we've realized that the package python3-grpcio 1.30.2-3build6 causes [0] opensnitch to use 100% of the CPU [1].

This problem has never occurred before, it has worked fine on previous versions, even with python3-grpcio 1.30.2-3 from 21.04.

If I install grpcio via pip, it works fine, with latest version 1.44.x (>= 1.16.x and <= 1.44.0).

I've tried to debug it (GRPC_VERBOSITY=debug, GRPC_TRACE=tcp,http,api...), but it prints nothing. Analyzing it with strace shows some activity, but nothing like printing continuously logs like it would be stucked on some operation.

On a normal execution grpc.server.start() is executed and the app continues with other tasks, but on this case it never passes start().

In fact, the basic example from the grpcio website doesn't work either, it also consumes 100% of the CPU: http://grpc.io/docs/languages/python/basics/

I'm running out of ideas on how to debug this problem. Can someone help to identify the reason of this problem?

[0] - https://github.com/evilsocket/opensnitch/
[1] - https://github.com/evilsocket/opensnitch/issues/647

--

lsb_release -rd:

  Description: Ubuntu 22.04 LTS
  Release: 22.04

apt-cache policy python3-grpcio:

  Candidate: 1.30.2-3build6
  Version table:
     1.30.2-3build6 500

--

Steps to reproduce the problem:

 0. sudo apt install python3-grpcio

 1. clone grpcio basic example ( http://grpc.io/docs/languages/python/basics/ )
     $ git clone -b v1.45.0 --depth 1 --shallow-submodules https://github.com/grpc/grpc

 2. $ cd grpc/examples/python/route_guide

 3. on one terminal: $python3 route_guide_server.py
     Observe that the script is taking up 100% of the CPU

4. On another terminal launch the client and observe that it does nothing: $ python3 route_guide_client.py

The workaround installing grpcio via pip:

  $ sudo apt install python3-pip
  $ pip3 install --ignore-installed grpcio
  $ python3 route_guide_server.py
  (open another terminal)
  $ python3 route_guide_client.py

Observe that now the client is able to connect to the server and the CPU is not at 100%

[Michael Webster (miketwebster)]

I get similar behavior with Warpinator (https://github.com/linuxmint/warpinator/issues/127).

In this particular case, it causes a complete lock of the program.

Same symptom - 100% on a core. Using the pip grpcio version also resolves the issue.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Gustavo Iñiguez Goya (gooffy1)]

Hi,

Any update or comment on this bug?

Thank you.

[Eero-t-tamminen (eero-t-tamminen)]

Any updates? In some environments use of (specific) LTS releases is mandated.

While one can take "python3-grpcio" from previous 20.04 LTS, that requires downgrading also several other python modules.

Whereas taking newer "python3-grpcio" package from non-LTS 23.04 (lunar) or newer, would require even more packages to be upgraded (starting from Python itself) due to package deps.

[Willem van de Velde (williamvdvelde)]

Hi all,

i also ran into this bug and have solved the issue according: https://bugzilla.redhat.com/show_bug.cgi?id=1893533#c2

Debian LTO page: https://wiki.debian.org/ToolChain/LTO, mentions that you can simply disable LTO using:

export DEB_BUILD_MAINT_OPTIONS=optimize=-lto

in the debian/rules file.
So i downloaded the source package and added the flag and rebuilded the package which solves the problem for me.

Simple test case to reproduce:

apt install python3-grpcio
GRPC_VERBOSITY=debug python3
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import grpc
>>> channel = grpc.insecure_channel('127.0.0.1:1234')
^Z
[2]+ Stopped GRPC_VERBOSITY=debug python3

if you hit enter nothing appears on screen and you cannot exit the process using ctrl+c

after the rebuild of the package i get:

GRPC_VERBOSITY=debug python3
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import grpc
>>> channel = grpc.insecure_channel('127.0.0.1:1234')
D1207 09:45:38.927796848 110677 ev_posix.cc:173] Using polling engine: epollex
D1207 09:45:38.928035846 110677 lb_policy_registry.cc:39] registering LB policy factory for "grpclb"
D1207 09:45:38.928085556 110677 lb_policy_registry.cc:39] registering LB policy factory for "cds_experimental"
D1207 09:45:38.928108937 110677 lb_policy_registry.cc:39] registering LB policy factory for "eds_experimental"
D1207 09:45:38.928153489 110677 lb_policy_registry.cc:39] registering LB policy factory for "lrs_experimental"
D1207 09:45:38.928194958 110677 lb_policy_registry.cc:39] registering LB policy factory for "priority_experimental"
D1207 09:45:38.928218444 110677 lb_policy_registry.cc:39] registering LB policy factory for "weighted_target_experimental"
D1207 09:45:38.928245238 110677 lb_policy_registry.cc:39] registering LB policy factory for "xds_routing_experimental"
D1207 09:45:38.928271376 110677 lb_policy_registry.cc:39] registering LB policy factory for "pick_first"
D1207 09:45:38.928296893 110677 lb_policy_registry.cc:39] registering LB policy factory for "round_robin"
D1207 09:45:38.928327387 110677 dns_resolver_ares.cc:504] Using ares dns resolver
>>>

So can the option 'export DEB_BUILD_MAINT_OPTIONS=optimize=-lto' be added to the rules file?
Otherwise this package is not working at all.

Best regards,
Willem van de Velde

[Mitchell Dzurick (mitchdz)]

This seems to still be an issue in Jammy.

FWIW following[0] I installed a newer version of using

```
pip3 install --ignore-installed grpcio==1.44.0
```

And now it seems to work fine.

[0] - https://github.com/evilsocket/opensnitch/issues/647

[Mitchell Dzurick (mitchdz)]

I wonder if just a newer version of grpcio makes it work as well. I upgraded grpcio to 1.44 from the Jammy 1.30, but it's not apples to oranges as I had python3-grpcio 1.30.2-3build6 installed via the archive, and upgraded to the one installed by pip.

Looking past grpc, I was curious if building opensnitch itself in Jammy could help - especially since the issue seems to be around link time optimization. I did a quick test trying to backport the noble version to Jammy [0] and ran into build-time dependency issues. I haven't dug further than that yet.

[0] - https://launchpad.net/~mitchdz/+archive/ubuntu/opensnitch-testing