netifaces.c:1:10: fatal error: Python.h: No such file or directory

[Ananya Banerjee]

tripleo-ci-centos-9-content-provider failing in stable/wallaby with netifaces.c:1:10: fatal error: Python.h: No such file or directory

We see this in TASK [run-test : run toci_gate_test.sh]

2022-02-21 14:22:59.047806 | primary | Building wheel for netifaces (setup.py): started
2022-02-21 14:23:00.928884 | primary | Building wheel for netifaces (setup.py): finished with status 'error'
2022-02-21 14:23:00.936893 | primary | error: subprocess-exited-with-error
2022-02-21 14:23:00.936926 | primary |
2022-02-21 14:23:00.936946 | primary | × python setup.py bdist_wheel did not run successfully.
2022-02-21 14:23:00.936951 | primary | │ exit code: 1
2022-02-21 14:23:00.936957 | primary | ╰─> [20 lines of output]
2022-02-21 14:23:00.936963 | primary | running bdist_wheel
2022-02-21 14:23:00.936968 | primary | running build
2022-02-21 14:23:00.936973 | primary | running build_ext
2022-02-21 14:23:00.936979 | primary | checking for getifaddrs...found.
2022-02-21 14:23:00.936984 | primary | checking for getnameinfo...found.
2022-02-21 14:23:00.936989 | primary | checking for IPv6 socket IOCTLs...not found.
2022-02-21 14:23:00.937004 | primary | checking for optional header files...netash/ash.h netatalk/at.h netax25/ax25.h neteconet/ec.h netipx/ipx.h netpacket/packet.h netrose/rose.h linux/atm.h linux/llc.h linux/tipc.h linux/dn.h.
2022-02-21 14:23:00.937011 | primary | checking whether struct sockaddr has a length field...no.
2022-02-21 14:23:00.937017 | primary | checking which sockaddr_xxx structs are defined...at ax25 in in6 ipx un rose ash ec ll atmpvc atmsvc dn llc.
2022-02-21 14:23:00.937033 | primary | checking for routing socket support...no.
2022-02-21 14:23:00.937038 | primary | checking for sysctl(CTL_NET...) support...no.
2022-02-21 14:23:00.937043 | primary | checking for netlink support...yes.
2022-02-21 14:23:00.937048 | primary | will use netlink to read routing table
2022-02-21 14:23:00.937053 | primary | building 'netifaces' extension
2022-02-21 14:23:00.937060 | primary | gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DNETIFACES_VERSION=0.10.9 -DHAVE_GETIFADDRS=1 -DHAVE_GETNAMEINFO=1 -DHAVE_NETASH_ASH_H=1 -DHAVE_NETATALK_AT_H=1 -DHAVE_NETAX25_AX25_H=1 -DHAVE_NETECONET_EC_H=1 -DHAVE_NETIPX_IPX_H=1 -DHAVE_NETPACKET_PACKET_H=1 -DHAVE_NETROSE_ROSE_H=1 -DHAVE_LINUX_ATM_H=1 -DHAVE_LINUX_LLC_H=1 -DHAVE_LINUX_TIPC_H=1 -DHAVE_LINUX_DN_H=1 -DHAVE_SOCKADDR_AT=1 -DHAVE_SOCKADDR_AX25=1 -DHAVE_SOCKADDR_IN=1 -DHAVE_SOCKADDR_IN6=1 -DHAVE_SOCKADDR_IPX=1 -DHAVE_SOCKADDR_UN=1 -DHAVE_SOCKADDR_ROSE=1 -DHAVE_SOCKADDR_ASH=1 -DHAVE_SOCKADDR_EC=1 -DHAVE_SOCKADDR_LL=1 -DHAVE_SOCKADDR_ATMPVC=1 -DHAVE_SOCKADDR_ATMSVC=1 -DHAVE_SOCKADDR_DN=1 -DHAVE_SOCKADDR_LLC=1 -DHAVE_PF_NETLINK=1 -I/home/zuul/workspace/.quickstart/include -I/usr/include/python3.9 -c netifaces.c -o build/temp.linux-x86_64-3.9/netifaces.o
2022-02-21 14:23:00.937073 | primary | netifaces.c:1:10: fatal error: Python.h: No such file or directory
2022-02-21 14:23:00.937079 | primary | 1 | #include <Python.h>
2022-02-21 14:23:00.937084 | primary | | ^~~~~~~~~~
2022-02-21 14:23:00.937089 | primary | compilation terminated.
2022-02-21 14:23:00.937095 | primary | error: command '/usr/bin/gcc' failed with exit code 1
2022-02-21 14:23:00.937099 | primary | [end of output]
2022-02-21 14:23:00.937105 | primary |
2022-02-21 14:23:00.937110 | primary | note: This error originates from a subprocess, and is likely not a problem with pip.
2022-02-21 14:23:00.937370 | primary | ERROR: Failed building wheel for netifaces
2022-02-21 14:23:00.937804 | primary | Running setup.py clean for netifaces

Logs:
https://962efa85a3cd561cd4ca-779b81139f4f29276caf956abf2a020f.ssl.cf5.rackcdn.com/830156/2/check/tripleo-ci-centos-9-content-provider/f8671c8/job-output.txt
https://5da20fcad62473d26298-fe304f23412f432b70850b4eb068f885.ssl.cf2.rackcdn.com/830156/2/check/tripleo-ci-centos-9-content-provider/d6c2ff1/job-output.txt

We can see this error from at least 2022-02-21 10:53:04

[Ananya Banerjee (frenzyfriday)]

In the passing jobs (master branch) the version of netifaces being downloaded is 0.11.0

Downloading https://mirror.gra1.ovh.opendev.org/pypifiles/packages/6b/07/613110af7b7856cf0bea173a866304f5476aba06f5ccf74c66acc73e36f1/netifaces-0.11.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl

In failing jobs (wallaby branch) the version is 0.10.9

Collecting netifaces>=0.10.4
2022-02-21 14:22:39.927192 | primary | Downloading https://mirror.bhs1.ovh.opendev.org/pypifiles/packages/0d/18/fd6e9c71a35b67a73160ec80a49da63d1eed2d2055054cc2995714949132/netifaces-0.10.9.tar.gz (28 kB)

[Ananya Banerjee (frenzyfriday)]

And in master it downloads wheel whereas in wallaby it downloads tar and compiles afterwards.

[Ananya Banerjee (frenzyfriday)]

Maybe something like these [1] [2] are missing in wallaby?

[1] https://review.opendev.org/c/zuul/zuul-jobs/+/728708

[2] https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/794158

[Jiri Podivin (jpodivin)]

#3
This could be it. I believe it would be prudent to follow existing convention of job differentiation by branches as we do with centos-8 content provider.[3] It could make things simpler down the line.

[3] https://zuul.opendev.org/t/openstack/jobs

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/830336

[Marios Andreou (marios-b)]

the patch at https://review.opendev.org/c/openstack/tripleo-quickstart/+/830336 seems to help us

(testing with https://review.opendev.org/c/openstack/python-tripleoclient/+/830165/2#message-405425649f7cc66b38ff0f256cf78fe0df267ce2 and it hasn't reported yet but the content providers are now passing in https://zuul.openstack.org/status#830132).

We can go with tripleo-quickstart/+/830336 (which also fixes a pre-existing/unrelated bug) to unblock us but I am not sure this is the final/right fix.

For some reason and as noted in comment #1 - there is a difference between master and wallaby C9. For master we are downloading the wheel but for wallaby we are building.

[Alan Pevec (apevec)]

sorry for maybe OT, but why is this installed from python wheel, don't we have python RPM for those deps?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/830336
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/233aaf7cedd10ec6921ebaee2d81953aaa0d20ff
Submitter: "Zuul (22348)"
Branch: master

commit 233aaf7cedd10ec6921ebaee2d81953aaa0d20ff
Author: Marios Andreou <email address hidden>
Date: Tue Feb 22 11:35:13 2022 +0200

    Install python3-devel - building netifaces c9 content provider

    As seen in related bug we need this for the c9 wallaby content
    provider. Also fixes existing bug where we install only the
    first in PYTHON_PACKAGES ignoring the rest.

    Related-Bug: 1961736
    Change-Id: I16e2bf179d462420942ff1d293b5541c587ff9e8

[Marios Andreou (marios-b)]

@Alan that is a good question - so one part is why do we download wheel on master but try to build it on wallaby. Your suggestion is lets try rpm install instead?

We aren't installing this explicitly - right now it (netifaces) is coming in as dependency for one of the things in https://opendev.org/openstack/tripleo-quickstart/src/commit/ecdbc6aaf04b94a50ab2a590af504d51a237cf62/quickstart.sh#L182-L183

e.g. looks like that in output:

2022-02-21 17:20:59.418421 | primary | running install_scripts
2022-02-21 17:20:59.446582 | primary | Running in bootstrap: python3 -m pip install --force-reinstall -r requirements.txt -r quickstart-extras-requirements.txt

so we could add an explicit install for the rpm or add it to one of those ^^ requirements files. Note though there is a whole bunch of those installed/built so will be try to add all of them to our explicit requirements - yes i think we do.

Right now we have gone with https://review.opendev.org/c/openstack/tripleo-quickstart/+/830336 since it also fixes a pre-existing bash bug and should be OK (install python-devel)

[Marios Andreou (marios-b)]

going to move the bug fix released for now - we may still do further work here but at least for the purposes of the original bug report we are no longer blocked on this