Udev fails to start

[dean]

Updating udev and systemd to 239-8 will cause udev service to fail to start.
This will also cause the system to fail to boot

journalctl snippet:

raspberrypi systemd[1]: Starting udev Kernel Device Manager...
raspberrypi systemd[17518]: Operating on architecture: arm
raspberrypi systemd[17518]: Operating on architecture: arm
raspberrypi systemd-udevd[17518]: /lib/systemd/systemd-udevd: error while loading shared libraries: cannot restore segment prot after reloc: Operation
raspberrypi systemd[17518]: Operating on architecture: arm
raspberrypi systemd[17518]: systemd-udevd.service: Executing: /lib/systemd/systemd-udevd
raspberrypi systemd[1]: systemd-udevd.service: Child 17518 belongs to systemd-udevd.service.
raspberrypi systemd[1]: systemd-udevd.service: Main process exited, code=exited, status=127/n/a
raspberrypi systemd[1]: systemd-udevd.service: Failed with result 'exit-code'.
raspberrypi systemd[1]: systemd-udevd.service: Changed start -> failed
raspberrypi systemd[1]: systemd-udevd.service: Job systemd-udevd.service/start finished, result=failed
raspberrypi systemd[1]: Failed to start udev Kernel Device Manager.
raspberrypi systemd[1]: systemd-udevd.service: Unit entered failed state.
raspberrypi systemd[1]: systemd-udevd.service: Changed failed -> auto-restart
raspberrypi systemd[1]: systemd-udevd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
raspberrypi systemd[1]: systemd-udevd.service: Trying to enqueue job systemd-udevd.service/restart/fail
raspberrypi systemd[1]: systemd-udevd.service: Installed new job systemd-udevd.service/restart as 32370
raspberrypi systemd[1]: systemd-udevd.service: Enqueued job systemd-udevd.service/restart as 32370
raspberrypi systemd[1]: systemd-udevd.service: Scheduled restart job, restart counter is at 4.
raspberrypi systemd[1]: systemd-udevd.service: Changed auto-restart -> dead
raspberrypi systemd[1]: systemd-udevd.service: Job systemd-udevd.service/restart finished, result=done
raspberrypi systemd[1]: Stopped udev Kernel Device Manager.
raspberrypi systemd[1]: systemd-udevd.service: Converting job systemd-udevd.service/restart -> systemd-udevd.service/start
raspberrypi systemd[1]: systemd-udevd.service: ConditionPathIsReadWrite=/sys succeeded.
raspberrypi systemd[1]: systemd-udevd.service: Passing 2 fds to service
raspberrypi systemd[1]: systemd-udevd.service: About to execute: /lib/systemd/systemd-udevd
raspberrypi systemd[1]: systemd-udevd.service: Forked /lib/systemd/systemd-udevd as 17519
raspberrypi systemd[1]: systemd-udevd.service: Changed dead -> start

uname -a:
Linux raspberrypi 4.14.69-v7+ #1141 SMP Mon Sep 10 15:26:29 BST 2018 armv7l GNU/Linux
dpkg -s libc6 | grep ^Version:
Version: 2.27-6+rpi1

[dean (d-sha)]

https://bugs.launchpad.net/raspbian/+bug/1792818 may be related

[peter green (plugwash)]

Submitter said on irc that -7 worked for him

[peter green (plugwash)]

Thanks.

I have just set up a test environment to look at this, what I have confirmed so-far.

Raspbian 239-7 works
Raspbian 239-8 breaks
Debian armhf 239-8 works
Debian armhf 239-9 works

[peter green (plugwash)]

Some discussions on #debian-systemd IRC that may or may not be useful.

<mbiebl> plugwash: there is #908658
-zwiebelbot/#debian-systemd- Debian#908658: systemd: apt-get upgrade from 239-7 to 239-8 failed - https://bugs.debian.org/908658
-BTS/#debian-systemd- Closed #908358 in systemd by Michael Biebl (biebl) «Display stays off after computer resumes from suspend». https://bugs.debian.org/908358
<--snip-->
<plugwash> mbiebl, mmm that is just weird, that bug claims it appeared with -8 and went away again with -9 yet there is nothing relavent in the changelog
<plugwash> all I can think is that it was caused by some library at build time.
<mbiebl> that or the compiler
<mbiebl> (or build flags rather)
<mbiebl> not sure if that changed between -7 and -8 (and again between -8 and -9)
<--snip-->
<mbiebl> plugwash: looks like having this library installed trips of MemoryDenyWriteExecute
<--snip-->
<mbiebl> plugwash: if raspi-copies-and-fills fiddles e.g. with the memcpy implementation, it sounds like this could very well be the culprit
<plugwash> <mbiebl> https://raspberrypi.stackexchange.com/questions/72594/failed-to-start-udev-kernel-device-manager <-- i'm aware of that one but this guy claims not to have raspi-copies-and-fills installed, also this guy claims his system works with -7 but not -8 (we don't have -9 in raspbian yet, some of our autobuilders seem unable to successfully run the systemd testsuite) so it seems to be a seperate issue.
<mbiebl> the only relevant changes on the systemd side between -7 and -8 are
<mbiebl> https://salsa.debian.org/systemd-team/systemd/commit/40dca899f180066a7e72b8fd227b3ad2a9f056ce
<mbiebl> https://salsa.debian.org/systemd-team/systemd/commit/5da767304781d6d06e65a93b8de25d06727faa88
<mbiebl> I don't see how they could trigger this failure.
<plugwash> me neither but I don't think it's something we did either, which leads to a puzzle of what caused this. It's not 100% clear but I don't think https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=908658 was using raspbian as raspbian doesn't have -9 yet.
-zwiebelbot/#debian-systemd- Debian#908658: systemd: apt-get upgrade from 239-7 to 239-8 failed - https://bugs.debian.org/908658
<plugwash> *was not using raspbian
<plugwash> i'm currently in the process of setting up a sacrificial system to try and reproduce this.
<mbiebl> plugwash: hm, Distributor ID: Raspbian
<mbiebl> curious
<mbiebl> some kind of frankendebian?
<mbiebl> raspbian with buster apt source enabled?
<mbiebl> libc6 2.27-5+rpi1 looks like it's not a pristine buster system
* pipedream has quit (Ping timeout: 480 seconds)
<plugwash> very possibly a system with both Debian and Raspbian apt sources enabled.
<mbiebl> so if 239-8 is provided by raspbian, does it mean it get's recompiled there?
<mbiebl> maybe -8 was build with the raspbian toolchain and -9 came straight from Debian
<mbiebl> which would be some kind of explanation
<plugwash> right, everything in raspbian gets recompiled, there is not enough information in that bug report to know if he was using the Debian built -8 or the raspbian built -8.
<mbiebl> plugwash: it would be the most logical explanation though, that the failing - 8 version ...

[peter green (plugwash)]

Current working theory is that this was triggered by the change in default gcc from 7 to 8, but why it happened in Raspbian and not in Debian I do not know, possibly related to the different compiler settings we use. Possiblly related to the fact that the raspbian package was built with gcc 8.1 but the Debian package was built with gcc 8.2.

I am currently building a new version based on -9 to see if the issue is still reproducible with a binary built in a current raspbian environment, if it is I will try altering the build to force gcc 7 and seeing if that makes a difference.

[dean (d-sha)]

Commenting out MemoryDenyWriteExecute=yes and udev will start as mentioned in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=908658#37

this line is printed at the bottom of systemctl status udev:
systemd-udevd.service: Failed to send unit change signal for systemd-udevd.service: Connection reset by peer
but that is probably not relevant.

As mentioned on the debian bugreport I also have libffi6 installed
apt-cache policy libffi6
libffi6:
Installed: 3.2.1-8
Candidate: 3.2.1-8
Version table:
*** 3.2.1-8 500
500 http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
100 /var/lib/dpkg/status

[peter green (plugwash)]

Issue still reproducible with a package based on -9 built in a current raspbian buster environment.

Now trying to force gcc 7.

[peter green (plugwash)]

Forcing gcc-7 seemed to solve the issue, I am in the process of uploading now, hopefully it should hit the public repo soon.

[dean (d-sha)]

version -9 has fixed the issue for me Thank you plugwash!