systemd-timedated fails to start with /tmp as a symlink

[Sebastian Wiesendahl]

When I try to start the `gnome-control-center`, then it hangs after a few seconds.

I recently changed the timezone due to travelling. I can login normally.

Another source of the error could be, that I symlinked the `/tmp` and `/opt` directory to another partition.

```
$ gnome-control-center

(gnome-control-center:29386): datetime-cc-panel-WARNING **: 10:13:23.508: could not get proxy for DateTimeMechanism: Fehler beim Aufruf von StartServiceByName für org.freedesktop.timedate1: GDBus.Error:org.freedesktop.DBus.Error.TimedOut: Failed to activate service 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms)
**
ERROR:../shell/cc-panel.c:202:cc_panel_size_allocate: assertion failed: (child)
Abgebrochen (Speicherabzug geschrieben)
```

```
$ gdbus introspect --system --dest org.freedesktop.timedate1 --object-path /org/freedesktop/timedate1
Fehler: Zeitüberschreitung wurde erreicht
```

```
$ gdbus introspect --system --dest org.freedesktop.timedate1 --object-path /
Fehler: GDBus.Error:org.freedesktop.DBus.Error.TimedOut: Failed to activate service 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms)
```

```
$ timedatectl
Failed to query server: Failed to activate service 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms)
```

```
$ timedatectl --version
systemd 237
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid
```

systemd-timedated service is not starting:
```
$ sudo systemctl status systemd-timedated.service
● systemd-timedated.service - Time & Date Service
   Loaded: loaded (/lib/systemd/system/systemd-timedated.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2019-02-19 10:47:33 CST; 2min 53s ago
     Docs: man:systemd-timedated.service(8)
           man:localtime(5)
           https://www.freedesktop.org/wiki/Software/systemd/timedated
  Process: 3618 ExecStart=/lib/systemd/systemd-timedated (code=exited, status=226/NAMESPACE)
 Main PID: 3618 (code=exited, status=226/NAMESPACE)

Feb 19 10:47:33 home systemd[1]: Starting Time & Date Service...
Feb 19 10:47:33 home systemd[3618]: systemd-timedated.service: Failed to set up mount namespacing: No such file or directory
Feb 19 10:47:33 home systemd[3618]: systemd-timedated.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-timedated: No such file or directory
Feb 19 10:47:33 home systemd[1]: systemd-timedated.service: Main process exited, code=exited, status=226/NAMESPACE
Feb 19 10:47:33 home systemd[1]: systemd-timedated.service: Failed with result 'exit-code'.
Feb 19 10:47:33 home systemd[1]: Failed to start Time & Date Service.
```

```
$ journalctl -xe
Feb 19 18:03:59 home ntpd[21785]: 176.221.42.125 local addr 192.168.0.102 -> <null>
Feb 19 18:04:05 home sudo[17719]: sebastian : TTY=pts/0 ; PWD=/home/sebastian ; USER=root ; COMMAND=/bin/systemctl start systemd-timedated
Feb 19 18:04:05 home sudo[17719]: pam_unix(sudo:session): session opened for user root by (uid=0)
Feb 19 18:04:05 home systemd[1]: Starting Time & Date Service...
-- Subject: Unit systemd-timedated.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit systemd-timedated.service has begun starting up.
Feb 19 18:04:05 home systemd[17724]: systemd-timedated.service: Failed to set up mount namespacing: No such file or directory
Feb 19 18:04:05 home systemd[17724]: systemd-timedated.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-timedated: No such file or directory
-- Subject: Process /lib/systemd/systemd-timedated could not be executed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The process /lib/systemd/systemd-timedated could not be executed and failed.
--
-- The error number returned by this process is 2.
Feb 19 18:04:05 home systemd[1]: systemd-timedated.service: Main process exited, code=exited, status=226/NAMESPACE
Feb 19 18:04:05 home systemd[1]: systemd-timedated.service: Failed with result 'exit-code'.
Feb 19 18:04:05 home systemd[1]: Failed to start Time & Date Service.
-- Subject: Unit systemd-timedated.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit systemd-timedated.service has failed.
--
-- The result is RESULT.
Feb 19 18:04:05 home sudo[17719]: pam_unix(sudo:session): session closed for user root
```

NTP deamon is running:
```
$ sudo service ntp status
● ntp.service - Network Time Service
   Loaded: loaded (/lib/systemd/system/ntp.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-02-19 10:42:37 CST; 2min 54s ago
     Docs: man:ntpd(8)
  Process: 21763 ExecStart=/usr/lib/ntp/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
 Main PID: 21785 (ntpd)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/ntp.service
           └─21785 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 127:135

Feb 19 10:42:40 home ntpd[21785]: Soliciting pool server 136.243.177.133
Feb 19 10:42:40 home ntpd[21785]: Soliciting pool server 2001:638:504:2000::36
Feb 19 10:42:41 home ntpd[21785]: Soliciting pool server 2001:418:3ff::53
Feb 19 10:42:41 home ntpd[21785]: Soliciting pool server 78.46.231.23
Feb 19 10:42:41 home ntpd[21785]: Soliciting pool server 129.70.132.32
Feb 19 10:42:41 home ntpd[21785]: Soliciting pool server 51.15.191.239
Feb 19 10:42:42 home ntpd[21785]: Soliciting pool server 129.70.132.33
Feb 19 10:42:42 home ntpd[21785]: Soliciting pool server 2a02:c205:2009:8290::1
Feb 19 10:42:42 home ntpd[21785]: Soliciting pool server 129.70.132.35
Feb 19 10:42:42 home ntpd[21785]: Soliciting pool server 2001:67c:1560:8003::c8
```

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: gnome-control-center 1:3.28.2-0ubuntu0.18.04.2
ProcVersionSignature: Ubuntu 4.15.0-45.48-generic 4.15.18
Uname: Linux 4.15.0-45-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Tue Feb 19 10:13:56 2019
InstallationDate: Installed on 2018-12-12 (68 days ago)
InstallationMedia: Ubuntu 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180725)
SourcePackage: gnome-control-center
UpgradeStatus: No upgrade log present (probably fresh install)

[Sebastien Bacher (seb128)]

The issue is with systemd, /lib/systemd/systemd-timedated is part of that package

What's the output of those commands
- dpkg -l | grep systemd
- debsums systemd
?

[Sebastian Wiesendahl (swiesend)]

@Sebastien Bacher thank you for looking into this!

```
$ dpkg -l | grep systemd
ii dbus-user-session 1.12.2-1ubuntu1 amd64 simple interprocess messaging system (systemd --user integration)
ii libnss-systemd:amd64 237-3ubuntu10.13 amd64 nss module providing dynamic user and group name resolution
ii libpam-systemd:amd64 237-3ubuntu10.13 amd64 system and service manager - PAM module
ii libsystemd0:amd64 237-3ubuntu10.13 amd64 systemd utility library
ii networkd-dispatcher 1.7-0ubuntu3.3 all Dispatcher service for systemd-networkd connection status changes
ii python3-systemd 234-1build1 amd64 Python 3 bindings for systemd
ii systemd 237-3ubuntu10.13 amd64 system and service manager
ii systemd-sysv 237-3ubuntu10.13 amd64 system and service manager - SysV links
```

See last line: Permission denied.
```
$ debsums systemd
/bin/journalctl OK
/bin/loginctl OK
/bin/networkctl OK
/bin/systemctl OK
/bin/systemd-ask-password OK
/bin/systemd-escape OK
/bin/systemd-inhibit OK
/bin/systemd-machine-id-setup OK
/bin/systemd-notify OK
/bin/systemd-sysusers OK
/bin/systemd-tmpfiles OK
/bin/systemd-tty-ask-password-agent OK
/lib/lsb/init-functions.d/40-systemd OK
/lib/modprobe.d/systemd.conf OK
/lib/systemd/libsystemd-shared-237.so OK
/lib/systemd/network/80-container-host0.network OK
/lib/systemd/network/80-container-ve.network OK
/lib/systemd/network/80-container-vz.network OK
/lib/systemd/resolv.conf OK
/lib/systemd/set-cpufreq OK
/lib/systemd/system-generators/systemd-cryptsetup-generator OK
/lib/systemd/system-generato...

[Sebastian Wiesendahl (swiesend)]

```
root@home:/var/lib/polkit-1/localauthority/10-vendor.d# ll
insgesamt 44
drwxr-xr-x 2 root root 4096 Feb 19 17:12 ./
drwxr-xr-x 7 root root 4096 Jul 25 2018 ../
-rw-r--r-- 1 root root 196 Apr 20 2018 50-com.canonical.indicator.sound.AccountsService.pkla
-rw-r--r-- 1 root root 154 Nov 9 07:42 60-libvirt.pkla
-rw-r--r-- 1 root root 2363 Nov 6 2015 com.ubuntu.desktop.pkla
-rw-r--r-- 1 root root 135 Sep 11 20:33 fwupd.pkla
-rw-r--r-- 1 root root 217 Jul 14 2017 geoclue-2.0.pkla
-rw-r--r-- 1 root root 485 Okt 31 13:01 gnome-initial-setup.pkla
-rw-r--r-- 1 root root 212 Mär 26 2018 org.freedesktop.NetworkManager.pkla
-rw-r--r-- 1 root root 209 Nov 9 02:24 org.freedesktop.packagekit.pkla
-rw-r--r-- 1 root root 199 Jan 28 2018 systemd-networkd.pkla
```

```
root@home:/var/lib/polkit-1/localauthority/10-vendor.d# cat systemd-networkd.pkla
[Allow systemd-networkd to set timezone and transient hostname]
Identity=unix-user:systemd-network
Action=org.freedesktop.hostname1.set-hostname;org.freedesktop.timedate1.set-timezone;
ResultAny=yes
```

[Sebastien Bacher (seb128)]

Weird, is /lib in another partition or something?

the log has
/lib/systemd/systemd-timedated OK

but the bug description
spawning /lib/systemd/systemd-timedated: No such file or directory

What happens if you start that binary manually?

[Sebastian Wiesendahl (swiesend)]

Nope, `/lib` is on the system's partition.

Besides / there is the /home partition

only those a symlinks:

`/opt` -> `/home/opt`
`/tmp` -> `/home/tmp`

```
sebastian@home:/$ ll
insgesamt 1551233
drwxr-xr-x 25 root root 4096 Feb 13 20:12 ./
drwxr-xr-x 25 root root 4096 Feb 13 20:12 ../
drwxr-xr-x 2 root root 4096 Feb 19 14:28 bin/
drwxr-xr-x 5 root root 1024 Feb 19 14:28 boot/
drwxr-xr-x 2 root root 4096 Dez 13 00:54 cdrom/
drwxr-xr-x 23 root root 4660 Feb 20 13:32 dev/
drwxr-xr-x 162 root root 12288 Feb 20 13:42 etc/
drwxr-xr-x 6 root root 4096 Feb 13 20:12 home/
lrwxrwxrwx 1 root root 33 Jan 31 16:55 initrd.img -> boot/initrd.img-4.15.0-45-generic
lrwxrwxrwx 1 root root 33 Jan 31 16:55 initrd.img.old -> boot/initrd.img-4.15.0-44-generic
drwxr-xr-x 25 root root 4096 Dez 13 20:17 lib/
drwxr-xr-x 2 root root 4096 Jan 31 23:51 lib32/
drwxr-xr-x 2 root root 4096 Jul 25 2018 lib64/
drwxr-xr-x 2 root root 4096 Jan 31 23:51 libx32/
drwx------ 2 root root 16384 Dez 13 00:53 lost+found/
drwxr-xr-x 3 root root 4096 Dez 13 02:01 media/
drwxr-xr-x 2 root root 4096 Jul 25 2018 mnt/
lrwxrwxrwx 1 root root 9 Feb 13 20:02 opt -> /home/opt/
dr-xr-xr-x 329 root root 0 Feb 20 13:30 proc/
drwx------ 10 root root 4096 Jan 11 21:09 root/
drwxr-xr-x 40 root root 1340 Feb 20 13:32 run/
drwxr-xr-x 2 root root 12288 Feb 19 14:28 sbin/
drwxr-xr-x 9 root root 4096 Dez 20 18:40 snap/
drwxr-xr-x 2 root root 4096 Jul 25 2018 srv/
-rw------- 1 root root 1588346880 Dez 13 00:53 swapfile
dr-xr-xr-x 13 root root 0 Feb 20 13:30 sys/
lrwxrwxrwx 1 root root 9 Feb 13 20:02 tmp -> /home/tmp/
drwx------ 4 root root 4096 Dez 17 23:16 .Trash-0/
drwxr-xr-x 12 root root 4096 Jan 31 23:51 usr/
drwxr-xr-x 15 root root 4096 Dez 16 21:54 var/
lrwxrwxrwx 1 root root 30 Jan 31 16:55 vmlinuz -> boot/vmlinuz-4.15.0-45-generic
lrwxrwxrwx 1 root root 30 Jan 31 16:55 vmlinuz.old -> boot/vmlinuz-4.15.0-44-generic
```

Running /lib/systemd/systemd-timedated manually
```
sebastian@home:~$ /lib/systemd/systemd-timedated
Failed to run event loop: Transport endpoint is not connected
sebastian@home:~$ sudo /lib/systemd/systemd-timedated
[sudo] Passwort für sebastian:
sebastian@home:~$ gnome-control-center
Failed to register: GDBus.Error:org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying
sebastian@home:~$ sudo /lib/systemd/systemd-timedated
sebastian@home:~$ gnome-control-center

(gnome-control-center:29931): datetime-cc-panel-WARNING **: 18:14:43.899: could not get proxy for DateTimeMechanism: Fehler beim Aufruf von StartServiceByName für org.freedesktop.timedate1: GDBus.Error:org.freedesktop.DBus.Error.TimedOut: Failed to activate service 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms)
**
ERROR:../shell/cc-panel.c:202:cc_panel_size_allocate: assertion failed: (child)
Abgebrochen (Speicherabzug geschrieben)
sebastian@home:~$ sudo /lib/systemd/systemd-timedated
sebastian@home:...

[Sebastian Wiesendahl (swiesend)]

that are my permissions on /home:

```
sebastian@home:/home$ ll
insgesamt 44
drwxr-xr-x 6 root root 4096 Feb 13 20:12 ./
drwxr-xr-x 25 root root 4096 Feb 13 20:12 ../
drwx------ 2 root root 16384 Dez 13 00:53 lost+found/
drwxr-xr-x 11 sebastian sebastian 4096 Feb 13 19:43 opt/
drwxr-xr-x 59 sebastian sebastian 4096 Feb 20 17:59 sebastian/
drwxrwxrwt 66 root root 12288 Feb 20 18:19 tmp/
```

[Sebastian Wiesendahl (swiesend)]

@Sebastien Bacher thank you very much for looking into this issue. I don't know if it is worth tracking this further.

I just moved the /tmp directory back to the / partition, to avoid further trouble. And this actually solved the issue.

Like they did here:
https://bbs.archlinux.org/viewtopic.php?id=217072

where this was mentioned:
```
systemd[476]: systemd-timesyncd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-timesyncd: Too many levels of symbolic links
systemd[1]: systemd-timesyncd.service: Main process exited, code=exited, status=226/NAMESPACE
```

I gave again full access `chmod 777 /tmp` and set the sticky flag `chmod +t /tmp`:
```
sebastian@home:/$ ll
insgesamt 1551245
drwxr-xr-x 26 root root 4096 Feb 20 21:24 ./
drwxr-xr-x 26 root root 4096 Feb 20 21:24 ../
drwxr-xr-x 2 root root 4096 Feb 19 14:28 bin/
drwxr-xr-x 5 root root 1024 Feb 19 14:28 boot/
drwxr-xr-x 2 root root 4096 Dez 13 00:54 cdrom/
drwxr-xr-x 23 root root 4660 Feb 20 21:25 dev/
drwxr-xr-x 162 root root 12288 Feb 20 19:18 etc/
drwxr-xr-x 6 root root 4096 Feb 13 20:12 home/
lrwxrwxrwx 1 root root 33 Jan 31 16:55 initrd.img -> boot/initrd.img-4.15.0-45-generic
lrwxrwxrwx 1 root root 33 Jan 31 16:55 initrd.img.old -> boot/initrd.img-4.15.0-44-generic
drwxr-xr-x 25 root root 4096 Dez 13 20:17 lib/
drwxr-xr-x 2 root root 4096 Jan 31 23:51 lib32/
drwxr-xr-x 2 root root 4096 Jul 25 2018 lib64/
drwxr-xr-x 2 root root 4096 Jan 31 23:51 libx32/
drwx------ 2 root root 16384 Dez 13 00:53 lost+found/
drwxr-xr-x 3 root root 4096 Dez 13 02:01 media/
drwxr-xr-x 2 root root 4096 Jul 25 2018 mnt/
lrwxrwxrwx 1 root root 9 Feb 13 20:02 opt -> /home/opt/
dr-xr-xr-x 298 root root 0 Feb 20 21:25 proc/
drwx------ 10 root root 4096 Jan 11 21:09 root/
drwxr-xr-x 40 root root 1320 Feb 20 21:25 run/
drwxr-xr-x 2 root root 12288 Feb 19 14:28 sbin/
drwxr-xr-x 9 root root 4096 Dez 20 18:40 snap/
drwxr-xr-x 2 root root 4096 Jul 25 2018 srv/
-rw------- 1 root root 1588346880 Dez 13 00:53 swapfile
dr-xr-xr-x 13 root root 0 Feb 20 21:25 sys/
drwxrwxrwt 16 root root 12288 Feb 20 21:27 tmp/
drwx------ 4 root root 4096 Dez 17 23:16 .Trash-0/
drwxr-xr-x 12 root root 4096 Jan 31 23:51 usr/
drwxr-xr-x 15 root root 4096 Dez 16 21:54 var/
lrwxrwxrwx 1 root root 30 Jan 31 16:55 vmlinuz -> boot/vmlinuz-4.15.0-45-generic
lrwxrwxrwx 1 root root 30 Jan 31 16:55 vmlinuz.old -> boot/vmlinuz-4.15.0-44-generic
```

[Sebastien Bacher (seb128)]

Thank for the debugging, I'm updating the title to reflect better the issue and it's up to the systemd team then to see how important they believe the issue to be

[Dan Streetman (ddstreet)]

please reopen if this is still an issue