> /sbin/shutdown, post upgrade not working, imho is a severe bug

I suspect no-one's bothered to report it because as soon as the machine is (somehow) rebooted, the issue goes away (and reproduction then involves the pain of going through a trusty install + xenial upgrade cycle). Still, I'm not entirely convinced this *is* a bug; I'll come back to this in a bit below...

> I guess the scheduling of timeout is failing, but /sbin/shutdown now works?

I thought I'd tested this and concluded it didn't, but it turns out I just didn't wait long enough. When trying "shutdown -r now" there's a fair delay, then it complains about a connection time out, exits with an error, and the system reboots anyway!

    ubuntu@client1:~$ sudo shutdown -r now
    Failed to start reboot.target: Connection timed out
    See system logs and 'systemctl status reboot.target' for details.
    ubuntu@client1:~$ packet_write_wait: Connection to 10.123.236.170 port 22: Broken pipe

So, it is mostly likely the scheduling portion that's failing, and if given an immediate request the operation does still work. I haven't got time to test the -h switch as well, but I'd assume it's probably a similar story.

Now consider those two cases from the perspective of a human administrator:

If you've typed "shutdown -r now" you're expecting an immediate reboot. Even if some bits of the shutdown command fail, you still expect it to do what you asked: reboot immediately. However, if you've just typed "shutdown -r +5" you're expecting a 5 minute delay before reboot. If it can't schedule that delay would you rather it exit with an error and tell you, or just immediately reboot the system? Probably the former given you're not expecting an immediate reboot.

The issue in this case was that it isn't a human scheduling the reboot (directly) and the only reason landscape uses the schedule at all is to determine whether the shutdown command has (likely) worked. Which is a bit of an abuse of the shutdown command's scheduling facility, but I can't think of a better way off the top of my head (that'll reliably work in all implementations).

Anyway, in landscape's case we *would* rather the system immediately reboots when shutdown fails because the delay's only there for the Landscape client's benefit, and we can't guarantee the administrator has any other means of accessing the system for the purpose of rebooting it. So in a sense we're dealing with an interface mismatch: shutdown is (understandably) written for the (direct) use of a human administrator, and landscape is (ab)using it to provide reboot/poweroff facilities on (indirect) behalf of a human administrator.

So we come back to "I can't think of a better way of doing it" (that'll reliably work in all our aforementioned scenarios of upstart, systemd, and post-upgrade-almost-sort-of-systemd).

> Re differences between halt & poweroff, in systemd, eventually it results in:
> [snip]

Yes, but have a look at the arg parsing earlier on (https://github.com/systemd/systemd/blob/master/src/systemctl/systemctl.c#L8069):

    case 'h':
            if (arg_action != ACTION_HALT)
                    arg_action = ACTION_POWEROFF;
            break;

In systemd, the -H and --halt options mean halt, but -h is just an alias for -P and --poweroff (unless -H or --halt also appear in the argument list ... which looks a bit weird to me, but it is what their man-page states as well, so I guess it's intentional?). I haven't dug into upstart's implementation but I assume from your comments that it's different, and that -h actually means halt.

Thanks for the link on the differences between halt and power-off - I'd not seen that before!