Ubuntu
ec2-hibinit-agent package

  1. Bug #1840909
  2. Comment #0

Comment 0 for bug 1840909

Revision history for this message
Sam Stenvall (negge) wrote : ec2-hibinit-agent-ignore-powerkey.conf prevents EC2 instances from stopping normally

Recently I've noticed a bunch of related issues with our AWS EC2 instances:

* stopping takes forever
* terminating takes forever (probably because it tries to stop first)
* lots of dangling nodes in our Consul cluster

Today I decided to debug what was going on. At first I thought it was something that we do to our AMIs that was the issue, but after starting a vanilla Ubuntu 18.04 official AMI (0cdab515472ca0bac to be exact) I could replicate the issue.

What happens is that you get "systemd-logind[816]: Power key pressed" in the journal when you issue a Stop action against your EC2 instance. However, after that nothing happens, until 300 seconds have passed and AWS terminates your instance instead. This means nothing exits cleanly, which explains why Consul nodes are left dangling.

At first I thought it was a bug in systemd-logind, until I found /usr/lib/systemd/logind.conf.d/ec2-hibinit-agent-ignore-powerkey.conf, containing:

[Login]
HandlePowerKey=ignore

Removing this file or uncommenting the last line fixes the problem.

So in effect this package completely prevents the normal shutdown mechanism from working correctly. I'm currently working on a workaround for this for our AMI building process but an official fix would be nice.

Just remove the file, it doesn't even come from upstream, but since it has been in this repository since version 1.0.0 I can't find anything in the git history regarding *why* it was added.