cloud-init service failed to start with FileExistsError: [Errno 17] File exists: '/var/lib/cloud/data'

[Shreenidhi Shedi]

When I restart cloud-init in a loop with the following script, I get the error as mentioned in summary.

#Start of test script

#!/bin/bash

set -e

for i in {1..30}; do
  echo -n "$i "
  cloud-init clean -ls
  sleep 1
  systemctl restart cloud-*.service
done
echo ""

#End of test script

This is observed after applying fix: https://github.com/canonical/cloud-init/commit/0755cff078d5931e1d8e151bdcb84afb92bc0f02

Error message is:

```
Jun 23 18:37:23 phdev cloud-init[2240]: Traceback (most recent call last):
Jun 23 18:37:23 phdev cloud-init[2240]: File "/usr/bin/cloud-init", line 11, in <module>
Jun 23 18:37:23 phdev cloud-init[2240]: load_entry_point('cloud-init==20.2', 'console_scripts', 'cloud-init')()
Jun 23 18:37:23 phdev cloud-init[2240]: File "/usr/lib/python3.7/site-packages/cloudinit/cmd/main.py", line 893, in main
Jun 23 18:37:23 phdev cloud-init[2240]: get_uptime=True, func=functor, args=(name, args))
Jun 23 18:37:23 phdev cloud-init[2240]: File "/usr/lib/python3.7/site-packages/cloudinit/util.py", line 2618, in log_time
Jun 23 18:37:23 phdev cloud-init[2240]: ret = func(*args, **kwargs)
Jun 23 18:37:23 phdev cloud-init[2240]: File "/usr/lib/python3.7/site-packages/cloudinit/cmd/main.py", line 600, in status_wrapper
Jun 23 18:37:23 phdev cloud-init[2240]: util.ensure_dirs((data_d, link_d,))
Jun 23 18:37:23 phdev cloud-init[2240]: File "/usr/lib/python3.7/site-packages/cloudinit/util.py", line 1645, in ensure_dirs
Jun 23 18:37:23 phdev cloud-init[2240]: ensure_dir(d, mode)
Jun 23 18:37:23 phdev cloud-init[2240]: File "/usr/lib/python3.7/site-packages/cloudinit/util.py", line 1698, in ensure_dir
Jun 23 18:37:23 phdev cloud-init[2240]: os.makedirs(path)
Jun 23 18:37:23 phdev cloud-init[2240]: File "/usr/lib/python3.7/os.py", line 221, in makedirs
Jun 23 18:37:23 phdev cloud-init[2240]: mkdir(name, mode)
Jun 23 18:37:23 phdev cloud-init[2240]: FileExistsError: [Errno 17] File exists: '/var/lib/cloud/data'
Jun 23 18:37:23 phdev systemd[1]: cloud-init.service: Main process exited, code=exited, status=1/FAILURE
Jun 23 18:37:23 phdev systemd[1]: cloud-init.service: Failed with result 'exit-code'.
Jun 23 18:37:23 phdev systemd[1]: Failed to start Initial cloud-init job (metadata service crawler).
```

Find the attached for the log tarball.

[Dan Watkins (oddbloke)]

Hi Shreenidhi,

I think we discussed previously that `systemctl restart` may not restart the specified services in the same order that they would be executed in boot. So it isn't clear to me that this is a bug in cloud-init; it may just be a bug in your reproducer.

Also, looking at the log, I see:

  2020-06-23 18:37:25,535 - util.py[DEBUG]: Failed to write boot finished file /var/lib/cloud/instance/boot-finished
  Traceback (most recent call last):
    File "/usr/lib/python3.7/site-packages/cloudinit/config/cc_final_message.py", line 81, in handle
      util.write_file(boot_fin_fn, contents, ensure_dir_exists=False)
  TypeError: write_file() got an unexpected keyword argument 'ensure_dir_exists'

which suggests that you only partially applied the patch.

If you see this issue during boot (or you find a more robust reproducer), please do run `cloud-init collect-logs` on an affected instance, attach the tarball here and move this bug back to New.

Thanks!

Dan

[Shreenidhi Shedi (sshedi)]

Hi Dan,

Thanks for the response, you're right, my patch was incomplete and I rectified it and now everything seems to be fine. If I face any issues again, I'll file a new bug, please close this one.

--
Shedi

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3749