umount /run/cloud-init/tmp/tmpl5n7csdd failed

[norman shen]

Hello,

I am using cloud-init version: /usr/bin/cloud-init 20.4.1-0ubuntu1~18.04.1, ubuntu version is root@ubuntu:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic

I found that umount configdrive fails with device busy reported, it further causes temp folder failed to be deleted.

Logs are

```
2022-07-25 02:13:01,732 - handlers.py[DEBUG]: finish: init-local/search-ConfigDrive: FAIL: no local data found from DataSourceConfigDrive
2022-07-25 02:13:01,733 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceConfigDrive.DataSourceConfigDrive'> failed
2022-07-25 02:13:01,733 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceConfigDrive.DataSourceConfigDrive'> failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/temp_utils.py", line 90, in tempdir
    yield tdir
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1687, in mount_cb
    return ret
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1571, in unmounter
    subp.subp(umount_cmd)
  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 295, in subp
    cmd=args)
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['umount', '/run/cloud-init/tmp/tmpl5n7csdd']
Exit code: 32
Reason: -
Stdout:
Stderr: umount: /run/cloud-init/tmp/tmpl5n7csdd: target is busy.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 771, in find_source
    if s.update_metadata([EventType.BOOT_NEW_INSTANCE]):
  File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 660, in update_metadata
    result = self.get_data()
  File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 279, in get_data
    return_value = self._get_data()
  File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceConfigDrive.py", line 81, in _get_data
    mtype=mtype)
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1687, in mount_cb
    return ret
  File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/cloudinit/temp_utils.py", line 92, in tempdir
    shutil.rmtree(tdir, ignore_errors=rmtree_ignore_errors)
  File "/usr/lib/python3.6/shutil.py", line 486, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 444, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/lib/python3.6/shutil.py", line 442, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
OSError: [Errno 30] Read-only file system: 'network_data.json'
2022-07-25 02:13:01,783 - main.py[DEBUG]: No local datasource found

```

[Chad Smith (chad.smith)]

Thanks for filing a bug and helping make cloud-init better.

I'm having a hard time with these logs deterimining what may be contributing to the configdrive device busy. I suppose we could add a bit of retry logic on device busy errors here, but I'm not sure whether this is due to the underlying system being under high load or some other symptom that leads to this.

Would it be possible to run `cloud-init collect-logs` and attach the resulting tarfile of logs/system configuration to this bug to help us triage.
Note if providing
collect-logs --include-userdata that you may want to inspect the userdata included to make sure no secrets or passwords are included in the tarfile.

Much thanks.

[Chad Smith (chad.smith)]

I'll set this bug to incomplete status, please reset it to new when you are able to provide the complete /var/log/cloud-init.log.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[norman shen (jshen28)]

thank you for reply,

  I am sorry but the virtual machine has been deleted and I current found no way to reproduce it..

Best,
Norman

[Brett Holman (holmanb)]

Cloud-init assumes it can mount/unmount the config drive filesystem. This kind of failure could easily be caused by external processes to cloud-init that don't realize they are interfering by accessing files on this filesystem.

> /usr/bin/cloud-init 20.4.1-0ubuntu1~18.04.1

FYI: This reported release is over a year old and contains bugs that have been fixed in recent releases on Bionic.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/4002