Error reading package list on ephemeral environemt

[Larry Michel]

Initially saw those errors "Stale file handle" with bug 1621065. Now, we're seeing them again which results in a number of failed deployments. Those are mainly with Trusty but I was obviously able to recreate the error with the Xenial while assisted by the preseeds.

What's strange is I was able to deploy Trusty one system that was freshly commissioned but was still being onboarded. However, upon trying to recommission the failing servers, they failed commissioning (powered off after PXE commissioning before timing out while powered off).

The error from cloud-init-output:

Hit http://archive.ubuntu.com trusty/multiverse amd64 Packages
Fetched 2083 kB in 3s (596 kB/s)
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://archive.ubuntu.com trusty-backports InRelease: Clearsigned file isn't valid, got 'NODATA' (does the network require authentication?)

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/InRelease rename failed, Stale file handle (/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_trusty-updates_InRelease -> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_InRelease).

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-backports/InRelease rename failed, Stale file handle (/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_trusty-backports_InRelease.reverify -> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-backports_InRelease).

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/Release rename failed, Stale file handle (/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_trusty_Release.gpg -> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_Release.gpg).

W: Failed to fetch copy:/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages rename failed, Stale file handle (/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages -> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages).

W: Failed to fetch copy:/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_trusty_restricted_binary-amd64_Packages rename failed, Stale file handle (/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_trusty_restricted_binary-amd64_Packages -> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_restricted_binary-amd64_Packages).

W: Failed to fetch copy:/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages rename failed, Stale file handle (/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages -> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages).

W: Failed to fetch copy:/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_trusty_multiverse_binary-amd64_Packages rename failed, Stale file handle (/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_trusty_multiverse_binary-amd64_Packages -> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_multiverse_binary-amd64_Packages).

W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
Package xfsprogs is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package lvm2 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package python-oauthlib is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'lvm2' has no installation candidate
E: Package 'xfsprogs' has no installation candidate
E: Package 'python-oauthlib' has no installation candidate
Unexpected error while running command.
Command: ['eatmydata', 'apt-get', '--quiet', '--assume-yes', '--option=Dpkg::options::=--force-unsafe-io', '--option=Dpkg::Options::=--force-confold', '--no-install-recommends', 'install', 'lvm2', 'mdadm', 'btrfs-tools', 'xfsprogs', 'bcache-tools', 'python-oauthlib']
Exit code: 100
Reason: -
Stdout: ''
Stderr: ''
2016-09-08 03:09:55,575 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [100]
2016-09-08 03:09:55,575 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2016-09-08 03:09:55,576 - util.py[WARNING]: Running scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
Cloud-init v. 0.7.5 finished at Thu, 08 Sep 2016 03:09:55 +0000. Datasource DataSourceMAAS [http://10.245.0.10:5240/MAAS/metadata/curtin]. Up 27.86 seconds

ubuntu@maas2-production:~$ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-==============================-============-=================================================
ii maas 2.0.0+bzr5189-0ubuntu1~16.04.1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS client and command-line interface
un maas-cluster-controller <none> <none> (no description available)
ii maas-common 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS server common files
ii maas-dhcp 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS DHCP server
ii maas-dns 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS DNS server
ii maas-proxy 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS Caching Proxy
ii maas-rack-controller 2.0.0+bzr5189-0ubuntu1~16.04.1 all Rack Controller for MAAS
ii maas-region-api 2.0.0+bzr5189-0ubuntu1~16.04.1 all Region controller API service for MAAS
ii maas-region-controller 2.0.0+bzr5189-0ubuntu1~16.04.1 all Region Controller for MAAS
un maas-region-controller-min <none> <none> (no description available)
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-maas-provisioningserver <none> <none> (no description available)
ii python3-django-maas 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS server provisioning libraries (Python 3)

[Andres Rodriguez (andreserl)]

Hi Larry,

To me this seems like broken archives:

Package lvm2 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package python-oauthlib is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'lvm2' has no installation candidate
E: Package 'xfsprogs' has no installation candidate
E: Package 'python-oauthlib' has no installation candidate

Or maybe curtin/cloud-init trying to isntall something they shouldn't ? If you are saing that you are deploying Trusty and you see the issues above, then this is not curtin/cloud-init provided that it is python2 and those packages should be available in the archive.

Also, have you seen at maas-proxy ? Is it running ? You can try restarting maas-proxy or even cleaning up its cache and restarting and see if that makes a difference ?

[Andres Rodriguez (andreserl)]

Ok. I confirm this is invalid in MAAS as I have stopped using maas-proxy to test this against and I see this issue in Yakkety:

Reading package lists... Error!
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/yakkety/InRelease rename failed, Stale file handle (/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_yakkety_InRelease -> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_yakkety_InRelease).
E: Failed to fetch store:/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_yakkety_main_binary-amd64_Packages.xz rename failed, Stale file handle (/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_yakkety_main_binary-amd64_Packages -> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_yakkety_main_binary-amd64_Packages).
E: Failed to fetch store:/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_yakkety_main_i18n_Translation-en.xz rename failed, Stale file handle (/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_yakkety_main_i18n_Translation-en -> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_yakkety_main_i18n_Translation-en).
E: Failed to fetch store:/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_yakkety_restricted_binary-amd64_Packages.xz rename failed, Stale file handle (/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_yakkety_restricted_binary-amd64_Packages -> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_yakkety_restricted_binary-amd64_Packages).
E: Failed to fetch store:/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_yakkety_restricted_i18n_Translation-en.xz rename failed, Stale file handle (/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_yakkety_restricted_i18n_Translation-en -> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_yakkety_restricted_i18n_Translation-en).
E: Some index files failed to download. They have been ignored, or old ones used instead.
E: Problem renaming the file /var/cache/apt/srcpkgcache.bin.zwSNvh to /var/cache/apt/srcpkgcache.bin - rename (116: Stale file handle)
E: Problem renaming the file /var/cache/apt/pkgcache.bin.dbbXqV to /var/cache/apt/pkgcache.bin - rename (116: Stale file handle)
W: You may want to run apt-get update to correct these problems
E: The package cache file is corrupted

So that seems to be a bug elsewhere

[Andres Rodriguez (andreserl)]

andres@node01:/etc/apt/apt.conf.d$ sudo apt-get install ntp
Reading package lists... Error!
E: Problem renaming the file /var/cache/apt/srcpkgcache.bin.ejqP36 to /var/cache/apt/srcpkgcache.bin - rename (116: Stale file handle)
E: Problem renaming the file /var/cache/apt/pkgcache.bin.jCjmSe to /var/cache/apt/pkgcache.bin - rename (116: Stale file handle)
W: You may want to run apt-get update to correct these problems
E: The package cache file is corrupted

[Andres Rodriguez (andreserl)]

Hi Larry,

So the state file handle issue you are mentioning, I believe is the same as https://bugs.launchpad.net/cloud-init/+bug/1618572. Which is a kernel issue. That should, however, apply to both Xenial and Yakkety. This may (or may not) apply to trusty too.

AS far as trusty, it may indeed be related to the same bug, but the weird this is this:

E: Package 'lvm2' has no installation candidate
E: Package 'xfsprogs' has no installation candidate
E: Package 'python-oauthlib' has no installation candidate

Which means that it is trying to install something that's not there.

[Andres Rodriguez (andreserl)]

Also, are you trying to deploy this machine with Trusty + hwe-x ?

[Larry Michel (lmic)]

Andres, yes we are deploying Trusty with hwe-x, but what's strange is that we can deploy Xenial now on these nodes. As far as lvm2, xfsprogs and python-oauthlib, wouldn't be related since apt-get update is failing? Also, would the commissioning failures be related as well?

[Larry Michel (lmic)]

Deploying with Trusty and hwe-t also works.. I've verified that it's only happening with trusty and hwe-x.