apt-cacher files have incorrect owners

[Kyle L. Henderson]

The repo-cacher.yml that was added to Newton sets up the apt-cacher-ng service to cache apt packages. It also setups up haproxy to provide a master and backup services on the repo hosts. These backup repo hosts are not completely functional however because the cache directory structure and packages don't have the correct authorities for apt-cacher-ng to operate on them.

Here is a sample of the master files:

 ls -lah /var/www/repo/pkg-cache/
total 68K
drwxrwsr-x 8 apt-cacher-ng www-data 4.0K Dec 12 07:58 .
drwxr-sr-x 8 nginx www-data 4.0K Dec 11 08:38 ..
drwxr-sr-x 2 apt-cacher-ng www-data 4.0K Dec 12 07:58 changelogs.ubuntu.com
-rw-r--r-- 1 apt-cacher-ng www-data 0 Dec 12 06:25 _expending_damaged
-rw-r--r-- 1 apt-cacher-ng www-data 33K Dec 12 06:25 _expending_dat
drwxr-sr-x 3 apt-cacher-ng www-data 4.0K Dec 11 09:08 mirror.rackspace.com
drwxr-sr-x 3 apt-cacher-ng www-data 4.0K Dec 11 10:08 ports.ubuntu.com
drwxr-sr-x 3 apt-cacher-ng www-data 4.0K Dec 11 09:05 security.ubuntu.com
drwxr-sr-x 3 apt-cacher-ng www-data 4.0K Dec 11 09:41 ubuntu-cloud.archive.canonical.com
drwxr-sr-x 4 apt-cacher-ng www-data 4.0K Dec 11 09:05 uburep

Here is a sample of the files on a backup repo host:

~# ls -lah /var/www/repo/pkg-cache/
total 72K
drwxrwsr-x 8 apt-cacher-ng www-data 4.0K Dec 12 07:58 .
drwxr-sr-x 8 nginx www-data 4.0K Dec 11 08:38 ..
drwxr-sr-x 2 nginx www-data 4.0K Dec 12 07:58 changelogs.ubuntu.com
-rw-r--r-- 1 apt-cacher-ng www-data 11 Dec 12 06:25 _exfail_cnt
-rw-r--r-- 1 nginx www-data 0 Dec 12 06:25 _expending_damaged
-rw-r--r-- 1 nginx www-data 33K Dec 12 06:25 _expending_dat
drwxr-sr-x 3 nginx www-data 4.0K Dec 11 09:08 mirror.rackspace.com
drwxr-sr-x 3 nginx www-data 4.0K Dec 11 10:08 ports.ubuntu.com
drwxr-sr-x 3 nginx www-data 4.0K Dec 11 09:05 security.ubuntu.com
drwxr-sr-x 3 nginx www-data 4.0K Dec 11 09:41 ubuntu-cloud.archive.canonical.com
drwxr-sr-x 4 nginx www-data 4.0K Dec 11 09:05 uburep

The difference in the owner is caused by using lsyncd to sync the files from the master to the backups. The backups can serve existing packages but any attempt install an non-cached package will fail with a 503 error.

For instance, if you take the master down and then go into one of the containers and try to load a new package (pick a small one without many new dependencies like atop or tmux) you'll see something like:

~# apt-get install atop
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  atop
0 upgraded, 1 newly installed, 0 to remove and 8 not upgraded.
Need to get 90.0 kB of archives.
After this operation, 237 kB of additional disk space will be used.
Err:1 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 atop amd64 1.26-2build1
  503 Cache storage error - No such file or directory
E: Failed to fetch http://us.archive.ubuntu.com/ubuntu/pool/universe/a/atop/atop_1.26-2build1_amd64.deb 503 Cache storage error - No such file or directory

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Fixing the permissions be the same as the master allows apt-cacher-ng to be fully functional on the backups but then screws up lsyncd because it can no longer maintain the directories/packages. For instance, when the master comes back up it'll try to delete the newly cached packages on the backup and be unable to.

[Jean-Philippe Evrard (jean-philippe-evrard)]

Worth of interest:
- https://github.com/openstack/openstack-ansible-repo_server/blob/master/tasks/repo_cacher.yml#L47
- https://github.com/openstack/openstack-ansible-repo_server/blob/master/tasks/repo_sync_manager.yml

I'd be enclined to have this consistent across all nodes.

[Jean-Philippe Evrard (jean-philippe-evrard)]

Multiple ways to solve that:
- Making sure apt-cacher can write to folders (being part of www-data group?)
- Making sure we synchronize correctly between the nodes (same permissions everywhere: do we have them?)

I'd be happy to see someone to confirm this.

[Jean-Philippe Evrard (jean-philippe-evrard)]

someone confirming*

[Jean-Philippe Evrard (jean-philippe-evrard)]

forgot to write in the other way to solve it is avoid syncing of apt-cacher folders.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/410916

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/410916
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-repo_server/commit/?id=1169edc47be92943690983a6d10857761038b473
Submitter: Jenkins
Branch: master

commit 1169edc47be92943690983a6d10857761038b473
Author: Kyle L. Henderson <email address hidden>
Date: Wed Dec 14 12:35:28 2016 -0600

    Fix apt-cacher-ng file owners during rsync

    The lsyncd service runs as the 'nginx' user such that files sync'd
    from the master node to the backups will have 'nginx' as the owner.
    However, the apt-cacher-ng service needs to be the owner to function
    properly. This fix consolidates the pre and post sync tasks into
    a script that can be called by lsyncd. The script can then change
    the file owners as needed before and after the rsync. The owners
    need to be 'nginx' before the rsync so that lsyncd can update
    files and 'apt-cacher-ng' after the sync so the cacher service works.

    Additionally, setup lsyncd to sync each service's directory separately
    rather than being rsync'd all together. This avoids lsyncd bouncing
    services when their respective files are not being sync'd.

    Change-Id: Ifaba17b89035398917f2b3257574e18eb9027c08
    Closes-bug: #1649339

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/420773

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/420773
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-repo_server/commit/?id=248b2d6c94b0c9c963c0226c8d6cb6c897a52520
Submitter: Jenkins
Branch: stable/newton

commit 248b2d6c94b0c9c963c0226c8d6cb6c897a52520
Author: Cameron Loader <email address hidden>
Date: Wed Jan 18 16:07:03 2017 -0600

    Fix apt-cacher-ng file owners during rsync

    The lsyncd service runs as the 'nginx' user such that files sync'd
    from the master node to the backups will have 'nginx' as the owner.
    However, the apt-cacher-ng service needs to be the owner to function
    properly. This fix consolidates the pre and post sync tasks into
    a script that can be called by lsyncd. The script can then change
    the file owners as needed before and after the rsync. The owners
    need to be 'nginx' before the rsync so that lsyncd can update
    files and 'apt-cacher-ng' after the sync so the cacher service works.

    Additionally, setup lsyncd to sync each service's directory separately
    rather than being rsync'd all together. This avoids lsyncd bouncing
    services when their respective files are not being sync'd.

    Backport modified to work with Upstart.

    Change-Id: Ifaba17b89035398917f2b3257574e18eb9027c08
    Closes-bug: #1649339
    (cherry picked from commit 1169edc47be92943690983a6d10857761038b473)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-repo_server 15.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-repo_server 14.0.8 release.