Apache configuration is necessary for tests but makes it impossible to use "make clean" safely in lxc

[Gary Poster]

We added make clean to our parallel testing setup, and discovered that the build would fail because the /var/tmp/bazaar.launchpad.dev directory was owned by root and not chmod-ed for full access. Brad and Graham traced the path to the following. First, "make install" creates the directory as root but with privileges for the user (buildbot, in this case) to do what it needs. It also installs a branch rewriter into Apache that creates the directory as root when Apache starts and the directory is missing. Second, when our build script starts, we run a make clean inside a newly-started lxc container. Third, inside that container, after our make clean has run, the branch rewriter is ready now that the container has had a bit of time to initialize, and it notices that the necessary file is missing. The script creates the directory as root, but without granting chmod privileges to everyone. Then, tests fail because the code is missing subdirectories within /var/tmp/bazaar.launchpad.dev that the tests need, and the directories cannot be created because the buildbot user does not have necessary privileges. Finally, the next make clean also fails because it tries to delete /var/tmp/bazaar.launchpad.dev and the buildbot user does not have the privileges to do so.

To address this problem, we identified three possible solutions so far.

1) Have the tests use make inplace instead of make install, so that the Apache configuration is not installed. Unfortunately, the Apache configuration is necessary for an unknown number of tests to pass (at least some of them dealing with code hosting), so this solution would also need to fix the tests that require Apache configuration.

2) Have the branch rewriter create the directory granting the necessary permissions to everyone, following the lead of make install. We don't know how this would affect production and are afraid it might cause a security issue, so that would need to be investigated.

3) Create a new make target for cleaning that does not try to delete /var/tmp/bazaar/launchpad.dev, and have our parallel testing use it instead of the normal make clean. This is arguably a bit of a hack because it doesn't seem like a target like this should be necessary. However, it is very easy to do.

We are pursuing option 3 for now.

[Gary Poster (gary)]

Robert requests that this bug not be closed until we address solution 1, or until we create a minimum high-level bug with data on how badly tests fail without Apache config installed. He is asking for this because we are currently running branch rewrite code from earlier branches of Launchpad when we run our tests, and this is bad.

[Brad Crittenden (bac)]

For tracking of option 1, see bug 1004514.