preflight check for librarian access on appservers

[Robert Collins]

Working on RT 41379, Tom reckons that we're good to go but we're changing a system that is rather static and heavyweight to change. I'd like to have a small script we can run both now, and when bringing up new machines, to check that they have the needed access.

It needs to establish that the librarian's restricted and public upload ports work, and the restricted private download port works (because we use that from appservers). It needs to let us check that things were uploaded appropriately from a machine that can access the public librarian (because we don't necessarily let that work from all sppservers directly).

So, I think it can be very simple:
- do a private-upload
- download from private-download
- do a public-upload
- report a url which can be used to check the public uploaded content is accessible (e.g. sysadmins can throw it at wget locally)

The last step is belts and braces, the first three are the crucial steps.

[Tom Haddon (mthaddon)]

We also want to be able to specify the URL and port to use for each one, so that if/when we're switching those, we can do so without needing to rewrite the script.

[Robert Collins (lifeless)]

On Mon, Nov 22, 2010 at 11:46 PM, Tom Haddon <email address hidden> wrote:
> We also want to be able to specify the URL and port to use for each one,
> so that if/when we're switching those, we can do so without needing to
> rewrite the script..

I don't think we need that - all we need is for it to source it from
the config : then when we're reconfiguring we can:
 - update the config on a temporary lp dir
 - test
 - trigger a deploy

If the script returns non-zero in the case of error, then we could
make running this be part of the production deploy cycle, and that
would let us just:
 - change the configs
 - trigger a deploy
 - any machines that fail will stop, and we can fix.

Of course, we'd want our high availalability story and spare capacity
to be a bit more robust first ;)

[Gary Poster (gary)]

Do I understand correctly that this is blocking?

Is this a Foundations bug because you are asking the Foundations team to schedule it, or because the Foundations bucket is the generic LP bucket that we have?

[Robert Collins (lifeless)]

On Tue, Nov 23, 2010 at 1:31 PM, Gary Poster <email address hidden> wrote:
> Do I understand correctly that this is blocking?

Its blocking moving to a HA librarian configuration, yes.

> Is this a Foundations bug because you are asking the Foundations team to
> schedule it, or because the Foundations bucket is the generic LP bucket
> that we have?

I will try and get to it before my leave starts but cannot guarantee
that. If I don't, I would really appreciate it if someone in
Foundations could take it on. It should be pretty shallow.

-Rob

[Gary Poster (gary)]

> I will try and get to it before my leave starts but cannot guarantee
> that. If I don't, I would really appreciate it if someone in
> Foundations could take it on. It should be pretty shallow.

Ack. Not sure when your leave starts; will check with Francis.

[Tom Haddon (mthaddon)]

Please update RT#41379 when this is complete - I'll mark that RT as incomplete in the meantime.

Thanks, Tom

[Benji York (benji)]

The script (utilities/smoke-test-librarian.py) in the related branch (lp:~benji/launchpad/bug-678375) should fit the bill. It's been reviewed already but I'd appreciate anyone else interested in this bug taking a look at it.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r12032 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/12032>.

[Tom Haddon (mthaddon)]

I've just run this on one of the appservers, and got the following:

launchpad@soybean:/srv/launchpad.net/production/launchpad$ ./utilities/smoke-test-librarian.py
adding a private file to the librarian...
retrieving private file from http://lplibrarian.internal/61153941/smoke-test-file
adding a public file to the librarian...
retrieving public file from http://launchpadlibrarian.net/61153942/smoke-test-file

So this bug report ties into RT#41379 in which we're trying to put into place load balanced frontends for uploads and private downloads using the following hostnames on port 80 in each case:

lplibrarian-private-download.internal
lplibrarian-private-upload.internal
lplibrarian-public-upload.internal

How can I use this script to test those? Would I need to update the lp-production-configs branch, or is there some way to ask it run using specific hostnames?

[Robert Collins (lifeless)]

Prepare and stage an lp tree with the candidate config; run the
script. - It pulls everything from the config system.

[Tom Haddon (mthaddon)]

Would my config diff be https://pastebin.canonical.com/41362/ ? I'd like to remove references to mizuho.c.c altogether, but not sure why there's a "restricted_download_url" and "restricted_download_host", for instance, or why we need to specify the upload_host and download_host. I'm also not sure where to specify the public upload URL (http://lplibrarian-public-upload.internal) - would that be upload_host?

[Tom Haddon (mthaddon)]

After IRC discussions with Rob, we've decided to ensure the three hosts are listening on different ports (for public and private uploads and private downloads). So I think the config I want now is https://pastebin.canonical.com/41364/, but what I'm still not sure about is where to specify the port for the public upload.

[Robert Collins (lifeless)]

Oh, thats because the prod configs don't set everything.

you can always look in configs/schema-lazr.conf to see the avialable options.

# Port number Librarian listens for storage requests on.
# datatype: int
upload_port: 9090