Setup new karlstaging with multiple appservers

[Paul Everitt]

Most of the work has been done on this. To wrapup:

1) Make sure the load balancer is actually distributing requests.

2) Shut down an app server, see if the LB stops sending it requests.

3) Switch karlstaging to the ux2 branch.

4) Ensure we have the email whitelist in effect.

5) Check the HTTP expires on the static resources, primarily the stuff from Bottlecap. Make sure they aren't loaded on every request.

6) Make sure we aren't running in debug mode.

7) Put a comment on this ticket with any kind of after-action report. Stuff you saw that looked fishy or could be improved, etc.

[Shane Hathaway (shane-hathawaymix)]

Thanks for the list! At the moment, it looks like there's some work required on ux2 to fix static resource caching. Also, I'm pulling production data into the staging database tonight, so staging is down at the moment, but it should be back up in the morning.

[Paul Everitt (paul-agendaless)]

If we get static asset caching on all the KARL stuff but not the Bottlecap stuff, that's enough for now.

--Paul

On Mar 15, 2012, at 4:24 AM, Shane Hathaway wrote:

> Thanks for the list! At the moment, it looks like there's some work
> required on ux2 to fix static resource caching. Also, I'm pulling
> production data into the staging database tonight, so staging is down at
> the moment, but it should be back up in the morning.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/955075
>
> Title:
> Setup new karlstaging with multiple appservers
>
> Status in KARL3:
> New
>
> Bug description:
> Most of the work has been done on this. To wrapup:
>
> 1) Make sure the load balancer is actually distributing requests.
>
> 2) Shut down an app server, see if the LB stops sending it requests.
>
> 3) Switch karlstaging to the ux2 branch.
>
> 4) Ensure we have the email whitelist in effect.
>
> 5) Check the HTTP expires on the static resources, primarily the stuff
> from Bottlecap. Make sure they aren't loaded on every request.
>
> 6) Make sure we aren't running in debug mode.
>
> 7) Put a comment on this ticket with any kind of after-action report.
> Stuff you saw that looked fishy or could be improved, etc.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/955075/+subscriptions

[Paul Everitt (paul-agendaless)]

It's still "Down for Maintenance".

I'll make a different ticket for the static caching on the bottlecap: resources.

--Paul

On Mar 15, 2012, at 4:24 AM, Shane Hathaway wrote:

> Thanks for the list! At the moment, it looks like there's some work
> required on ux2 to fix static resource caching. Also, I'm pulling
> production data into the staging database tonight, so staging is down at
> the moment, but it should be back up in the morning.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/955075
>
> Title:
> Setup new karlstaging with multiple appservers
>
> Status in KARL3:
> New
>
> Bug description:
> Most of the work has been done on this. To wrapup:
>
> 1) Make sure the load balancer is actually distributing requests.
>
> 2) Shut down an app server, see if the LB stops sending it requests.
>
> 3) Switch karlstaging to the ux2 branch.
>
> 4) Ensure we have the email whitelist in effect.
>
> 5) Check the HTTP expires on the static resources, primarily the stuff
> from Bottlecap. Make sure they aren't loaded on every request.
>
> 6) Make sure we aren't running in debug mode.
>
> 7) Put a comment on this ticket with any kind of after-action report.
> Stuff you saw that looked fishy or could be improved, etc.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/955075/+subscriptions

[Paul Everitt (paul-agendaless)]

Shane, what remains on closing this out?

[Shane Hathaway (shane-hathawaymix)]

#3 is done. #5 is broken on the ux2 branch, an issue that probably needs a new ticket (if there isn't one already) and some coordination within the team.

The rest are simple manual tests I need to perform now that the new servers are fully up. I plan to do those tests tonight.

BTW, the ux2 link now works on karlstaging. :-)

[Paul Everitt (paul-agendaless)]

#5 is in lp:955886

Once you do the manual tests, we'll consider this one wrapped up.

[Shane Hathaway (shane-hathawaymix)]

1) Check. Successive loads of zodbinfo.html return results from different app servers, so nginx is indeed working in round-robin fashion.

2) Check. I stopped each app server in turn and the site continued to function.

3) Confirmed earlier.

4) Check. I verified the white list works by spamming Chris and me until I realized all the email was arriving in my spam folder (doh!) and then taking myself off the white list and confirming email was not being sent to me anymore.

5) Deferred to lp:955886

6) All debug mode settings are off.

7) Does Gocept intend to run memcached on karlstaging10 and karlstaging11? That would speed up the servers a lot.

[Paul Everitt (paul-agendaless)]

Very good, thanks!

We'll check with gocept about memcached but after we make it through Monday.

--Paul

On Mar 17, 2012, at 6:58 PM, Shane Hathaway wrote:

> 1) Check. Successive loads of zodbinfo.html return results from
> different app servers, so nginx is indeed working in round-robin
> fashion.
>
> 2) Check. I stopped each app server in turn and the site continued to
> function.
>
> 3) Confirmed earlier.
>
> 4) Check. I verified the white list works by spamming Chris and me until
> I realized all the email was arriving in my spam folder (doh!) and then
> taking myself off the white list and confirming email was not being sent
> to me anymore.
>
> 5) Deferred to lp:955886
>
> 6) All debug mode settings are off.
>
> 7) Does Gocept intend to run memcached on karlstaging10 and
> karlstaging11? That would speed up the servers a lot.
>
>
> ** Changed in: karl3
> Status: New => Fix Committed
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/955075
>
> Title:
> Setup new karlstaging with multiple appservers
>
> Status in KARL3:
> Fix Committed
>
> Bug description:
> Most of the work has been done on this. To wrapup:
>
> 1) Make sure the load balancer is actually distributing requests.
>
> 2) Shut down an app server, see if the LB stops sending it requests.
>
> 3) Switch karlstaging to the ux2 branch.
>
> 4) Ensure we have the email whitelist in effect.
>
> 5) Check the HTTP expires on the static resources, primarily the stuff
> from Bottlecap. Make sure they aren't loaded on every request.
>
> 6) Make sure we aren't running in debug mode.
>
> 7) Put a comment on this ticket with any kind of after-action report.
> Stuff you saw that looked fishy or could be improved, etc.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/955075/+subscriptions