| 2010-09-16 00:09:36 |
Robert Collins |
bug |
|
|
added bug |
| 2010-09-16 00:48:44 |
James Westby |
bug |
|
|
added subscriber James Westby |
| 2010-09-21 19:28:43 |
Robert Collins |
description |
See bug 380504 for historical context.
In short:
- when we start to upgrade an appserver it may have requests live
- haproxy will keep sending requests until the appserver falls over and dies
We need to change this to:
- start the upgrade by telling the appserver to go out of rotation
- that should close the listening port
- we then check that that has happened (connect should error)
- and then we bring up the new appserver
- if the old appserver hasn't shut down after 200seconds (because of librarian proxied files)
- kill it
ha proxy will stop forwarding requests cleanly once the listening socket is gone AIUI. |
See bug 380504 for historical context.
In short:
- when we start to upgrade an appserver it may have requests live
- haproxy will keep sending requests until the appserver falls over and dies
We need to change this to:
- take the server out of rotation (sysadmin change, rt to be filed)
- start the upgrade by telling the appserver to gracefully stop
- that should close the listening port
- and report that it has done so
- at this point the new instance can be brought up (sysadmin change, future rt)
- and then we bring up the new appserver
- if the old appserver hasn't shut down after 200seconds (because of librarian proxied files)
- kill it (sysadmin change, first rt)
ha proxy may stop forwarding requests cleanly once the listening socket is gone, but we want to be completely robust against bugs there. |
|
| 2010-09-21 19:52:59 |
Robert Collins |
description |
See bug 380504 for historical context.
In short:
- when we start to upgrade an appserver it may have requests live
- haproxy will keep sending requests until the appserver falls over and dies
We need to change this to:
- take the server out of rotation (sysadmin change, rt to be filed)
- start the upgrade by telling the appserver to gracefully stop
- that should close the listening port
- and report that it has done so
- at this point the new instance can be brought up (sysadmin change, future rt)
- and then we bring up the new appserver
- if the old appserver hasn't shut down after 200seconds (because of librarian proxied files)
- kill it (sysadmin change, first rt)
ha proxy may stop forwarding requests cleanly once the listening socket is gone, but we want to be completely robust against bugs there. |
See bug 380504 for historical context.
In short:
- when we start to upgrade an appserver it may have requests live
- haproxy will keep sending requests until the appserver falls over and dies
We need to change the appservers to have a graceful shutdown:
- close the socket
- report that its closed
- let requests complete
- after a configured interval (200 seconds probably) forcibly take down remaining threads
This will let the overall process become:
- take the server out of rotation (sysadmin change, rt to be filed)
- start the upgrade by telling the appserver to gracefully stop
- wait for the 'socket xxxx closed' message
- start the new instance (sysadmin change, future rt)
- if the old appserver hasn't shut down after (long time) kill -9 it (belt and braces)
ha proxy may stop forwarding requests cleanly once the listening socket is gone, but we want to be completely robust against bugs there.
|
|
| 2010-10-05 19:56:34 |
Robert Collins |
description |
See bug 380504 for historical context.
In short:
- when we start to upgrade an appserver it may have requests live
- haproxy will keep sending requests until the appserver falls over and dies
We need to change the appservers to have a graceful shutdown:
- close the socket
- report that its closed
- let requests complete
- after a configured interval (200 seconds probably) forcibly take down remaining threads
This will let the overall process become:
- take the server out of rotation (sysadmin change, rt to be filed)
- start the upgrade by telling the appserver to gracefully stop
- wait for the 'socket xxxx closed' message
- start the new instance (sysadmin change, future rt)
- if the old appserver hasn't shut down after (long time) kill -9 it (belt and braces)
ha proxy may stop forwarding requests cleanly once the listening socket is gone, but we want to be completely robust against bugs there.
|
See bug 380504 for historical context.
In short:
- when we start to upgrade an appserver it may have requests live
- haproxy will keep sending requests until the appserver falls over and dies
- RT 41503 will change this to workaround in a kludgy fashion.
We need to change the appservers to have a graceful shutdown:
- close the socket
- report that its closed
- let requests complete
- after a configured interval (200 seconds probably) forcibly take down remaining threads
This will let the overall process become:
- take the server out of rotation (sysadmin change, rt 41503)
- start the upgrade by telling the appserver to gracefully stop
- wait for the 'socket xxxx closed' message
- start the new instance (sysadmin change, future rt)
- if the old appserver hasn't shut down after (long time) kill -9 it (belt and braces)
ha proxy may stop forwarding requests cleanly once the listening socket is gone, but we want to be completely robust against bugs there.
|
|
| 2010-10-06 23:29:02 |
Robert Collins |
tags |
|
rfwtad |
|
