producing Apache 500 error in KARL

[Jim B. Glenn]

We received 2 support requests to <email address hidden>

RT 82663: URGENT: Access to intranet lost (Nov 11, 2010 @ 7:14:36AM)

RT 82661: 500 Internal server error (Nov 11,2010 @ 4:20:15 AM)

Typically these types of Apache 500 errors mean mod_wsgi has had an issue and is falling over. These often mean ALL the sites are down.

Hancock Telecom called Andrew (on call pirate) in the 8AM hour.

We should have received call at 4 AM hour too?

Zenoss didn't produce any errors.

8:00 AM Jim saw RT tickets and verified sites were all up and responded to RT tickets saying the site had a problem but should now be resolved. We would be investigation to prevent in the future.

(Jim went to Super 6)

9:59 AM Jim e-mailed <email address hidden> notifying all customers of the short downtime and we would be investigation.

Firoze (fahamu) replied back to list and said he was still having issues.

Updated preferences for mailing list for karl-news to require moderation on everyone, but Jim.

10:03 Jim e-mailed OSI Devs (OSI-Dev OSI-Dev <email address hidden>) of the issue and asked for ideas in troubleshooting.

[Thu Nov 11 10:02:02 2010] [error] [client 217.36.220.22] mod_wsgi (pid=15816): Exception occurred processing WSGI script '/opt/karl/fahamu/current/etc/wsgi-scripts/karl3.wsgi'.
[Thu Nov 11 10:02:02 2010] [error] [client 217.36.220.22] TypeError: expected string object for header value

10:20 AM: Ben (OXFAM) e-mailed RT:

I can now get into Karl, but it won't allow me to save my log in. If I
click 'remember' the 500 error comes back.

Tres suspects an issue with "last login time" and the login page.

10:44 AM: Jim e-mails Firoze to try unchecking "last login time", same problem.

Jim forwards info to developers.

[Tres Seaver (tseaver)]

Something inside webob.HTTPFound is morphing the headers,
created by repoze.who's authtkt plugin, into a form which causes
Apache / mod_wsgi to barf. The particular cookies which trigger
the problem are those created with 'max_age' in the credentials.

The older sites worked because the who middleware added the
headers using raw WSGI (no Paste or Webob invovled).

[Tres Seaver (tseaver)]

I just checked in a change on the karl trunk which fixes this bug.

[JimPGlenn (jpglenn09)]

released