Internal Server Errors Loading Templates

Bug #1306675 reported by Jason Stephenson on 2014-04-11
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Evergreen
Undecided
Unassigned
2.5
Undecided
Unassigned
2.6
Undecided
Unassigned

Bug Description

Evergreen: Master as of 20131208 + some bug fixes may also affect current master
OpenSRF: master as of 20131208, but likely irrelevant
Pg 9.1.something but also likely irrelevant

We get a number of these in the Apache error logs:

[Fri Apr 11 10:38:22 2014] [warn] [client REDACTED] egweb: template error: undef error - Can't call method "maketext" on an undefined value at /usr/local/share/perl/5.14.2/OpenILS/WWW/EGWeb.pm line 94.\n, referer: https://haverhill.mvlc.org/eg/opac/myopac/main
[Fri Apr 11 10:39:41 2014] [warn] [client REDACTED] egweb: template error: undef error - Can't call method "maketext" on an undefined value at /usr/local/share/perl/5.14.2/OpenILS/WWW/EGWeb.pm line 94.\n
[Fri Apr 11 10:43:07 2014] [warn] [client REDACTED] egweb: template error: undef error - Can't call method "maketext" on an undefined value at /usr/local/share/perl/5.14.2/OpenILS/WWW/EGWeb.pm line 94.\n
[Fri Apr 11 10:44:21 2014] [warn] [client REDACTED] egweb: template error: undef error - Can't call method "maketext" on an undefined value at /usr/local/share/perl/5.14.2/OpenILS/WWW/EGWeb.pm line 94.\n
[Fri Apr 11 10:44:37 2014] [warn] [client REDACTED] egweb: template error: undef error - Can't call method "maketext" on an undefined value at /usr/local/share/perl/5.14.2/OpenILS/WWW/EGWeb.pm line 94.\n, referer: https://newbury.mvlc.org/eg/opac/myopac/main

After the warning, the server returns an internal server error, see EGWEb.pm lines 77 & 78.

It appears to be happening to staff and to patrons, seemingly at random. Reloading the page in the browser or staff client makes the error go away and the page loads.

It may be a local thing, but we have no customizations that should affect this part of the code. I also suspect it could be a load/performance issue on the server, hence the difficulty in reproducing it.

I opened this bug mainly to see if anyone else can corroborate a similar experience.

Jason Stephenson (jstephenson) wrote :

Some more information.

We are on Ubuntu 12.04. We update the OS packages on the server this week (for obvious reasons if you'be been following tech news lately). This problem started the day after the update in earnest. We have 1 entry with this message from last week. The entries have been dominating our logs since, 316/317 errors on one day and 273/273 errors so far today, (Friday).

Ben Shum (bshum) wrote :

I haven't been able to find any errors like the one described in our master systems. Not entirely sure what this means yet.

Marking as incomplete pending further details.

Changed in evergreen:
status: New → Incomplete
Jason Stephenson (jstephenson) wrote :

As a followup, it does not appear to be this bug: https://bugs.launchpad.net/evergreen/+bug/1104369

We have the 0723 upgrade script loaded on 2012-07-29, four days after tsbere asked about this very problem in IRC.

The frustrating thing about this, is I find numerous other web packages that have a similar issue, and usually there are no reported fixes. In one or two cases, the solution was to reinstall the web package. Unfortunately, we have done that a number of times with Evergreen and this problem still persists for us.

Jason Etheridge (phasefx) wrote :

FWIW, I've seen this on one system but could not figure out why it was happening. Bill wrote some defensive programming for it, and I was meaning to launchpad it:

diff --git a/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGWeb.pm b/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGWeb.pm
index 9536aae..d111cd2 100644
--- a/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGWeb.pm
+++ b/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGWeb.pm
@@ -90,7 +90,8 @@ sub set_text_handler {
     $r->log->debug("egweb: messages locale = $locale");

     return sub {
- my $lh = OpenILS::WWW::EGWeb::I18N->get_handle($locale);
+ my $lh = OpenILS::WWW::EGWeb::I18N->get_handle($locale)
+ || OpenILS::WWW::EGWeb::I18N->new;
         return $lh->maketext(@_);
     };
 }

Jason Stephenson (jstephenson) wrote :

I set this to confirmed because it has been seen in one other Evergreen instance.

While I'd really like to find the ultimate cause*, a workaround will do.

We've loaded the above patch in our production system this morning. I tried it in development yesterday just to make sure it didn't blow up, which I didn't think it would.

Jason, if you or Bill could throw that in a branch, I'd be happy to sign off in a week or so if the errors don't come back. Alternatively, I could throw it in a branch and set the commit's author to Bill's name and email address.

* I suspect it is a failure of a module to completely initialize in some cases owing to a slightly hosed package somewhere. (Probably an upgrade hang over that debsums doesn't catch.) I've seen this reported with other projects, and the only effective fix seems to be to reinstall most everything.

Changed in evergreen:
status: Incomplete → Confirmed
Bill Erickson (berick) wrote :

Here you go, Jason. This was copied back over by hand, because I couldn't find my original branch (if there was one), so testing of this also appreciated. (I'm pretty sure it's the exact same).

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/berick/lp1306675-tpac-maketext-failure

FWIW, I see this from time to time on my Wheezy dev server.

Jason Stephenson (jstephenson) wrote :

We have been using the above change in our production server for almost a week now with no negative repercussions. The error messages are definitely gone.

We never could pin the problem down to a single PID. Just before applying the above change we added an extra log definition to log the PIDs of the Apache children, and these errors actually seemed to come from different PIDs at relatively the same time. It seems we'd bet spurts of it within a few minutes of each other, and then silence for a while before it would happen again.

Anyway, since we've been using this in production, I'd like to push it to master. However, we've not added the pullrequest tag. Also, I wonder if this should be targeted at 2.6.1 or other milestones as well? We installed master from a branch created on 4/16, so we're basically running 2.6.0 with a couple of bug fix patches thrown in. It seems to me this ought to target 2.6 and 2.6.1.

Bill Erickson (berick) wrote :

Seeing this more in the wild. Added pullrequest and 2.5+ targets.

tags: added: opac pullrequest
Changed in evergreen:
milestone: none → 2.7.0-beta2
Download full text (3.4 KiB)

I have noticed the same error on Sitka's servers. The last time I looked
at this was over a year ago, but if I recall, at the time, I traced it to a
problem with TT2's interaction with Apache. Unfortunately, I did not have
the time to probe further, but I agree with your assessment that it could
be a load/performance issue because it seems random in our case as well.

On Wed, Aug 20, 2014 at 8:18 AM, Bill Erickson <email address hidden> wrote:

> Seeing this more in the wild. Added pullrequest and 2.5+ targets.
>
> ** Tags added: opac pullrequest
>
> ** Changed in: evergreen
> Milestone: None => 2.7.0-beta2
>
> ** Also affects: evergreen/2.5
> Importance: Undecided
> Status: New
>
> ** Also affects: evergreen/2.6
> Importance: Undecided
> Status: New
>
> ** Changed in: evergreen/2.5
> Milestone: None => 2.5.7
>
> ** Changed in: evergreen/2.6
> Milestone: None => 2.6.3
>
> --
> You received this bug notification because you are subscribed to
> Evergreen.
> Matching subscriptions: EG-Bug-Mail
> https://bugs.launchpad.net/bugs/1306675
>
> Title:
> Internal Server Errors Loading Templates
>
> Status in Evergreen - Open ILS:
> Confirmed
> Status in Evergreen 2.5 series:
> New
> Status in Evergreen 2.6 series:
> New
>
> Bug description:
> Evergreen: Master as of 20131208 + some bug fixes may also affect
> current master
> OpenSRF: master as of 20131208, but likely irrelevant
> Pg 9.1.something but also likely irrelevant
>
> We get a number of these in the Apache error logs:
>
> [Fri Apr 11 10:38:22 2014] [warn] [client REDACTED] egweb: template
> error: undef error - Can't call method "maketext" on an undefined value at
> /usr/local/share/perl/5.14.2/OpenILS/WWW/EGWeb.pm line 94.\n, referer:
> https://haverhill.mvlc.org/eg/opac/myopac/main
> [Fri Apr 11 10:39:41 2014] [warn] [client REDACTED] egweb: template
> error: undef error - Can't call method "maketext" on an undefined value at
> /usr/local/share/perl/5.14.2/OpenILS/WWW/EGWeb.pm line 94.\n
> [Fri Apr 11 10:43:07 2014] [warn] [client REDACTED] egweb: template
> error: undef error - Can't call method "maketext" on an undefined value at
> /usr/local/share/perl/5.14.2/OpenILS/WWW/EGWeb.pm line 94.\n
> [Fri Apr 11 10:44:21 2014] [warn] [client REDACTED] egweb: template
> error: undef error - Can't call method "maketext" on an undefined value at
> /usr/local/share/perl/5.14.2/OpenILS/WWW/EGWeb.pm line 94.\n
> [Fri Apr 11 10:44:37 2014] [warn] [client REDACTED] egweb: template
> error: undef error - Can't call method "maketext" on an undefined value at
> /usr/local/share/perl/5.14.2/OpenILS/WWW/EGWeb.pm line 94.\n, referer:
> https://newbury.mvlc.org/eg/opac/myopac/main
>
> After the warning, the server returns an internal server error, see
> EGWEb.pm lines 77 & 78.
>
> It appears to be happening to staff and to patrons, seemingly at
> random. Reloading the page in the browser or staff client makes the
> error go away and the page loads.
>
> It may be a local thing, but we have no customizations that should
> affect this part of the code. I also suspect it could be a
> load/performance issue on the server, hence the difficul...

Read more...

Ben Shum (bshum) wrote :

Pushed this to master, rel_2_6 and rel_2_5

Changed in evergreen:
status: Confirmed → Fix Committed
Changed in evergreen:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers