XML-RPC Gateway errors on Ubuntu 12.04

[Michael Peters]

Evergreen 2.3.4 and Master (c0cfe99)
OpenSRF 2.1.2
Ubuntu 12.04

In attempting to use the xml-rpc gateway on Evergreen 2.3.4 (collections, for example) clients receive an internal server error 500 which seems to tie back to the following in the logs:

Can't locate object method "args" via package "Error creating request object: RPC::XML::request::new: Invalid method name specified at document line 6, column 0 (byte 112)" (perhaps you forgot to load "Error creating request object: RPC::XML::request::new: Invalid method name specified at document line 6, column 0 (byte 112)"?) at /usr/local/share/perl/5.14.2/OpenILS/WWW/XMLRPCGateway.pm line 69.\n

This can be tested with a nice Python script from Lebbeous (thanks for this!) as shown below:

$ python
>>> import xmlrpclib
>>> s = xmlrpclib.ServerProxy("https://localhost/xml-rpc/open-ils.actor")
>>> f = getattr(s, "open-ils.actor.org_tree.retrieve")
>>> f()

This script works against versions earlier than 2.3.4 but fails against a 2.3.4 server with the fixes from https://bugs.launchpad.net/evergreen/+bug/1050596 applied, along with a stock version of master (c0cfe99).

[Michael Peters (mrpeters)]

One last note, I'm running RPC::XML 1.56

[Jason Stephenson (jstephenson)]

It is working fine for me on Ubuntu 12.04 using the packaged librpc-xml-perl 0.76-1 that Ubuntu ships.

[Michael Peters (mrpeters)]

Jason -- which version of Evergreen?

I just installed the latest version of master available on 3/31/13 and the smoke test above still results in an internal server error on Ubuntu Precise.

The only relevant log entry was in my apache logs:

localhost:443 127.0.0.1 - - [31/Mar/2013:18:33:53 -0400] "POST /xml-rpc/open-ils.actor HTTP/1.1" 500 2001 "-" "xmlrpclib.py/1.0.1 (by www.pythonware.com)"

According to apt, I'm using the latest version of librpc-xml-perl which for me also appears to be 0.76-1.

[Jason Stephenson (jstephenson)]

I installed OpenSRF Master and Evergreen Master both as of 20130328. I followed the directions in the README. I used the script that I posted the other day to test. It works for me. This was tested after a fresh VM was built, btw, so no potential for crud hanging around.

You should check to see if you don't have two RPC-XML modules installed, one from the package and one from CPAN. If you do, I think the one from CPAN shows up in the path first and should probably be removed.

[Jason Stephenson (jstephenson)]

Addendum: I am also running the test script on computer other than the server, i.e. my workstation.

[Chris Sharp (chrissharp123)]

Okay, I built a 2.3.4 Evergreen server on top of 2.1.2 OpenSRF (both built from git) and I was able to see the gateway working as expected. I then built a vanilla server with our PINES customizations installed and again saw success. However, I saw that both the ubuntu-packaged version and the CPAN vesrion of RPC-XML were installed. (Deeper looking revealed that OpenSRF installed the Ubuntu package while the EG 2.3.4 Makefile.install installs RPC::XML::Function via CPAN, which pulled in the RPC-XML dependency along with it. As that issue is resolved by the fix for bug 1050596, there's probably not a need to address it further).

After commenting out RPC::XML::Function in EG's Makefile.install, only Ubuntu's version of RPC-XML is installed and I see the same problem as reported above:

[Mon Apr 01 11:24:52 2013] [error] [client 127.0.0.1] Can't locate object method "args" via package "Error creating request object: RPC::XML::request::new: Invalid method name specified at document line 6, column 0 (byte 112)" (perhaps you forgot to load "Error creating request object: RPC::XML::request::new: Invalid method name specified at document line 6, column 0 (byte 112)"?) at /usr/local/share/perl/5.14.2/OpenILS/WWW/XMLRPCGateway.pm line 69.\n

Not sure where to go from here, so we're now looking at the code to see why "args" would be passed to the RPC::XML::request::new function.

[Jason Stephenson (jstephenson)]

Master and 2.3.4 are different. I believe 2.3.4 still needs the RPC-XML::Function module installed from CPAN. The bug fix did not actually go in until 2.3.5. I have changed that bug description to reflect that.

[Chris Sharp (chrissharp123)]

Jason,

Thanks - we can confirm that removing the Ubuntu-packaged version (an OpenSRF dependency - see bug 1162907), and installing the 0.74 CPAN package that comes with RPC::XML::Function solves our issue. Looks like a bug that crosses both projects because of the conflict in shared dependencies.

Thanks,

Chris

[Chris Sharp (chrissharp123)]

Confirmed on 2.3.5:

[Mon Apr 01 14:58:02 2013] [error] [client 127.0.0.1] Can't locate object method "args" via package "Error creating request object: RPC::XML::request::new: Invalid method name specified at document line 6, column 0 (byte 112)" (perhaps you forgot to load "Error creating request object: RPC::XML::request::new: Invalid method name specified at document line 6, column 0 (byte 112)"?) at /usr/local/share/perl/5.14.2/OpenILS/WWW/XMLRPCGateway.pm line 69.\n

[Lebbeous Fogle-Weekley (lebbeous)]

For what it's worth, I can't see any reason why OpenSRF currently has a dependency on RPC::XML, whether from CPAN or distro packages. OpenSRF itself does not seem to use RPC::XML in any way.

Is anyone here saying that removing it from OpenSRF's Makefile.install would make things go more smoothly? We can open a bug against OpenSRF specifically to remove it.

[Jason Stephenson (jstephenson)]

If that is the case, it would need to be added to Evergreen as master does not currently list it as a prerequisite.

I'm not sure what is going on here and I don't run 2.3.x to have a look. All I can say is that this works fine in master as it is.

It also looks like 2.3.5 has all of the necessary changes for this to work properly there.

[Jason Stephenson (jstephenson)]

"It" in my previous comment being the librcp-xml package.

[Jason Stephenson (jstephenson)]

Lebbeous: https://bugs.launchpad.net/opensrf/+bug/1162907

[Galen Charlton (gmc)]

Per Evergreen commit 0889830f6da97bde7bc957, shouldn't the test script be using two underscores rather than a hyphen in the method name, e.g.,

$ python
>>> import xmlrpclib
>>> s = xmlrpclib.ServerProxy("https://localhost/xml-rpc/open-ils.actor")
>>> f = getattr(s, "open__ils.actor.org_tree.retrieve")
>>> f()

[Jason Stephenson (jstephenson)]

Quoting Galen Charlton <email address hidden>:

> Per Evergreen commit 0889830f6da97bde7bc957, shouldn't the test script
> be using two underscores rather than a hyphen in the method name, e.g.,

It actually should not matter as long as the client permits the - in
the function name.

[Galen Charlton (gmc)]

On Tue, Apr 2, 2013 at 5:20 AM, Jason Stephenson
<email address hidden> wrote:
> It actually should not matter as long as the client permits the - in
> the function name.

Based on the error messages reported, RPC::XML on the server clearly
does care. On #evergreen Mike Peters reported that adjusted the test
script as I had suggested worked for him.

I think the question becomes whether the Evergreen XML-RPC gateway
should accept both forms of the method name.

[Michael Peters (mrpeters)]

It would be optimal if there were a way to make the old method name still work.

I'm sure more and more people/entities are going to find that things are broken as more and more people upgrade into the 2.3 and beyond realm.

[Jason Stephenson (jstephenson)]

Apparently, it does matter now. I suppose that is a change in the RPC-XML perl modules, then. I notice that I get an error when I change my function names back to using the - instead of __.

[Lebbeous Fogle-Weekley (lebbeous)]

Hmm. I think as RPC::XML progressed in version, it started enforcing no dashes in method names on the client side, and must have added no-dashes on the server side later. It's not Evergreen or our XML-RPC gateway (even in the commit Galen indicates) that's being particular.

If we want to keep supporting method names with dashes, as I think we should since things used by third parties to extract e.g. collections information from Evergreen would have to be changed otherwise, it seems we'll either have to start requiring an old version of the RPC::XML package (bad), or perhaps apply some monkey patch (worse?)

[Lebbeous Fogle-Weekley (lebbeous)]

Ah, okay, I was seeing a difference where there isn't one. RPC::XML::Request is of course used both in our gateway (the server, relatively speaking) and clients. I am understanding this problem better now.

A monkey patch solution would just have to target this line of RPC::XML::Request::new(), which probably should have been obvious earlier, if we want to make a system keep accepting XML-RPC requests with method names with dashes in them.

    if ($name !~ m{^[\w.:/]+$}

[Jason Stephenson (jstephenson)]

Instead of monkey patching RPC::XML::Request, some code could go into OpenILS/WWW/XMLRPCGateway.pm at about line 65 to parse the post data and change the - to __ on incoming requests. Of course you'd only want that for function names.

Either way, I think it is better to tell vendors to change their code. After all, we're apparently the ones who have not been following the specification.

[Galen Charlton (gmc)]

My vote: employ the spit and bailing wire for now, but also deprecate support for non-standard method names in the 2.4 release and remove in the next release after 2.4.

[Jason Stephenson (jstephenson)]

This was fixed a long time ago, like in 2.4 or so, and never got closed for whatever reason.

[Chris Sharp (chrissharp123)]

I have no problem with the ticket being considered closed, since this isn't a problem for now, but I'm not sure the Evergreen/OpenSRF-side issue ever received a fix. In the case that triggered this bug report, the XML-RPC client coded around the change, resolving our particular problem with Galen's "spit and bailing wire" approach. As far as I'm aware, there wasn't any EG/opensrf-side change.

Just wanted to mention that in case others experience this problem later and wonder what the fix actually was.