XML RPC simpleXML limits payload size

Bug #1422837 reported by guy thomas
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Mahara
Won't Fix
Medium
Unassigned

Bug Description

Operating System: Ubuntu 14.04.1 LTS

Mahara version: 1.9.3

Database: mysql

Browser: chrome 40.0.2214.111

------------------------------

Steps to reproduce - you need a system capable of XMLRPC communication with Mahara (e.g. Moodle)

1) With Moodle - set up mahara networking / portfolios.
2) Open a course with a forum
3) Add 2 entries. One with an attachment under 5 MB. One with an attachment over 5 MB. On Each click "Export to portfolio" in the bottom. 4) Go through the screens to start export - ask it to do it now (i.e. not wait for cron).
5) Check the Mahara site to see what made it through successfully.

------------------------------

simpleXML may not be the best parser for XMLRPC payloads

in api/xmlrpc/lib.php

function parse_payload($payload) {
    try {
        $xml = new SimpleXMLElement($payload);
        return $xml;
    } catch (Exception $e) {
        throw new MaharaException('Encrypted payload is not a valid XML document', 6002);
    }
}

This means if you send a large file (in our tests greater than approx 7mb) base 64 encoded over XMLRPC it will fail to parse.
It didn't seem to make any difference if we increased the memory limit to 800M in php.ini or even by setting the php memory limit to 800M via PHP ini_set in function parse_payload - this suggests that there is a limit to what SimpleXML can handle regardless of the PHP memory limit.

Note - 932 in my source code is the 'throw new exception' following $xml = new SimpleXMLElement($payload);

called at [/vagrant/www/maharadev/mahara/lib/errors.php:465] #1 exception(MaharaException Object ([] => 1,[] => Encrypted payload is not a valid XML document,[] => ,[] => 6002,[] => /vagrant/www/maharadev/mahara/api/xmlrpc/lib.php,[] => 932,[] => Array ([0] => Array ([file] => /vagrant/www/maharadev/mahara/api/xmlrpc/lib.php,[line] => 921,[function] => parse_payload,[args] => Array ([0] =>ERROR 4:
4: remote server error: code: , message: A nonrecoverable error occurred. This probably means you have encountered a bug in the system #0 MaharaException->handle_exception() called at [/vagrant/www/maharadev/mahara/lib/errors.php:465] #1 exception(MaharaException Object ([] => 1,[] => Encrypted payload is not a valid XML document,[] => ,[] => 6002,[] => /vagrant/www/maharadev/mahara/api/xmlrpc/lib.php,[] => 932,[] => Array ([0] => Array ([file] => /vagrant/www/maharadev/mahara/api/xmlrpc/lib.php,[line] => 921,[function] => parse_payload,[args] => Array ([0] =>

guy thomas (brudinie-d)
description: updated
guy thomas (brudinie-d)
description: updated
Revision history for this message
Aaron Wells (u-aaronw) wrote :

Hi Guy,

Thanks for the bug report! What version of Moodle are you using? And were there any other additional configuration steps you had to take, to be able to export a Moodle forum post with a file attachment?

I ask because I wasn't able to get any file attachment to go through from Moodle to Mahara. I tried Moodle 2.6 and 2.8, and I found that when I attached a file to a Moodle forum post, Moodle did not display the "Export to portfolio" link next to that particular post.

I also tried using the TinyMCE "image" embed button, to put a very small image into a Moodle forum post. This did show me the "Export to portfolio" link, but it gave the error "No common formats between any available portfolio plugin and the calling location forum_portfolio_caller (caller supported )". Which is apparently a different problem from what you're reporting, because this image file was much smaller than 5MB.

It's worth noting that we're in the process of replacing MNet with a new webservices system. But, the new webservices system also uses SimpleXML, so it may well face this same problem.

Cheers,
Aaron

Changed in mahara:
status: New → Incomplete
importance: Undecided → Medium
Revision history for this message
guy thomas (brudinie-d) wrote :

Sorry for the delay Aaron - I was maxed out last week.

Moodle version is 2.7

Export to portfolio instructions to follow in the next comment.

Revision history for this message
guy thomas (brudinie-d) wrote :

Full instructions

Enable networking in Mahara

[mahara]/admin/site/networking.php

Create a new institution for your moodle in mahara

[mahara]/admin/users/institutions.php

Add an XML-RPC authentication instance

Copy the certificate from moodle into Mahara

[moodle]/admin/mnet/index.php (copy certificate from here)

(NOTE - your front page settings short site name has to be your host name for the certificate to be correct - if its not correct then change the front page settings and then recreated (delete) the noodle certificate ([moodle]/admin/settings.php?section=frontpagesettings)

Add mahara as a host to moodle

[moodle]/admin/mnet/peers.php

(set hostname to mahara host name and application type to mahara then save)

enable all services by clicking the services tab (publish and subscribe) and then save

Add the networking block to your moodle front page - it will probably complain with “MNet authentication plugin must be enabled to see the list of MNet network servers”

Enable MNet authentication in

[moodle]/admin/settings.php?section=manageauths

The networking block will probably now complain with “Users need the capability 'Roam to a remote application via MNet' to see the list of MNet network servers”

Add the capability ‘Roam to a remote application’ to the ‘authenticated user’ role [moodle]/admin/roles/manage.php

Save the role definition

Sometimes saving the role definition does not save the capability you just checked, so check the front page block again.

All being good you should now see your mahara server as an option under the network servers block on your moodle front page.

Make sure portfolios are switched on in moodle
[moodle]/admin/settings.php

Make sure that mahara eportfolios are switched on in moodle
http://joule2.dev/admin/portfolio.php

Create a test course in moodle

Create a test forum

Add discussion and replies with a few small attachments

Next to the attachments you will see an icon with an up arrow (yes, its really that bad)

Click this arrow to export to Mahara

Now add some big attachments (7mb +)

Try exporting these files and it will fail

Aaron Wells (u-aaronw)
Changed in mahara:
status: Incomplete → Triaged
milestone: none → 1.9.5
Revision history for this message
Aaron Wells (u-aaronw) wrote :

Okay, I was able to replicate the problem. I think the step I must have been missing before, is that in the Mahara portfolio plugin's settings in Moodle, you need to set "Enable Leap2A portfolio support" to "Yes".

Once I do that, then I find that I am able to export a forum post with a small file attachment, but not a forum post with more than 5 MB of file attachments.

Changed in mahara:
status: Triaged → Confirmed
no longer affects: mahara/15.10
tags: added: mnet
tags: added: no-behat-needed
Aaron Wells (u-aaronw)
no longer affects: mahara/1.8
Revision history for this message
Kristina Hoeppner (kris-hoeppner) wrote :

Ghada also discovered this problem.

no longer affects: mahara/1.10
no longer affects: mahara/1.9
no longer affects: mahara/15.04
no longer affects: mahara/15.10
no longer affects: mahara/16.04
Revision history for this message
Ghada El-Zoghbi (ghada-z) wrote :

Hi,

Yes, I did find this as an issue while using Moodle 2.7 and 2.8.

I did try to use LIBXML_PARSEHUGE as a parameter in the SimpleXMLElement() call but the result came back empty - as in this php bug:

https://bugs.php.net/bug.php?id=65604

Thanks,
Ghada

Revision history for this message
Aaron Wells (u-aaronw) wrote :

Maybe we should switch to XMLReader (another PHP core library)? It's a "pull parser" that only reads part of the XML file into memory at a time; whereas SimpleXML is a "DOM parser" that reads and process the entire XML file at once.

XMLReader is also using libxml on the back end, but it seems to go through a different interface that is specifically designed to minimize memory use, so the LIBXML_PARSEHUGE thing might not matter.

Robert Lyon (robertl-9)
Changed in mahara:
milestone: 16.10.1 → 17.04.0
no longer affects: mahara/16.10
Changed in mahara:
milestone: 17.04.0 → 17.10.0
Robert Lyon (robertl-9)
Changed in mahara:
milestone: 17.10.0 → 18.04.0
Robert Lyon (robertl-9)
Changed in mahara:
milestone: 18.04.0 → 18.10.0
Revision history for this message
Kristina Hoeppner (kris-hoeppner) wrote :

We are moving away from MNet to LTI.

Changed in mahara:
status: Confirmed → Won't Fix
milestone: 18.10.0 → none
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.