Console-Output does not escape XML correctly

[justinsb]

The console output does not escape characters in the output correctly, when XML output is selected.

Here's an example error stack trace from Java, the important bit is "An invalid XML character (Unicode: 0x1b) was found in the element content of the document.]":

Error running command: get-consoleoutput adb2a214-8728-45d8-ad4d-4ee28a9124f6
javax.ws.rs.WebApplicationException: javax.xml.bind.UnmarshalException
 - with linked exception:
[org.xml.sax.SAXParseException; lineNumber: 335; columnNumber: 36; An invalid XML character (Unicode: 0x1b) was found in the element content of the document.]
 at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.readFrom(AbstractRootElementProvider.java:113)
 at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:554)
 at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:506)
 at com.sun.jersey.api.client.WebResource.handle(WebResource.java:674)
 at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
 at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:563)
 at org.openstack.client.compute.ServerResource.executeAction(ServerResource.java:361)
 at org.openstack.client.compute.ServerResource.getConsoleOutput(ServerResource.java:355)
 at org.openstack.client.cli.commands.GetConsoleOutput.runCommand(GetConsoleOutput.java:21)
 at com.fathomdb.cli.CliSimpleRepl.executeCommand(CliSimpleRepl.java:152)
 at com.fathomdb.cli.CliSimpleRepl.runCommand(CliSimpleRepl.java:90)
 at com.fathomdb.cli.CliBase.mainWithReturnCode(CliBase.java:138)
 at com.fathomdb.cli.CliBase.nailMain(CliBase.java:61)
 at org.openstack.client.cli.OpenstackCli.nailMain(OpenstackCli.java:43)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:601)
 at com.martiansoftware.nailgun.NGSession.run(NGSession.java:298)
Caused by: javax.xml.bind.UnmarshalException
 - with linked exception:
[org.xml.sax.SAXParseException; lineNumber: 335; columnNumber: 36; An invalid XML character (Unicode: 0x1b) was found in the element content of the document.]
 at javax.xml.bind.helpers.AbstractUnmarshallerImpl.createUnmarshalException(AbstractUnmarshallerImpl.java:335)
 at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.createUnmarshalException(UnmarshallerImpl.java:523)
 at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:220)
 at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:189)
 at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:140)
 at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:123)
 at com.sun.jersey.core.impl.provider.entity.XMLRootElementProvider.readFrom(XMLRootElementProvider.java:115)
 at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.readFrom(AbstractRootElementProvider.java:111)
 ... 18 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 335; columnNumber: 36; An invalid XML character (Unicode: 0x1b) was found in the element content of the document.
 at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
 at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
 at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
 at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
 at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1375)
 at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2854)
 at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607)
 at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:116)
 at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:488)
 at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835)
 at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
 at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123)
 at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210)
 at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:568)
 at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:216)
 ... 23 more

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/5545

[justinsb (justin-fathomdb)]

Vishy and I discussed this on IRC.

It's down to a bug in MiniDom, and affects all XML output.
Any string that outputs characters < 32 should really be escaped. Minidom doesn't do this: http://bugs.python.org/issue5752
I don't believe this has any e.g. security consequences other than the fact that we produce invalid XML. i.e. this isn't as bad as escape in HTML.
We probably want to move to lxml anyway. This may fix the problem for free.
It doesn't seem like a good idea to mess with all XML output at this stage.
Console output is particularly problematic because ANSI sequences start with 0x1B.
The patch therefore masks the problem for console-output (only)
Personally, I don't mind if the patch goes in or not; I can easily switch between using JSON and using XML in my binding.
In Folsom, we should fix this properly (with lxml or otherwise)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/5545
Committed: http://github.com/openstack/nova/commit/13ca1b923e479905f9d1debdea6125be92f44808
Submitter: Jenkins
Branch: master

commit 13ca1b923e479905f9d1debdea6125be92f44808
Author: Justin Santa Barbara <email address hidden>
Date: Mon Mar 19 16:33:16 2012 -0700

    Strip out characters that should be escaped from console output

    Minidom has a bug where it doesn't escape characters < 32 correctly.
    That is http://bugs.python.org/issue5752

    We probably want to move to lxml anyway.

    Console-output is particularly likely to have invalid characters,
    because ANSI output include the ESC character (0x1B). Strip out
    invalid characters for this case (although the problem is still
    present anywhere we write a string in the XML)

    Bug #939386

    Change-Id: I4689c0f26fb4887867a8d5b6f5e5fb5e2590be91