http client throws error with no content-type header in http response

[Dennis Knochenwefel]

in modules/com/zorba-xquery/www/modules/http-client.xq.src/http_response_parser.cpp the function HttpResponseParser::parse() throws an error if the response doesn't contain a content-type. In this case theCurrentCharset is "".

the surprising part is that in the following code transcode::is_necessary("") returns true and an icu_streambuf is created internally with an empty charset:

if ( transcode::is_necessary( theCurrentCharset.c_str() ) ) {
  lStream.reset(
    new transcode::stream<std::istream>(
      theCurrentCharset.c_str(), theStreamBuffer
    )
  );
}

in the constructor of the icu_streambuf (src/util/icu_streambuf.cpp) the following code throws the error:

external_conv_( no_conv_ ? nullptr : create_conv( charset ) )

the correct behavior would be if the http-client would not throw an error if the content-type header is missing.

[Paul J. Lucas (paul-lucas)]

Are you also saying that the correct behavior would also be to not do transcoding and just hope that the unspecified character set is UTF-8?

[Dennis Knochenwefel (dennis-knochenwefel)]

a comment in the http_response_parser.cpp says that the HTTP/1.1 spec says that the default charset is ISO-8859-1. That is the charset used in the response parser if a content-type without charset is returned.

[Paul J. Lucas (paul-lucas)]

When I run the sample query, I get:

</Users/pjl/src/flwor/zorba/repo/bug-996593/zorba/build/URI_PATH/org/expath/ns/http-client.xq>:108,1: static error [err:XQST0059]: "http://www.zorba-xquery.com/modules/converters/html-options": target namespace not found for schema/module "http://www.zorba-xquery.com/modules/converters/html-options.xsd": Could not create stream resource; raised at /Users/pjl/src/flwor/zorba/repo/bug-996593/zorba/src/compiler/translator/translator.cpp:2069

[Paul J. Lucas (paul-lucas)]

Also, what should happen if a Content-Type and charset are provided, but the charset is not supported by ICU?

[Matthias Brantner (matthias-brantner)]

      throw XQUERY_EXCEPTION(
          zerr::ZXQP0006_UNKNOWN_ENCODING,
          ERROR_PARAMS( lEncodingStr.c_str() ),
          ERROR_LOC( loc )
        );

[Paul J. Lucas (paul-lucas)]

@Matthias: that code works only for the core and core modules, not modules like the HTTP module.

[Matthias Brantner (matthias-brantner)]

I know but we can still raise zerr:ZXQP0006 from the http-client module.

[Paul J. Lucas (paul-lucas)]

Still waiting for Dennis to give me a working query to test....

[Dennis Knochenwefel (dennis-knochenwefel)]

> When I run the sample query, I get:
>
> </Users/pjl/src/flwor/zorba/repo/bug-996593/zorba/build/URI_PATH/org/expath/ns
> /http-client.xq>:108,1: static error [err:XQST0059]: "http://www.zorba-
> xquery.com/modules/converters/html-options": target namespace not found
> for schema/module "http://www.zorba-xquery.com/modules/converters/html-
> options.xsd": Could not create stream resource; raised at
> /Users/pjl/src/flwor/zorba/repo/bug-996593/zorba/src/compiler/translator/translator.cpp:2069
>

There is a dependency of the external http-client module to the
data-converters module:

FIND_PACKAGE(zorba_data-converters_module REQUIRED HINTS
${DATA_CONVERTERS_BUILD_DIR})

In addition to the http-client I think you need to check it out to

/Users/pjl/src/flwor/zorba/repo/bug-996593/zorba_modules

I just double checked and the query works on ubuntu 12.04. with the data-converters module installed.