Swift authentication fails with transfer-encoding chunked

[JC]

We're trying to use the xb utilities, and specifically xbcloud, against Rackspace CloudFiles. While this may not be officially supported we thought it might be helpful to report a bug that has the potential to affect others and gets us one step closer to working on Rackspace.

After digging through the code the issue seems to be with fetch_buffer_header_cb or swift_keystone_auth_v2 (depending on your perspective) within https://github.com/percona/percona-xtrabackup/blob/2.4/storage/innobase/xtrabackup/src/xbcloud.cc. Rackspace is returning a response with Transfer-Encoding: chunked rather then a Content-Length. As a result it appears no memory is allocated for the response body and the below assertion is thrown.

fetch_buffer_header_cb could be updated to calculate the size for a chunked stream, OR possibly swift_keystore_auth_v2 could be updated to create a download_buffer_info with a custom header callback that handles calculating the size for a chunked stream.

I am not a C programmer so forgive me if I grossly overlooked something.

(gdb) run get --verbose --swift-container=testing --swift-auth-url=https://identity.api.rackspacecloud.com/ --swift-user=org --swift-auth-version=2.0 --swift-password="ryolxcSR3xg" baz
Starting program: /usr/bin/xbcloud get --verbose --swift-container=testing --swift-auth-url=https://identity.api.rackspacecloud.com/ --swift-user=org --swift-auth-version=2.0 --swift-password="ryolxcSR3xg" baz
[Thread debugging using libthread_db enabled]
Detaching after fork from child process 24198.
* About to connect() to identity.api.rackspacecloud.com port 443 (#0)
* Trying 166.78.226.217... * connected
* Connected to identity.api.rackspacecloud.com (166.78.226.217) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
Detaching after fork from child process 24199.
Detaching after fork from child process 24200.
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
* subject: CN=identity.api.rackspacecloud.com,O="Rackspace US, Inc.",L=San Antonio,ST=Texas,C=US
* start date: Jun 17 00:00:00 2016 GMT
* expire date: Jun 17 23:59:59 2018 GMT
* common name: identity.api.rackspacecloud.com
* issuer: CN=thawte SSL CA - G2,O="thawte, Inc.",C=US
> POST /v2.0/tokens HTTP/1.1
Host: identity.api.rackspacecloud.com
Content-Type: application/json
Accept: application/json
Content-Length: 85

< HTTP/1.1 200 OK
< Server: nginx
< Date: Wed, 24 Aug 2016 03:55:00 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Accept, Accept-Encoding, X-Auth-Token
< X-NewRelic-App-Data: PxQCWVFVDwQTVVRTBAQAUlATGhE1AwE2QgNWEVlbQFtcC2VOYwRAFjNTVTIDEU5aUwE9TUNeXAMKQhYbMikwNhlAFFIWCBgCHVUKVwhXAVtRBBtMV08aVgcBWFYDUA9TXAlTB1teAkAcBFkOS11p
< x-trans-id: eyJyZXF1ZXN0SWQiOiI1MmUyYjY2OS1lZDc5LTQ5NDktYTExZS04NmU5MGI2NTliNjgiLCJvcmlnaW4iOm51bGx9
< Front-End-Https: on
<
xbcloud: /mnt/workspace/percona-xtrabackup-2.4-redhat-binary/label_exp/centos6-64/rpmbuild/BUILD/percona-xtrabackup-2.4.4/storage/innobase/xtrabackup/src/xbcloud.cc:1534: size_t fetch_buffer_cb(char*, size_t, size_t, void*): Assertion `buffer_info->size >= buffer_info->offset + size * nmemb' failed.

Program received signal SIGABRT, Aborted.
0x0000003179232625 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6.x86_64 glibc-2.12-1.149.el6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-33.el6.x86_64 libaio-0.3.107-10.el6.x86_64 libcom_err-1.41.12-21.el6.x86_64 libcurl-7.19.7-37.el6_5.3.x86_64 libev-4.03-3.el6.x86_64 libgcc-4.4.7-11.el6.x86_64 libgcrypt-1.4.5-11.el6_4.x86_64 libgpg-error-1.7-4.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6.x86_64 libstdc++-4.4.7-11.el6.x86_64 nspr-4.10.6-1.el6_5.x86_64 nss-3.16.1-14.el6.x86_64 nss-softokn-3.14.3-17.el6.x86_64 nss-softokn-freebl-3.14.3-17.el6.x86_64 nss-sysinit-3.16.1-14.el6.x86_64 nss-util-3.16.1-3.el6.x86_64 numactl-2.0.9-2.el6.x86_64 openldap-2.4.39-8.el6.x86_64 openssl-1.0.1e-30.el6.x86_64 sqlite-3.6.20-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x0000003179232625 in raise () from /lib64/libc.so.6
#1 0x0000003179233e05 in abort () from /lib64/libc.so.6
#2 0x000000317922b74e in __assert_fail_base () from /lib64/libc.so.6
#3 0x000000317922b810 in __assert_fail () from /lib64/libc.so.6
#4 0x00000000004083da in fetch_buffer_cb (buffer=<value optimized out>, size=<value optimized out>, nmemb=<value optimized out>, out_buffer=0x7fffffff42f0)
    at /usr/src/debug/percona-xtrabackup-2.4.4/storage/innobase/xtrabackup/src/xbcloud.cc:1534
#5 0x0000003181a17a48 in Curl_client_write () from /usr/lib64/libcurl.so.4
#6 0x0000003181a2e69c in Curl_httpchunk_read () from /usr/lib64/libcurl.so.4
#7 0x0000003181a2ab22 in Curl_readwrite () from /usr/lib64/libcurl.so.4
#8 0x0000003181a2c0f8 in Curl_perform () from /usr/lib64/libcurl.so.4
#9 0x000000000040a2c5 in swift_keystone_auth_v2 (auth_url=0x7fffffffc3f0 "https://identity.api.rackspacecloud.com/v2.0/tokens", info=0x7fffffff43f0)
    at /usr/src/debug/percona-xtrabackup-2.4.4/storage/innobase/xtrabackup/src/xbcloud.cc:2267
#10 0x000000000040dbdd in main (argc=<value optimized out>, argv=<value optimized out>) at /usr/src/debug/percona-xtrabackup-2.4.4/storage/innobase/xtrabackup/src/xbcloud.cc:2650
(gdb) quit

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-1400