Qpid cannot serialize objects containing strings longer than 65535 characters

[Ben Nemec]

Qpid has a limitation where it cannot serialize a Python dict containing a string longer than 65535 characters. This can result in problems when making a conductor call that returns a large structure - for example, instance_get_all_by_host on one of my systems returns 38 instances, which when serialized as JSON is too long for Qpid to handle.

There's a discussion of the limitation here: https://issues.apache.org/jira/browse/QPID-4583

I believe the best fix here is to wrap the string in a buffer as suggested in the link above. This works fine on my local system and seems to fix the problem (note: I haven't tested against RabbitMQ yet, and right now I put the change in common code so that will need to be done). However, that will be a backward compatibility breaking change because it will need to be handled differently on the receiving end as well.

[Michael Still (mikal)]

This is implemented in oslo.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/28711

[Ben Nemec (bnemec)]

One note on my proposed fix: I ended up not wrapping the long string in a buffer because that required the qpid code to know about the message format, which seemed like a bad thing. This change is transparent to external code and the only incompatibility is that the old implementation won't be able to handle the new messages, but the other direction will work fine.

[Mark McLoughlin (markmc)]

The code is in oslo-incubator but does need to be synced across to Nova, so think we should keep a Nova task open

[Steven Hardy (shardy)]

Added Heat, as you can see from the qpid JIRA I've already been down the path of investigating this, and came to the conclusion (for the Heat use-case at least) that the best option was *not* to JSON encode, since the recursive map serialization works OK for the data we typically need to pass (where none of individual the dict/map values contain strings >65535 but the JSON encoded entire template can easily be)

If there is a transparent workaround in oslo then we should sync that into Heat, as it's still possible users could encounter this limitation if they put particularly huge strings into a Heat template.

bug #1072917 has further details on my previous investigation into this.

[Russell Bryant (russellb)]

Doing encoding before it hits qpid is required so that we can do message signing, though.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/28711
Committed: http://github.com/openstack/oslo-incubator/commit/7ce54410485b33cffc39c7ffb96eae422b88601c
Submitter: Jenkins
Branch: master

commit 7ce54410485b33cffc39c7ffb96eae422b88601c
Author: Ben Nemec <email address hidden>
Date: Thu May 9 19:06:45 2013 +0000

    Fix problem with long messages in Qpid

    Qpid has a limitation where it cannot serialize a dict containing
    a string greater than 65535 characters. This change alters the
    Qpid implementation to JSON encode the dict before sending it, but
    only if Qpid would fail to serialize it. This maintains as much
    backward compatibility as possible, though long messages will
    still fail if they are sent to an older receiver.

    Fixes bug 1175808

    Change-Id: I5d104e099f523508dae2b657f7d06d96984b10f0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/31689

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/31803

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/31803
Committed: http://github.com/openstack/nova/commit/781a8f908cd3e5e69ff8b88d998fa93c48532e15
Submitter: Jenkins
Branch: master

commit 781a8f908cd3e5e69ff8b88d998fa93c48532e15
Author: Andrew Laski <email address hidden>
Date: Wed Jun 5 10:02:07 2013 -0400

    Update rpc/impl_qpid.py from oslo

    The current qpid driver cannot serialize objects containing strings
    longer than 65535 characters. This just became a breaking issue when
    the message to scheduler_run_instance went over that limit. The fix has
    been commited to oslo, so this just syncs it over to Nova.

    Bug 1175808
    Bug 1187595

    Change-Id: If95c11a7e03c81d89133f6cad0dcbb6d8acb8148

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/32173

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/32291

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/32173
Committed: http://github.com/openstack/quantum/commit/68a5a387020cdb0c5d664b7add2d897134877d19
Submitter: Jenkins
Branch: master

commit 68a5a387020cdb0c5d664b7add2d897134877d19
Author: Ben Nemec <email address hidden>
Date: Fri Jun 7 16:35:14 2013 +0000

    Sync Qpid RPC fix from Oslo

    Qpid cannot serialize dicts containing strings longer than 65535
    characters. This change syncs the fix from Oslo to Quantum.

    Fixes bug 1175808

    Change-Id: I48071abffa86e71727deed05aca08ac475cbaf05

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/31689
Committed: http://github.com/openstack/oslo-incubator/commit/860827fafcc020ab0d7f9d56b6f03b6e8b8ebc7b
Submitter: Jenkins
Branch: stable/grizzly

commit 860827fafcc020ab0d7f9d56b6f03b6e8b8ebc7b
Author: Ben Nemec <email address hidden>
Date: Tue Jun 4 18:52:06 2013 +0000

    Grizzly compatibility with Havana Qpid messages

    A change had to be made in the Havana Qpid implementation that is
    not entirely backwards compatible. This change allows Grizzly to
    receive messages in the new format.

    Backport of change for bug 1175808

    Change-Id: I8b6c5734b67cd33cdd5ef0659bdd77504317187d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/33476

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/33478

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/32291
Committed: http://github.com/openstack/heat/commit/fee94d28e780eb63be2bed608ecaab5750c07267
Submitter: Jenkins
Branch: master

commit fee94d28e780eb63be2bed608ecaab5750c07267
Author: guohliu <email address hidden>
Date: Wed Jun 19 20:18:39 2013 +0800

    Updated common module from oslo

    This patch updated common module from oslo for known issue.

    Fixed bug: #1175808
    Fixed bug: #1072917

    Change-Id: Ic1de6b786a9d67b39d186e31a37049729d367464

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/33478
Committed: http://github.com/openstack/quantum/commit/a99cc45585759014c14491ab5eb45429f207e749
Submitter: Jenkins
Branch: stable/grizzly

commit a99cc45585759014c14491ab5eb45429f207e749
Author: Ben Nemec <email address hidden>
Date: Tue Jun 18 17:11:33 2013 +0000

    Sync Qpid RPC fix from Oslo

    A change had to be made in the Havana Qpid implementation that is
    not entirely backwards compatible. This change allows Grizzly to
    receive messages in the new format.

    Backport of change for bug 1175808

    Change-Id: I6ab63c3fccf562aeeb793e09f057c66bc109ef41

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/33476
Committed: http://github.com/openstack/nova/commit/d2cb4b25f7226cd2051d38905942e515d257931c
Submitter: Jenkins
Branch: stable/grizzly

commit d2cb4b25f7226cd2051d38905942e515d257931c
Author: Ben Nemec <email address hidden>
Date: Tue Jun 18 16:59:04 2013 +0000

    Sync RPC module from Oslo

    Includes backport of change for bug 1175808
    Fixes bug 1172922 and bug 1122763

    Change-Id: Ib52e9458a9db01e2c8fd2a6d84010733f85980f8