Error during pymqi.connect on linux x86_64

[Alexander Fomichev]

Python version is 2.7
WebSphere MQ version (client and sever) is 7.5
Server - RedHat 6.2 x86_64
Client - Fedora 16 x86_64

Steps to reproduce:
1. install:
     MQSeriesClient-7.5.0-0.x86_64.rpm
     MQSeriesRuntime-7.5.0-0.x86_64.rpm
     MQSeriesSDK-7.5.0-0.x86_64.rpm
2. download pymqi-1.2.tar.gz
     unpack
     python setup.py build client
     sudo python setup.py install
3. run script with following content:
    #!/usr/bin/env python
    # -*- coding:utf-8 -*-

    import pymqi

    queue_manager = "TEST_QM"
    channel = "CLIENT_TEST_CH"
    host = "10.10.10.20"
    port = "1414"

    conn_info = "%s(%s)" % (host, port)
    qmgr = pymqi.connect(queue_manager, channel, conn_info)

    qmgr.disconnect()

4. Exception on connect:

qmgr = pymqi.connect(queue_manager, channel, conn_info)
  File "/usr/lib64/python2.7/site-packages/pymqi.py", line 2170, in connect
    qmgr.connect_tcp_client(queue_manager, CD(), channel, conn_info)
  File "/usr/lib64/python2.7/site-packages/pymqi.py", line 1245, in connectTCPClient
    self.connectWithOptions(name, cd)
  File "/usr/lib64/python2.7/site-packages/pymqi.py", line 1224, in connectWithOptions
    rv = pymqe.MQCONNX(name, options, ocd.pack())
pymqe.error: MQCD wrong size. Given: 1912, expected 1920

64bit errors again?

[Dariusz Suchojad (dsuch)]

Hi Alex,

> pymqe.error: MQCD wrong size. Given: 1912, expected 1920
> 64bit errors again?

This could also be case because you're using MQ 7.5 which I haven't tried yet. I didn't even realize it was out, I'm with 7.1 everywhere.

As I won't be able to have a look at it for at least a week now - any chance you could downgrade to 7.1 before I look into it?

Many thanks!

[Alexander Fomichev (fomichev-alexander)]

Hi Dariusz,

Unfortunately, I can't change MQ version, because it's not my installation. I'm working only under python-wrapper for our infrastructure. But I can test your changes here on MQ 7.5 and give you immediate feedback.

May be I can do some fixes by myself here if you guide me.

Anyway, thank you for your responce.

[Dariusz Suchojad (dsuch)]

> May be I can do some fixes by myself here if you guide me.

Sure, so the whole deal looks like:

- When compiling PyMQI, pymqe.c makes a note how much bytes certain MQ C structures take up. Here we focus on PYMQI_MQCD_SIZEOF

- In run, pymqi.py uses the CD class for constructing MQCD instances. If you have a look at this class you'll see that it's composed of several dozen attributes, such as XmitQName or ExitNameLength.

- Now, each of these attributes has a name, default value and a data type. The data type must be in the format understood by Python's struct module.

- In runtime, pymqi.py constructs instances of CD and their length, which is the sum of all of the attributes, must be equal to what PYMQI_MQCD_SIZEOF has been set to during the compilation. If they differ, an exception like the one you've stumbled upon, is raised.

So the thing is to find out - by comparing CD with IBM infocenter's docs regarding MQCD - what fields, if any, are missing, superfluous or have incorrect length set in the CD class.

[Alexander Fomichev (fomichev-alexander)]

According to the IBM MQ documentation version 7.5 these fields were added to tagMQCD structure:

 MQLONG BatchDataLimit; /* Batch data limit */
 MQLONG UseDLQ; /* Use Dead Letter Queue */
 MQLONG DefReconnect; /* Default client reconnect */
                                                          /* option */

In header file in /opt/mqm/inc/cmqxc.h these lines were added:

102 #if defined(MQ_64_BIT)
103 #define MQCD_LENGTH_10 1920
104 #else
105 #define MQCD_LENGTH_10 1876
106 #endif
107 #if defined(MQ_64_BIT)
108 #define MQCD_CURRENT_LENGTH 1920
109 #else
110 #define MQCD_CURRENT_LENGTH 1876
111 #endif

[Alexander Fomichev (fomichev-alexander)]

I've made some fixes and successfully connected to websphere ver. 7.5
Please check it and if it pass all tests make a new version for linux and win platform.

[Dariusz Suchojad (dsuch)]

Thanks Alexander, sorry for a late reply but this looks really good and will go into a new release some time in an almost-near future!

[Fred Cohen (a-launchpad)]

Ran into the same problem with MQ Client 7.1.0 on linux x64.

After applying the patch I had to change pymqe.c to treat the 7.1 client as though it was a 7.5.
I don't have a 7.5 install to actually compare it to so there may be some side-effects to doing this. If I get some more time I'll go back and add a full "7.1" support to the __mqlevels__ tuple

pymqe.c line 1273:
#ifdef MQCMDL_LEVEL_711
      PyList_Append(versions, PyString_FromString("7.5"));

[Dariusz Suchojad (dsuch)]

> Ran into the same problem with MQ Client 7.1.0 on linux x64.
>
> After applying the patch I had to change pymqe.c to treat the 7.1 client as though it was a 7.5.
> I don't have a 7.5 install to actually compare it to so there may be some side-effects to doing this.
> If I get some more time I'll go back and add a full "7.1" support to the __mqlevels__ tuple
>
> pymqe.c line 1273:
> #ifdef MQCMDL_LEVEL_711
> PyList_Append(versions, PyString_FromString("7.5"));
>

Ah bummer :(

So basically, it doesn't work for you with 7.1? Really sorry to hear
that, please let me know if you manage to get it working or if you'd
rather if I had a look at it.

Cheers!

[Fred Cohen (a-launchpad)]

Sorry, I should have mentioned that I was getting the exact same error reported in this bug:
pymqe.error: MQCD wrong size. Given: 1912, expected 1920

As far as I can tell, with the combination of the patch and my change I'm at least able to list the queues in our queue manager. I can't go any further in testing because I don't have a queue setup for me to use yet.

I'm using a 64-bit Python 2.6.6 w/ WebSphere MQ client 7.1.0

I wish I had a 7.5 client install readily available to compare the C headers to, but I'm guessing this version is very similar if not totally compatible.

[Fred Cohen (a-launchpad)]

I was able to find all the data type changes documented between 7.0.1 to 7.5 in my installation of 7.1

These are the changes listed here: http://pic.dhe.ibm.com/infocenter/wmqv7/v7r5/index.jsp?topic=%2Fcom.ibm.mq.doc%2Fmi77539_.htm

The only obvious difference is that MQCMDL_LEVEL_750 is not defined while these two are:
#define MQCMDL_LEVEL_710 710
#define MQCMDL_LEVEL_711 711

[Alexander Fomichev (fomichev-alexander)]

Hi Fred,
I'll have a look, but my patch is not a universal patch for all version. I did it against version 1.2 to support my 7.5 version.
This patch will work with MQ version 7.5 but I can not be sure about previous versions with the same error.
Please note, that this patch is not a direct and universal cure for "pymqe.error: MQCD wrong size. Given: 1912, expected 1920" error.
The error appeared because of absence of MQ 7.5 support.
At the moment I do not have a Queue to connect, but I will check the patch and constants.

[Alexander Fomichev (fomichev-alexander)]

CMQC.py
line 1761: MQCMDL_LEVEL_750 = 750 should be added

But this changes will not fix any behavior in your version, because this defines will be set in IBM headers, and they will appear only in 7.5 version and higher. So, this patch will not add any differences if you have MQ headers from version lower than 7.5, because all changes are under #ifdef declarations.
I will do a new patch with changes stated above, but it will not help you with previous versions.

We should ask Dariusz to look in to this stuff.

Please be patient.

[Alexander Fomichev (fomichev-alexander)]

There it is.
I have added MQCMDL_LEVEL_750 declaration.

[Dariusz Suchojad (dsuch)]

On 01/19/2013 01:16 AM, Alexander Fomichev wrote:

> this defines will be set in IBM headers, and they will appear only in 7.5 version and higher.

I believe Alexander is right. The very error 'Given: 1912, expected
1920' that the MQ version PyMQI was compiled with was 7.5 - I know this
may sound crazy, Ken, but this is how what is says. This is because it
was MQ 7.5 that changed MQCD's length to 1920.

So please, can you double-check that you don't have two MQ versions
somehow laying about?

I'll try to download both 7.1 and 7.5 the next week and see how it looks
like. I've got some problems with my dev environment so it can take some
time.

--
Dariusz Suchojad

[Removed by request (removed4124968)]

Too bad I can't edit the comments ... (https://bugs.launchpad.net/launchpad/+bug/119420).

with 7.5 client still installed, I rebuilt PyMQI may times, updating MQCMDL_LEVEL with the comments above to have 750, as well as updating pymqe.c to add ifdef MQCMDL_LEVEL def ... but still no love.

so i removed 7.5 client and installed 7.1 client, and updated pymqe.c and CMQC.py also as above

But still no love ... with 7.1 client

Getting the same error:
    rv = pymqe.MQCONNX(name, options, ocd.pack())
error: MQCD wrong size. Given: 1912, expected 1920

The main reason that I have updated the client is that it offers the --prefex on the rpm. I require mq client to be transportable on a san mount. According to the IBM webpage the client version for the 7.x are backward compatible, is that true about pymqi? (server is 7.0.1.2)

[Alexander Fomichev (fomichev-alexander)]

Dear Jason,

along with the client you should keep the same version of runtime and SDK, as far as I remember, headers for pymqi come from SDK package and you can rebuild your PyMQI as many times as you want it will not change anything, because you rebuild it against the same headers.

And again, I will try to highlight it, THIS PATCH IS FOR :
* Client version 7.5 with Runtime version 7.5 and with PyMQI 1.2 built against headers from SDK version 7.5
* MQ Server should have version 7.5 as well

IF ANY OF THESE COMPONENTS has a different version, you will not be able to build PyMQI properly or it will not work properly!

to make sure that you do everything right, please follow the instructions in the first comment, make test and check that you
have this error, than apply the patch and check everything again.
If you have any component from the described list with the different version, it is not the case.

Only the same version of Client, Server, Runtime and SDK will lead you to the successful build of PyMQI and it will work only for this environment.

Good luck

[Dariusz Suchojad (dsuch)]

I've just released PyMQI 1.3 and the development effort has moved to GitHub - https://github.com/dsuch/pymqi