group_id problem in Partner Import

[Grzegorz Grzelak (OpenGLOBE.pl)]

Before Partner Import I Imported successfully Customer Groups (no errors 5 - groups imported).

Then None Partner was imported. Typical bug report is below.

2013-08-10 12:08:54,378 2591 DEBUG v70_ff_20130809_migrated_01 openerp.addons.connector.queue.worker: <Job 221ffab1-c6d2-482c-bea8-d383581e3384, priority:10> started
2013-08-10 12:08:55,046 2591 DEBUG v70_ff_20130809_migrated_01 openerp.addons.magentoerpconnect.unit.backend_adapter: api.call(customer.info, [656, None]) returned {'customer_id': '656', 'updated_at': '2013-02-01 13:20:57', 'increment_id': ''}
2013-08-10 12:08:55,295 2591 ERROR v70_ff_20130809_migrated_01 openerp.addons.connector.queue.worker: Traceback (most recent call last):
  File "/home/gg/addons-70-custom/connector/queue/worker.py", line 121, in run_job
    job.perform(session)
  File "/home/gg/addons-70-custom/connector/queue/job.py", line 460, in perform
    self.result = self.func(session, *self.args, **self.kwargs)
  File "/home/gg/addons-70-custom/magentoerpconnect/unit/import_synchronizer.py", line 311, in import_record
    importer.run(magento_id, force=force)
  File "/home/gg/addons-70-custom/magentoerpconnect/unit/import_synchronizer.py", line 155, in run
    self._import_dependencies()
  File "/home/gg/addons-70-custom/magentoerpconnect/partner.py", line 272, in _import_dependencies
    if binder.to_openerp(record['group_id']) is None:
KeyError: 'group_id'

[Grzegorz Grzelak (OpenGLOBE.pl) (grzegorz-og.pl)]

Just to be more precise. Before of Customer Groups Import I had 8 Partner Categories (tags in 7.0). After import I have still 8 Partner Categories but 5 of them are binded to Magento Customer Groups. It is perfect behavior of Import.

[Grzegorz Grzelak (OpenGLOBE.pl) (grzegorz-og.pl)]

This patch fixed this problem. But it was developed by my collegues and I have no comment.

[Grzegorz Grzelak (OpenGLOBE.pl) (grzegorz-og.pl)]

Above patch is probably invalid. It allows to import Partners but these Partners are not connected to Magento Partners. After such import I don see any record in magento_res_partner nor in magento_address.

[Guewen Baconnier @ Camptocamp (gbaconnier-c2c)]

Hi Grzegorz,

I can't reproduce your issue with a standard Magento 1.7 and the last revisions of the connector.

I also tested a simple XML/RPC call against Magento 1.7 with Ruby (using [0]) which show that the nil/None value is correct to return 'all the fields'. (contrarily to the false value which is not used by the connector here.)
>> m.call('customer.info', 1, nil)
=> {"customer_id"=>"1", "created_at"=>"2007-08-30 23:23:13", "updated_at"=>"2008-08-08 12:28:24", "increment_id"=>"000000001", "store_id"=>"1", "website_id"=>"1", "confirmation"=>nil, "created_in"=>nil, "default_billing"=>"274", "default_shipping"=>"274", "disable_auto_group_change"=>"0", "dob"=>nil, "email"=>"<email address hidden>", "firstname"=>"John", "gender"=>nil, "group_id"=>"1", "lastname"=>"Doe", "middlename"=>"", "password_hash"=>"2049484a4020ed15d0e4238db22977d5:eg", "prefix"=>"", "rp_token"=>nil, "rp_token_created_at"=>nil, "suffix"=>"", "taxvat"=>""}
>> m.call('customer.info', 1, false)
=> {"customer_id"=>"1", "updated_at"=>"2008-08-08 12:28:24", "increment_id"=>"000000001"}

This thing questions me about the integrity of the data on your Magento instance unfortunately:
api.call(customer.info, [656, None]) returned {'customer_id': '656', 'updated_at': '2013-02-01 13:20:57', 'increment_id': ''}
As you can see, the 'increment_id' is empty. It should have returned a value, as you can see in my example call above.

Do you have any custom modules on Magento which could alter the normal behavior of the API?
Can you open the customer 656 on the Magento admin panel and see how it looks, if it looks normal?

[0] https://gist.github.com/guewen/1875404

[Guewen Baconnier @ Camptocamp (gbaconnier-c2c)]

Thanks for having proposed a patch anyway, even if I think that for the moment we still have to ensure that the bug does not come from Magento itself.

[Grzegorz Grzelak (OpenGLOBE.pl) (grzegorz-og.pl)]

I checked Magento db under mysql. All records in customer_entity table have increment_id field empty. Some of them have value NULL.

[Guewen Baconnier @ Camptocamp (gbaconnier-c2c)]

Do you have some news regarding these data issues?
I put the bug as 'invalid' since that's a Magento problem.

[Thorsten Vocks (OpenBig.org) (openbig.org)]

We have simulated entry "increment_id"=>"000000001" for the customer 656.
Still we get the same error. We can open the customer 656 on the Magento admin panel and
it looks normal.

The issue still remains: KeyError: 'group_id'

[Guewen Baconnier @ Camptocamp (gbaconnier-c2c)]

What is the content of the 'group_id' in Magento? Does the call returns it?
Does the call to 'customer.info' returns the expected result as seen here:

>> m.call('customer.info', 1, nil)
=> {"customer_id"=>"1", "created_at"=>"2007-08-30 23:23:13", "updated_at"=>"2008-08-08 12:28:24", "increment_id"=>"000000001", "store_id"=>"1", "website_id"=>"1", "confirmation"=>nil, "created_in"=>nil, "default_billing"=>"274", "default_shipping"=>"274", "disable_auto_group_change"=>"0", "dob"=>nil, "email"=>"<email address hidden>", "firstname"=>"John", "gender"=>nil, "group_id"=>"1", "lastname"=>"Doe", "middlename"=>"", "password_hash"=>"2049484a4020ed15d0e4238db22977d5:eg", "prefix"=>"", "rp_token"=>nil, "rp_token_created_at"=>nil, "suffix"=>"", "taxvat"=>""}

The error indicates that Magento did not returned the 'group_id' field which is mandatory.

[Clemens Rambow (openbig.org) (c-rambow)]

Magento does indeed not return the group_id. Thank you very much for your ruby file, which helped us to analyze the problem.

First we tested with our own magento demo-server (magento_demo points to it), which looks good:
irb(main):105:0> magento_demo.call('customer.info', 1, nil)
=> {"customer_id"=>"1", "created_at"=>"2007-08-30 23:23:13", "updated_at"=>"2008-08-08 12:28:24", "increment_id"=>"000000001", "store_id"=>"1", "website_id"=>"1", "confirmation"=>nil, "created_in"=>nil, "default_billing"=>"274", "default_shipping"=>"274", "disable_auto_group_change"=>"0", "dob"=>nil, "email"=>"<email address hidden>", "firstname"=>"John", "gender"=>nil, "group_id"=>"1", "lastname"=>"Doe", "middlename"=>"", "password_hash"=>"2049484a4020ed15d0e4238db22977d5:eg", "prefix"=>"", "rp_token"=>nil, "rp_token_created_at"=>nil, "suffix"=>"", "taxvat"=>""}

Then we made the same call to the customers magento, which resulted in only three fields being returned:
irb(main):108:0> magento_ff.call('customer.info', 1, nil)
=> {"customer_id"=>"1", "updated_at"=>"2012-12-19 15:36:25", "increment_id"=>""}

Interestingly, if we remove the 'nil' (=None) parameter, all the wanted values are being returned (I have removed sensitive data):
irb(main):109:0> magento_ff.call('customer.info', 1)
=> {"customer_id"=>"1", "created_at"=>"REMOVED", "increment_id"=>"", "store_id"=>"1", "website_id"=>"1", "confirmation"=>nil, "created_in"=>"Default Store View", "debit_payment_acount_blz"=>"", "debit_payment_acount_name"=>nil, "debit_payment_acount_number"=>"", "debit_payment_acount_update"=>nil, "default_bin"=>"REMOVED", "failed_logins"=>"0", "firstname"=>"REMOVED", "gender"=>nil, "group_id"=>"1", "lastname"=>"REMOVED", "last_failed_login"=>"0", "last_unlock_time"=>nil, "middlename"=>nil, "password_hash"=>"REMOVED", "prefix"=>"Herr", "rp_token"=>nil, "rp_token_created_at"=>nil, "suffix"=>"", "taxvat"=>nil, "unlock_customer"=>nil}

The patch above (partner_group_id.patch ) basically also implements removing the param if it's None type and allows us to import the customers, as we've verified right now.

[Guewen Baconnier @ Camptocamp (gbaconnier-c2c)]

Okay, that's good to have found a solution. However, I wonder why the behavior on your 'real' magento is like this. Do you have other magento modules which impacts the 'customer.info' call?

I'm a bit worried to apply a patch such as this one, because I don't want to end with an argument checking in all the calls when it should just work with the None value, and I'm not sure that's an issue reproducible anywhere else than on your magento instance.

I'll leave this bug as 'opinion' and see if the same issue happens again for other users.
On your side, you can override this call using your own version of backend, see the documentation: https://bazaar.launchpad.net/~openerp-connector-core-editors/openerp-connector/7.0-magentoerpconnect/view/head:/magentoerpconnect/doc/guides/tutorial_customize.rst

[Andreas Brueckl (andreas-brueckl)]

Hello,

I just encountered the same problem.
At my local Magento 1.7.0.2 instance everything is working fine and "customer.info" returns a bunch of values.
But on another Magento instance (also 1.7.0.2) the call only returns " {'customer_id': '7', 'updated_at': '2013-10-10 11:30:34', 'increment_id': None}"

It seems that the magento installations are not equal, but I do not find out where the difference is ...

I also found out that removing the param "None" would resolve the problem, but this can not be the final solution.
The problem also occurs with other objects like "catalog_category".

Any suggestions are welcome

[Andreas Brueckl (andreas-brueckl)]

I saw the following log in the Magento exception.log. Maybe this is usefull for a magento developer ...

2013-11-12T13:52:20+00:00 ERR (3):
exception 'Zend_XmlRpc_Server_Exception' with message 'Internal Error. Please see log for details.' in /kunden/340904_38446/webseiten/camadeus-mag/app/code/core/Mage/Api/Model/Server/Adapter/Xmlrpc.php:125
Stack trace:
#0 /kunden/340904_38446/webseiten/camadeus-mag/app/code/core/Mage/Api/Model/Server/Handler/Abstract.php(140): Mage_Api_Model_Server_Adapter_Xmlrpc->fault('1', 'Internal Error....')
#1 /kunden/340904_38446/webseiten/camadeus-mag/app/code/core/Mage/Api/Model/Server/Handler/Abstract.php(48): Mage_Api_Model_Server_Handler_Abstract->_fault('internal')
#2 /kunden/340904_38446/webseiten/camadeus-mag/app/code/core/Mage/Catalog/Model/Category/Api.php(168): Mage_Api_Model_Server_Handler_Abstract->handlePhpError(4096, 'Argument 1 pass...', '/kunden/340904_...', 168, Array)
#3 /kunden/340904_38446/webseiten/camadeus-mag/app/code/core/Mage/Catalog/Model/Category/Api.php(159): Mage_Catalog_Model_Category_Api->_nodeToArray(NULL)
#4 [internal function]: Mage_Catalog_Model_Category_Api->tree('', '')
#5 /kunden/340904_38446/webseiten/camadeus-mag/app/code/core/Mage/Api/Model/Server/Handler/Abstract.php(296): call_user_func_array(Array, Array)
#6 [internal function]: Mage_Api_Model_Server_Handler_Abstract->call('14c86dcb9b21efe...', 'catalog_categor...', Array)
#7 /kunden/340904_38446/webseiten/camadeus-mag/lib/Zend/Server/Abstract.php(232): call_user_func_array(Array, Array)
#8 /kunden/340904_38446/webseiten/camadeus-mag/lib/Zend/XmlRpc/Server.php(599): Zend_Server_Abstract->_dispatch(Object(Zend_Server_Method_Definition), Array)
#9 /kunden/340904_38446/webseiten/camadeus-mag/lib/Zend/XmlRpc/Server.php(337): Zend_XmlRpc_Server->_handle(Object(Zend_XmlRpc_Request_Http))
#10 /kunden/340904_38446/webseiten/camadeus-mag/app/code/core/Mage/Api/Model/Server/Adapter/Xmlrpc.php(113): Zend_XmlRpc_Server->handle()
#11 /kunden/340904_38446/webseiten/camadeus-mag/app/code/core/Mage/Api/Model/Server.php(138): Mage_Api_Model_Server_Adapter_Xmlrpc->run()
#12 /kunden/340904_38446/webseiten/camadeus-mag/app/code/core/Mage/Api/controllers/XmlrpcController.php(39): Mage_Api_Model_Server->run()
#13 /kunden/340904_38446/webseiten/camadeus-mag/app/code/core/Mage/Core/Controller/Varien/Action.php(419): Mage_Api_XmlrpcController->indexAction()
#14 /kunden/340904_38446/webseiten/camadeus-mag/app/code/core/Mage/Core/Controller/Varien/Router/Standard.php(250): Mage_Core_Controller_Varien_Action->dispatch('index')
#15 /kunden/340904_38446/webseiten/camadeus-mag/app/code/core/Mage/Core/Controller/Varien/Front.php(176): Mage_Core_Controller_Varien_Router_Standard->match(Object(Mage_Core_Controller_Request_Http))
#16 /kunden/340904_38446/webseiten/camadeus-mag/app/code/core/Mage/Core/Model/App.php(354): Mage_Core_Controller_Varien_Front->dispatch()
#17 /kunden/340904_38446/webseiten/camadeus-mag/app/Mage.php(683): Mage_Core_Model_App->run(Array)
#18 /kunden/340904_38446/webseiten/camadeus-mag/index.php(87): Mage::run('', 'store')
#19 {main}

[Zeeshan Khan (zeeshan-khan)]

Hi

I have also been facing the same problem with Magento 1.8.0.0

I cannot import partners.

Here is the Exception Information:

Traceback (most recent call last):
  File "/opt/openerp/addons/openerp-connector/connector/queue/worker.py", line 122, in run_job
    job.perform(session)
  File "/opt/openerp/addons/openerp-connector/connector/queue/job.py", line 460, in perform
    self.result = self.func(session, *self.args, **self.kwargs)
  File "/opt/openerp/addons/openerp-connector-magento/magentoerpconnect/unit/import_synchronizer.py", line 329, in import_record
    importer.run(magento_id, force=force)
  File "/opt/openerp/addons/openerp-connector-magento/magentoerpconnect/unit/import_synchronizer.py", line 174, in run
    self._import_dependencies()
  File "/opt/openerp/addons/openerp-connector-magento/magentoerpconnect/partner.py", line 272, in _import_dependencies
    if binder.to_openerp(record['group_id']) is None:
KeyError: 'group_id'

[Andreas Brueckl (andreas-brueckl)]

I think that this topic is related to the issue:
http://libjack.com/2009/03/26/java-magento-xmlrpc-api-nil-issue/

[Andreas Brueckl (andreas-brueckl)]

I was able to find out he reason of the problem - its the PHP version.

Magento installations with PHP <= 5.3 run without the mentioned issue.

If PHP version >= 5.4 is used, then the mentioned problem occurs. Its because the XMLRPC library uses the function "call_user_func_array" which has been modified in version 5.4.

Therefore I think the problem should be solved in the Magento OpenERP Connector to ensure that it is also running with PHP >= 5.4.

[Guewen Baconnier @ Camptocamp (gbaconnier-c2c)]

Hi Andreas,

Thanks for the investigation!

So if I understand correctly, this is a Magento error, right? Can we do something about that?

On this page: http://www.magentocommerce.com/system-requirements I see that the PHP compatibility is 5.2.13 - 5.3.24, so it could be just unsupported?

[Andreas Brueckl (andreas-brueckl)]

Hi Guewen,

thanks for your reply.
In that case we do not have to support it.

However I think that I have found I good way to support users with "unsupported" Magento installations. The provided patch removes all trailing "None" values in the arguments of all magento API calls and after some testings it looks good.

Any comments are welcome.

[Javier martin (t-javier)]

Hello.

I had the same problem with a 5.4.23 php version instalation.
I think i applied the patch correctly but it didn't fix the problem.

Can you explain a little more detailled how to apply the patch? I add the lines of the patch with + where proceed on the file backend-adapter.py

Kind regards and thanks.

Javier M.

[Maxime Chambreuil (http://www.savoirfairelinux.com) (max3903)]

Hello,

We applied Andreas' patch and it is working fine for us now. We are using PHP 5.4.23 on CentOS.

[Javier martin (t-javier)]

Hello Maxime.

At first, Happy new year to all.

Is the same php version than me and my Magento version is 1.7.0.2
After aplicate the patch did you do somehting more?

Regards.

[Maxime Chambreuil (http://www.savoirfairelinux.com) (max3903)]

Hello Javier,

We didn't do anything more than applying the patch.

We also use Magento 1.7.0.2.

[Javier martin (t-javier)]

This is my error.

http://pastebin.com/E4pWS0Nt

It comes when i import the orders.

[Maxime Chambreuil (http://www.savoirfairelinux.com) (max3903)]

Did you restart OpenERP after applying the patch ?

[Javier martin (t-javier)]

Yes, and i restart the box too.

[Javier martin (t-javier)]

Magento announce a 5.4 PHP Patch, i hope this fix our problems.

Regards.

P.D: I downgrade to 5.3.28 and problem is fixed.

[Jan-Philipp Fischer (jan-philipp-fischer)]

Note:

Here is the PHP 5.4 Patch for Magento:

http://magento.com/blog/magento-news/magento-now-supports-php-54#.Uud_3bQweUk

[Guewen Baconnier @ Camptocamp (gbaconnier-c2c)]

I set the bug status to Opinion because this is a Magento bug (when installed on an unsupported version of PHP), but I want to keep the report open for people encountering the same issue.

Thanks all for your researches and feedbacks.

[Thomas Klosinsky (tmomas)]

Also with PHP 5.4 patch installed I get this error on new OpenERP Installation without Guewens Patch for partner.py
...

[Landis (larnold)]

How and Where to install the patch to OpenERP? Does it go into one of the Connector folders or into OpenERP root itself?

php_version.patch Edit (860 bytes, text/plain)

I have applied the Magento 5.4 Patch but I still get the error in OpenERP Connector. Next I will see if I can upgrade my full OpenERP and Connector to current.

I tried demoting my Magento to Squeeze (5.3 PHP) but first pass did not work. Seems best if I can stay w/ Wheezy 5.4 somehow.

[Thomas Klosinsky (tmomas)]

Hi Landis,

I just went through all that. Patching OpenERP and downgrading PHP on Wheezy because of some more errors while importing products...
And I can tell it's a little ride ;). But worth it.

I patched manually and found the files mentioned in the patch here:
https://launchpadlibrarian.net/147252239/partner_group_id.patch
in the connector addon directories (mine is /opt/openerp/addons/) openerp-connector-magento/magentoerpconnect/partner.py

Good luck!
Thomas

[Thomas Klosinsky (tmomas)]

and the following patch:
https://launchpadlibrarian.net/156823948/php_version.patch in the same main direction:
openerp-connector-magentomagentoerpconnect/unit/backend_adapter.py

[Landis (larnold)]

Thank you Thomas.
I am working through this now. (first false start breakpoint) Hopefully I can retrograde PHP in Wheezy (could also go to Squeeze again but moving data is more difficult than system installs sometimes. In fact I tried to backup to Squeeze and it didn't work. I could throw some orders away and cross grade to Squeeze again but I would rather not.

Thanks for the guidance on the patches.
I am trying to understand which direcories to put them in. Manual installs?
I think they, IE: partner_group_id.patch
and php_version.patch

 would go into my openerp-connector-magento folder.

The patches I understand will rewrite partner.py and backend_adapter.py

Finally, Why may I ask did you need to install the patches when you downgraded Wheezy to PHP 5.3 ? Isn't that the purpose of the Patches, ie to handle PHP 5.4?

thanks again,

[Landis (larnold)]

What is the command to simply "install a patch"? I have tried several "patch commands" but not found.

I have manually tried to apply them to
backend_adapter.py and
partner.py

backend_adapter.py is giving me an error:
Indentation Error - expected an indented block
                                              (likely not to the errors in partner.py)

(the following is what I added changed, leading and trailing lines left intact)

        try:
# with magentolib.API(self.magento.location,
# self.magento.username,
# self.magento.password) as api:
# result = api.call(method, arguments)
# # Uncomment to record requests/responses in ``recorder``
# # record(method, arguments, result)
# (Added next 12 lines from php_version.patch)
             with magentolib.API(self.magento.location,
                                 self.magento.username,
                                 self.magento.password) as api:

                # If magento is used with PHP version >= 5.4, then we have to remove all
                # trailing None values
                if isinstance(arguments, list) and len(arguments) > 0 and arguments[-1] is None:
                    arguments = [a for a in arguments if a is not None]

                 result = api.call(method, arguments)
                 # Uncomment to record requests/responses in ``recorder``
                 # record(method, arguments, result)
                _logger.debug("api.call(%s, %s) returned %s",

(Thoughts on this>) (Also somehow my "module web is no longer loading - I don't recall it but likely important)

The Trace Error is a bit like this from screenshot:
File "/opt/openerp/source/openerp-connector-magento/magentoerpconnect/unit/backend_adapter.py", line 139
if isinstance(arguments, list) and len(arguments) > 0 and arguments[-1] is None:
A
IndentationError: expected an indented block
2014-04-10 22:17:26,031 INFO HTTP service (werkzeug) running on 0.0.0.0:40069
2014-04-10 22:17:26,031 1450 INFO ? openerp.service.wsgi_server: HTTP service (werkzeug) running on 0.0.0.0:40069
2014-04-10 22:17:26,179 INFO OpenERP server is running, waiting for connections...
2014-04-10 22:17:26,179 1450 INFO ? openerp: OpenERP server is running, waiting for connections...
2014-04-10 22:17:31,431 INFO loading 1 modules...
2014-04-10 22:17:31,431 1450 INFO active openerp.modules.loading: loading 1 modules...
2014-04-10 22:17:33,402 INFO loading 101 modules...
2014-04-10 22:17:33,402 1450 INFO active openerp.modules.loading: loading 101 modules...
2014-04-10 22:17:33,423 CRITICAL Couldn't load module web
2014-04-10 22:17:33,423 1450 active openerp.modules.module: Couldn't load module web
2014-04-10 22:17:33,423 CRITICAL expected an indented block (backend_adapter.py, line 139)
2014-04-10 22:17:33,423 1450 active openerp.modules.module: expected an indented block (backend_adapter.py, line 139)
Exception in thread Thread-1:
Traceback (most recent call last):

[Landis (larnold)]

I just restored backend_adapter.py and the errors went away.
Connector is not working, but the whole thing about "couldn't load module web dissappeared)

Then I saw some auto-syncronization begin (previously I would have to trigger it). Then got the errors ending with:

I think I need to get that Patch In for backend_adapter.py (the partner.py manual patch may have worked. ??)

File "/opt/openerp/source/openerp-connector-magento/magentoerpconnect/unit/backend_adapter.py", line 129, in _call
result = api.call(method, arguments)
File "/usr/local/lib/python2.6/dist-packages/magento-0.4-py2.6.egg/magento/api.py", line 161, in call
return self.client.call(self.session, resource_path, arguments)
File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in call______________
return self. send(self. name, args)
File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in request
ve rbose=self. ve rbose
File "/usr/lib/python2.6/xmlrpclib.py", line 1253, in request
return self._parse_response(h.getfile(), sock)
File "/usr/lib/python2.6/xmlrpclib.py", line 1392, in _parse_response
return u.closeO
File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close
raise Fault(**self._stack[0])
Fault: <Fault 1: 'Internal Error. Please see log for details.'>

[Landis (larnold)]

I will stop posting all these logs here and work it through. I will find how to install patches. (I actually removed both of my manual patches to get here) Something is still not happy (likely 5.4 PHP)_

OpenERP did actually import 3 orders and OpenERP (in client mode is not complaining). Just the Server Console is unhappy.

[Landis (larnold)]

How to Install Patches (difficult to find the simple stuff)

apt-get install bzrtools
then load the patches into the folders they actually work on
bzr patch patchname.patch

Somehow one had "self installed" - not sure how, but I expect this is how I was getting "some data in" before I installed the patches.

Sorry for all the posts and emails.