1600 attributes limitation + performance issue

[Guewen Baconnier @ Camptocamp]

Hello,

As the attributes are managed today (one column per attribute in the product_product table), if the number of Magento's attributes is over 1600, the synchronisation won't work because the number of columns in PostgreSQL is limited to 1600 columns per table.

1600 attributes may seem incredible, but we have a client which is near this limit with ~1400 attributes.

I don't have any solution yet, but I guess that a reflexion should be done.

Thank you
Guewen

[Guewen Baconnier @ Camptocamp (gbaconnier-c2c)]

Hello,

I just would like to mention that when so many attributes are used (more than a thousand), the loading of products becomes outrageously slow, it takes about 1 minute to open a product list (it's around 1 second per product). It takes 2-3 seconds to open a product with magento informations.

We'll search if we could do some modifications to speed up that without modifying the core of the connector, but it's pretty sure that if the attributes management is modified to avoid the "1600 attributes bug", the speed will come back.

Guewen

[Raphaël Valyi - http://www.akretion.com (rvalyi)]

Hello Guewen,

Actually I think there is a design issue in the connector (but it's like this because OpenObject v5 had implementation issues). Let me explain: as long as your don't have tons of attributes, the current solution works.

Now, if your Magento has dozens of attribute sets with each dozens/hundreds of attributes, all attributes will be created in the product table in OpenERP and indeed that's an issue as you notice here.

Actually, instead of wrapping all Magento attributes inside the product_product OpenERP table, we whould have used the '_inherits' OpenERP features: every Magento attribute set likess 'shoes' would define a new OpenERP object with its onw table like 'product_shoes'. This table would hold the 'shoes' attributes and inherits from the OpenERP product_product object/table.
Shoes attributes would be read only when required and the rest of the time, only standard common attributes would be fetched.

We aren't stupid and though about doing that since the beginning with Sharoon Thomas. Sharoon even tried by mid 2009, but OpenERP v5 would support it at the ORM level: there were 2 levels of inherits (product_shoes inherits product_product inherits product_template) and multiple dynamically defined inheritances. First tests have shown OpenERP would blow out in many bugs, so we abandoned the idea.

However, it would be interesting to know if that's doable in OpenERP v6, may be as the general quality of the ORM has been improved. Supposing OpenObject would support it, that's a moderately difficult development. IMHO the biggest issue would be to migrate between one connector version to an other.

I don't know if you can think about work around for your specific case, but in the absolute I think what I describe would be the right solution.

Thoughts?

[Sharoon Thomas http://openlabs.co.in (sharoonthomas)]

Hello All,

As Raphael said, we had tried but later dropped the idea of _inherits. I am not sure if even v6 supports _inherits on an _inherits table (product.product). Also there were several other bugs because of which we abandoned the idea. We were very much aware of this limitation but 1600 attributes is a lot for a magento implementation.

Probably, the best would be to follow EAV and then do the presentation magic in the views. Of course this would break backward compatibility. But, it would be easier to have migration scripts which support this transition than an _inherits transition.

@Raphael: Views please

[Guewen Baconnier @ Camptocamp (gbaconnier-c2c)]

Hello,

Thank you for your responses.

I understand the problematic.

As the performance problem is urgent for us, we would like to found a solution which doesn't require a too big modification of the connector like redesign the attributes in an EAV way.

Our idea to enhance the speed is to overload the read method of product.product.
In that overloaded method, we get all the fields excluding the x_magerp.
We add a context "attribute set" which, when given, will load all the fields of this set.
Then, on the attributes sets menus, we pass their set in the context.

This won't take off the 1600 attributes limitations, but we suppose (and hope) that will improve the speed loading of the products as only the necessary attributes will be loaded.

What do you think about this suggestion ?

[ag2888 (q-contact-gdaudit-com)]

https://bugs.launchpad.net/magentoerpconnect/+bug/675013

Sincerely

[Sébastien BEAU - http://www.akretion.com (sebastien.beau)]

FIx in trunk version. You can already try it, please use this version of the server https://code.launchpad.net/~akretion-team/openobject-server/sparse_field