[6.0] regression: inheriting ir_model_data causes data from 'base' module to be discarded

[Raphaël Valyi - http://www.akretion.com]

Hello,

sorry to shout that one loudly, but I know that v6 "stable" is about to be release, so I want to make sure you take a look to it:

A change introduced recently in the server (after rev #3224 I think), will now destroy important database records in the update process if you have any object inheriting the ir.model.data object.

Simple code like this one will trigger the bug, it's attached as a "kaboom" module so you can test:

class ir_model_data(osv.osv):
    _inherit = "ir.model.data"

ir_model_data()

After loading such code, if you restart the server with option --update=all --database=your_base,
your database will be broken, for instance all your menu will be broken at the next login. You can also see that in the console with logs such as:
[2011-01-17 15:24:43,983][magento] INFO:addons.base.ir.model.data:Deleting <email address hidden>act_window
[2011-01-17 15:24:43,991][magento] INFO:addons.base.ir.model.data:Deleting <email address hidden>
[2011-01-17 15:24:43,997][magento] INFO:addons.base.ir.model.data:Deleting <email address hidden>
[2011-01-17 15:24:44,004][magento] INFO:addons.base.ir.model.data:Deleting <email address hidden>
[2011-01-17 15:24:44,009][magento] INFO:addons.base.ir.model.data:Deleting <email address hidden>
[2011-01-17 15:24:44,015][magento] INFO:addons.base.ir.model.data:Deleting <email address hidden>
[2011-01-17 15:24:44,022][magento] INFO:addons.base.ir.model.data:Deleting <email address hidden>
[2011-01-17 15:24:44,028][magento] INFO:addons.base.ir.model.data:Deleting <email address hidden>
...

An important consequence of that bug is that the magentoerpconnect certified module is now broken:
https://bugs.launchpad.net/magentoerpconnect/+bug/701644
We really need to override or_model_data in the base_external_referentials module, if you don't fix what triggers this behavior in the server, it would be pretty hard to make base_external_referentials and dependent connectors work again without patch...

[Raphaël Valyi - http://www.akretion.com (rvalyi)]

A few comments:

The issue occurs because in bin/addons/base/ir/ir_model.py,
the self.loads collection doesn't contain everything it should.
Here is the instrumentation of the code I'm using:

@@ -723,9 +725,22 @@
         module_in = ",".join(["%s"] * len(modules))
         cr.execute('select id,name,model,res_id,module from ir_model_data where module IN (' + module_in + ') and noupdate=%s', modules + [False])
         wkf_todo = []
+# print "*********** loads", self.loads
+ for item in self.loads:
+ if (item[0] == unicode('base') or item[0] == 'base'):
+ print "**** in loads:", item
+
         for (id, name, model, res_id,module) in cr.fetchall():
             if (module,name) not in self.loads:
- self.unlink_mark[(model,res_id)] = id
+# self.unlink_mark[(model,res_id)] = id
+# print "module", module
+ if module != 'base':
+ #print "****", (module,name)
+ self.unlink_mark[(model,res_id)] = id
+ else:
+ self.unlink_mark[(model,res_id)] = id
+ print "not in loads: :(module,name)", (module,name)

And then it will print (with kaboom installed):
**** in loads: ('base', 'user_root')
not in loads: :(module,name) (u'base', u'menu_administration_shortcut')
not in loads: :(module,name) (u'base', u'menu_custom')
not in loads: :(module,name) (u'base', u'next_id_4')
not in loads: :(module,name) (u'base', u'menu_low_workflow')
not in loads: :(module,name) (u'base', u'menu_custom_action')
not in loads: :(module,name) (u'base', u'menu_config')
not in loads: :(module,name) (u'base', u'menu_translation')
not in loads: :(module,name) (u'base', u'menu_translation_app')
not in loads: :(module,name) (u'base', u'menu_translation_export')
not in loads: :(module,name) (u'base', u'menu_users')
not in loads: :(module,name) (u'base', u'menu_security')
not in loads: :(module,name) (u'base', u'menu_management')
not in loads: :(module,name) (u'base', u'reporting_menu')
...

Whereas, without the 'kaboom' module installed, there are dozens of entries in the self.loads collection instead:
**** in loads: ('base', 'values_view_form_action')
**** in loads: ('base', 'bw')
**** in loads: ('base', 'JPY')
**** in loads: ('base', 'ratePLN')
**** in loads: ('base', 'CRC')
**** in loads: ('base', u'access_res_widget')
**** in loads: ('base', 'res_partner_event-wopartner-view_tree')
**** in loads: ('base', 'action_view_base_module_upgrade_window')
**** in loads: ('base', 'config_wizard_simple_view')
**** in loads: ('base', 'action_res_users')
**** in loads: ('base', 'view_translation_form')
**** in loads: ('base', u'access_res_widget_user_group_user')
**** in loads: ('base', 'ir_cron_act')
**** in loads: ('base', 'vi')
**** in loads: ('base', u'access_res_log_all')
**** in loads: ('base', 'RON')
**** in loads: ('base', 'sa')
and nothing that is not in self.loads.

I've still not been able to track why this self.loads is empty at update with kaboom installed.

Other remark: if you flag the module kaboom as 'uninstalled' using SQL, your update seems able to restore the menus but this is no long ter...

[Olivier Dony (Odoo) (odo-openerp)]

Raphael, thanks for investigating in details this issue and providing a reproducible testcase.
It makes it easier to fix the issue! :-)

Basically, inheriting from ir.model.data causes the osv instance for 'ir.model.data' in the pool to be replaced, discarding the current list of known "XML ids", stored in ir.model.data.loads.
Recently we changed the module data cleanup to be done globally at the end of the module install/update process, to avoid other issues, and thus it became important for this 'loads' cache of known XML ids to survive replacement of the osv instance in the pool.

The fix places a reference to this cache in an attribute of the pool object, to have it survive properly. It was pushed in
server trunk with revision revision-id: <email address hidden>

Note that the amount of module data being discarded by this bug depends on the number of modules that were processed before the base_external_referentials modules, which should be quite low, as it only depends on base, and is processed before any module with deeper dependencies. Also note that after updating to the latest revision and starting another --update=all, any missing module data will be properly restored, as long as discarded data only included actual module data, and no data that 'pretended' to be part of a module.

Thanks for the bug report!

[Raphaël Valyi - http://www.akretion.com (rvalyi)]

Hello,

I server confirm rev #3288 fixes the bug, many thanks to Olivier and OpenERP SA for that.