Import / Export to review in Trunk

[Fabien (Open ERP)]

Hello,

The import/export tool must be reviewed completly in trunk in order to be easier to use. The main change is the "import compatible" option that allows to export any kind of data (including one2many, many2one, etc) and reimport the file after to update or create new data. The v5 behaviour was unuseable at all.

I already commited the changes in the server and GTK client to support this. Only the web client is remaining.

Have a look at the attached PDF file for the useability of the two screens to review.

Import Tool
-----------

The import tool must be simplified. We don't need to select the fields anymore, it will automatically take the columns defined by the first row of the .CSV file.

When you select a file, the web client should make a preview of the first three rows. You can change the CSV options "field separator, field delimitor, encoding". Just display the columns as it, without any interpretation. (if they are ID or XML ID, just display the IDs).

Export Tool
-----------

There is a real difference between the normal mode and the "Import Compatible" mode.

If you set the import compatible mode:
1. For every field:
  - many2one: just display two children: id and .id
  - many2many: just display this field without children
  - one2many: normal behaviour
2. When you export, put the name of the fields in the first columns, and not their label. (example: partner_id/id)

If you don't set the import compatible mode. For every:
1. For every field:
  - many2one: you can see all children
  - many2many: you can see all children
  - one2many: you can see all children
2. When you export, put the label of the fields in the first columns, and not their name. (example: Partner/ID)

I changed the format of the csv file. Instead of "db_id", I used ".id". An example of "import compatible" file exported from partners:

  id,.id,name,address.id,address/id,address/name
  base.res_partner_agrolait,3,Agrolait,11,base.res_partner_address_8delivery,Paul

This has to be done in trunk, before v6.0. I think it's critical for OpenERP Online users.

Test with the GTK client to be sure the behaviour is the same in the web and GTK client.

[Ferdinand (office-chricar)]

I wouild realy appreciate to have a default export list definition for all fields and a bit of intelligence choosing this definition excluding fields not available

see attachment
it is not userfriendly and most users will have problems and loos much time to choose the correct fields to ask users to create a default export list
it's much easier to remove some unwanted fields or to drop a column in the spreadsheet

[Raphaël Valyi - http://www.akretion.com (rvalyi)]

Guys,

I add that if you could make the 'module' (the referential actually)
acceptable in the import context or specifiable in a field somewhere that
would be very good, it would make it very integrated with the
base_external_referentials module: so one could specify in which referential
(eg module) the "XML" ids he is providing are to be considered. This is
terribly useful when OpenERP is connected to other systems: like external
ecommerce, external accounting tool, external CRM and all: each OpenERP
business entity can hence have different ids each referential
and base_external_referentials makes this straightfoward and even allows to
define field mapping operations right inside OpenERP.

On Wed, Dec 22, 2010 at 8:50 AM, Ferdinand @ Camptocamp <
<email address hidden>> wrote:

> I wouild realy appreciate to have a default export list definition for
> all fields and a bit of intelligence choosing this definition excluding
> fields not available
>
> see attachment
> it is not userfriendly and most users will have problems and loos much
> time to choose the correct fields to ask users to create a default export
> list
> it's much easier to remove some unwanted fields or to drop a column in the
> spreadsheet
>
>
> ** Attachment added: "chricar_tools_export.zip"
>
> https://bugs.launchpad.net/openobject-client-web/+bug/693308/+attachment/1773432/+files/chricar_tools_export.zip
>
> --
> You received this bug notification because you are a member of OpenERP
> Framework Experts, which is subscribed to OpenObject Web Client.
> https://bugs.launchpad.net/bugs/693308
>
> Title:
> Import / Export to review in Trunk
>
> Status in OpenObject Web Client:
> Confirmed
>
> Bug description:
> Hello,
>
> The import/export tool must be reviewed completly in trunk in order to be
> easier to use. The main change is the "import compatible" option that allows
> to export any kind of data (including one2many, many2one, etc) and reimport
> the file after to update or create new data. The v5 behaviour was unuseable
> at all.
>
> I already commited the changes in the server and GTK client to support
> this. Only the web client is remaining.
>
> Have a look at the attached PDF file for the useability of the two screens
> to review.
>
> Import Tool
> -----------
>
> The import tool must be simplified. We don't need to select the fields
> anymore, it will automatically take the columns defined by the first row of
> the .CSV file.
>
> When you select a file, the web client should make a preview of the first
> three rows. You can change the CSV options "field separator, field
> delimitor, encoding". Just display the columns as it, without any
> interpretation. (if they are ID or XML ID, just display the IDs).
>
> Export Tool
> -----------
>
> There is a real difference between the normal mode and the "Import
> Compatible" mode.
>
> If you set the import compatible mode:
> 1. For every field:
> - many2one: just display two children: id and .id
> - many2many: just display this field without children
> - one2many: normal behaviour
> 2. When you export, put the name of the fields in the first columns, and
> not their label. (example: partner_id/id)
>
> If you don...

[xrg (xrg)]

On Wednesday 22 December 2010, you wrote:
> Guys,
>
> I add that if you could make the 'module' (the referential actually)
> acceptable in the import context or specifiable in a field somewhere that
> would be very good, it would make it very integrated with the
> base_external_referentials module: ...

I would vote for this idea, to have an overal review of impor/export
methodologies[1] for OpenERP, after v6.0 [2].
Remember the Gnucash module? that was my early take on the subject, a few
years ago. Then, I realized that the mechanism should be central to OpenERP,
just like the base_external_referentials now.

[1] not rewrite them, but evaluate the ones we have and perhaps enhance.
[2] hopefully, v6.1 will be a smooth continuation of v6.0, meaning that those
early adopters of 6.0 will be able to use v6.1 features once they are
implemented, without further migration pain.

[Fabien (Open ERP) (fp-tinyerp)]

@Raphaël Valyi
The import / export we made support external referancials (through xml id, accepting a module name)

Note that this bug report is just for the UI of the import / export that has already been implemented in trunk GTK: http://www.youtube.com/watch?v=e1LbaiSa1aM

[Raphaël Valyi - http://www.akretion.com (rvalyi)]

Fabien,

for the import I believe it's ok you can specify the referential indeed.

Now, I believe for the export you can still no choose a specific
referential/module, indeed, here is how the get_xml_id method is implemented
in server/bin/orm.py:

    def get_xml_id(self, cr, uid, ids, *args, **kwargs):
        """Find out the XML ID of any database record, if there
        is one. This method works as a possible implementation
        for a function field, to be able to add it to any
        model object easily, referencing it as ``osv.osv.get_xml_id``.

        **Synopsis**: ``get_xml_id(cr, uid, ids) -> { 'id': 'module.xml_id'
}``

        :return: the fully qualified XML ID of the given object,
                 defaulting to an empty string when there's none
                 (to be usable as a function field).
        """
        result = dict.fromkeys(ids, '')
        model_data_obj = self.pool.get('ir.model.data')
        data_ids = model_data_obj.search(cr, uid,
                [('model', '=', self._name), ('res_id', 'in', ids)])
        data_results = model_data_obj.read(cr, uid, data_ids,
                ['name', 'module', 'res_id'])
        for record in data_results:
            result[record['res_id']] = '%(module)s.%(name)s' % record
        return result

I think, that in the case your record has several ids in different
referentials (a common case), then you have several data_results for the
same result[record['res_id']], no?
But in that case, only the last record in data_results will be returned or I
missed something?

IMHO, even if it's not implemented in the clients, in order to maximize the
future server API stability, I think it would be nice this get_xml_id method
accepts a 'module' argument in its context.
If present, the given module will restrict the search in data_ids =
model_data_obj.search(cr, uid, [('model', '=', self._name), ('res_id', 'in',
ids)])

Then it will only be about adding the feature in the clients, migration will
be smooth. I believe this has added value.
On a side note: I personally don't really like calling this id "xml_id", I
mean XML is just a format you used to feed those ids, could also have been
YAML, webservices, whatever.
Not sure what would be more appropriate, may be something like module_ref_id
? Not sure. In any case naming is very important for the success of a
framework/API.

Did I miss something?
BTW, thanks for the video, will definitely help lots of folks getting
started to OpenERP import/export.

On Wed, Dec 29, 2010 at 3:29 PM, Fabien (Open ERP) <email address hidden> wrote:

> @Raphaël Valyi
> The import / export we made support external referancials (through xml id,
> accepting a module name)
>
> Note that this bug report is just for the UI of the import / export that
> has already been implemented in trunk GTK:
> http://www.youtube.com/watch?v=e1LbaiSa1aM
>
> --
> You received this bug notification because you are a member of OpenERP
> Framework Experts, which is subscribed to OpenObject Web Client.
> https://bugs.launchpad.net/bugs/693308
>
> Title:
> Import / Export to review in Trunk
>
> Status in OpenObject Web Client:
> In Progress
>
> Bug description:
> Hello,
>
> The imp...

[Raphaël Valyi - http://www.akretion.com (rvalyi)]

on a side note, just giving an example of serious "multi-referential".
Simples cases are: you have your partners in OpenERP but also in an
accounting system and in some ecommerce: already 3 referentials.
We now have a US/Brazil customer planning to connect 25 different Magento
installations on the same OpenERP v6, that's really multi-referentials, and
it's important to support it...

2010/12/29 Raphaël Valyi <email address hidden>

> Fabien,
>
> for the import I believe it's ok you can specify the referential indeed.
>
> Now, I believe for the export you can still no choose a specific
> referential/module, indeed, here is how the get_xml_id method is implemented
> in server/bin/orm.py:
>
> def get_xml_id(self, cr, uid, ids, *args, **kwargs):
> """Find out the XML ID of any database record, if there
> is one. This method works as a possible implementation
> for a function field, to be able to add it to any
> model object easily, referencing it as ``osv.osv.get_xml_id``.
>
> **Synopsis**: ``get_xml_id(cr, uid, ids) -> { 'id': 'module.xml_id'
> }``
>
> :return: the fully qualified XML ID of the given object,
> defaulting to an empty string when there's none
> (to be usable as a function field).
> """
> result = dict.fromkeys(ids, '')
> model_data_obj = self.pool.get('ir.model.data')
> data_ids = model_data_obj.search(cr, uid,
> [('model', '=', self._name), ('res_id', 'in', ids)])
> data_results = model_data_obj.read(cr, uid, data_ids,
> ['name', 'module', 'res_id'])
> for record in data_results:
> result[record['res_id']] = '%(module)s.%(name)s' % record
> return result
>
>
>
> I think, that in the case your record has several ids in different
> referentials (a common case), then you have several data_results for the
> same result[record['res_id']], no?
> But in that case, only the last record in data_results will be returned or
> I missed something?
>
> IMHO, even if it's not implemented in the clients, in order to maximize the
> future server API stability, I think it would be nice this get_xml_id method
> accepts a 'module' argument in its context.
> If present, the given module will restrict the search in data_ids =
> model_data_obj.search(cr, uid, [('model', '=', self._name), ('res_id', 'in',
> ids)])
>
> Then it will only be about adding the feature in the clients, migration
> will be smooth. I believe this has added value.
> On a side note: I personally don't really like calling this id "xml_id", I
> mean XML is just a format you used to feed those ids, could also have been
> YAML, webservices, whatever.
> Not sure what would be more appropriate, may be something like
> module_ref_id ? Not sure. In any case naming is very important for the
> success of a framework/API.
>
> Did I miss something?
> BTW, thanks for the video, will definitely help lots of folks getting
> started to OpenERP import/export.
>
>
> On Wed, Dec 29, 2010 at 3:29 PM, Fabien (Open ERP) <email address hidden> wrote:
>
>> @Raphaël Valyi
>> The import / export we made support external referancials...

[Kunal Chavda (kunal-chavda)]

Hello,

We have fixed the problem in lp:~openerp-dev/openobject-client-web/import_export
It will be merged soon in trunk (lp:openobject-client-web).
Revision Info: 4243 <email address hidden>

Thanks.

[Fabien (Open ERP) (fp-tinyerp)]

Feedback:

Export:
* The default value in the selection box must be "Import Compatible Export" instead of "Export all Data"
* The load/save feature has a very bad useability. Especially when they are pre-defined export fields.

Import:
The useability is wrong too. The "CSV Options" must be part the the section "2. ". This section can always be visible (instead of only appearing when you have selected the file).

The CSV options must be foldable, like in the given mockup:
https://bugs.launchpad.net/openobject-client-web/+bug/693308/+attachment/1773378/+files/import_export.pdf

We should be able to load a wrong file (like the one in attachment that has ";" as separator instead of ","), change the CSV options and recompute the preview table.

[Ronald Portier (Therp) (rportier1962)]

I would like to add another requirement to make import really usefull:

Now data can only be imported, basically in an empty table, or a table that has no overlap whatsoever with the data to be importerd regadring unique keys.

This makes it really difficult to use the import function to synchronize changes from other databases systems.

In my opinion there should be an option on the import function to:
EITHER throw an error when an row in the csv is encountered that is already in the database (existing functionality),
OR to ignore duplicates altogether (that is only add rows when their keys are not already in the database),
OR to update existing rows (having the same keys as in the csv) with the new values,
OR to use the import as a complete replace of existing data.

Kind regards,