importing a bad csv file in module, and subsequent correction causes issues

[Graeme Gellatly]

There are a few strands to this. Some I can't confirm except to say they occur on my machine. I'm not even sure what the bug is I'm reporting.

1. If the relative path of a symlink includes a hyphen then I think the import doesn't work. It chucks a no reference error in the client. On checking, this is because there are no entries in ir.model.data when _get_id is called, the code excepts, silently passes and continues. Often this means the error at the end is in no way related to the actual error, as it excepts all the way through, but doesn't raise immediately so the exception gets overwritten. I think the csv importer deals with it fine, so when debugging you see the right lines in the right place. I think the issue might be where ir.model.data gets populated. I seem to remember that xml has an issue with hyphens from when installing the bzr-xmloutput plugin for bazaar where you need to rename the directory with an underscore.

2. Upon doing a number of things, like renaming the directory, reformatting all csv's so everything was stored as strings in double quotes, it still failed on an existing database (updated) but would work fine on a fresh database. Somehow it exhibits the same behaviour where ir.model.data search returns an empty list. So maybe some entry is left over. Deleting the module entry in GUI, and then restarting the server and updating the modules it worked.

3. This behaviour is not limited to the hyphen problem. I had an entry called Travel, Meals and Entertainment in the chart, it was not enclosed in quotes. This brought about a quite correct error of no reference to parent xxx.Meals and Entertainment. However after correcting the csv's it starts to exhibit the above error.

4. There is additional behaviour that is exhibited sometimes, if the file is corrected without following the above steps. And that is that it will just not import anything at all, and start any configuration actions.

I have experienced on latest 6.0.2 sources (as of yesterday) on Linux, and have user reports confirmed on Windows as well. The best way to see this behaviour is to set a breakpoint at line 733 or 735 (or the exception line) in the ORM within _get_id function and step over, while using a bad csv file or directory.

I'm not concerned that it kicks up a fuss about bad csv files and stupid developer symlinks and chucks an error, but that it is so difficult to recover after correcting - and I think the bug is more related to ir.model.data than csv import. Maybe just throw an exception if a hyphen is in the path.

While I highly do not recommend this, if you want files to test, good files is latest rev of lp:~gdgellatly/openerp-au/l10n-nz-minimal-coa, bad files is rev 3 of the same. Just do it on something you don't mind breaking (a fresh database is fine).

[Graeme Gellatly (gdgellatly)]

I just went off to work on something else, and I think I may have found the
root cause. Unlike xml, csv is effectively always noupdate=1. So I geuss
what I'm saying is that perhaps, in the case of a module csv import that
this is only set to true if the import completes successfully.

On Tue, May 24, 2011 at 10:54 AM, Graeme Gellatly <<email address hidden>
> wrote:

> Public bug reported:
>
> There are a few strands to this. Some I can't confirm except to say
> they occur on my machine. I'm not even sure what the bug is I'm
> reporting.
>
> 1. If the relative path of a symlink includes a hyphen then I think the
> import doesn't work. It chucks a no reference error in the client. On
> checking, this is because there are no entries in ir.model.data when
> _get_id is called, the code excepts, silently passes and continues.
> Often this means the error at the end is in no way related to the actual
> error, as it excepts all the way through, but doesn't raise immediately
> so the exception gets overwritten. I think the csv importer deals with
> it fine, so when debugging you see the right lines in the right place.
> I think the issue might be where ir.model.data gets populated. I seem
> to remember that xml has an issue with hyphens from when installing the
> bzr-xmloutput plugin for bazaar where you need to rename the directory
> with an underscore.
>
> 2. Upon doing a number of things, like renaming the directory,
> reformatting all csv's so everything was stored as strings in double
> quotes, it still failed on an existing database (updated) but would work
> fine on a fresh database. Somehow it exhibits the same behaviour where
> ir.model.data search returns an empty list. So maybe some entry is left
> over. Deleting the module entry in GUI, and then restarting the server
> and updating the modules it worked.
>
> 3. This behaviour is not limited to the hyphen problem. I had an entry
> called Travel, Meals and Entertainment in the chart, it was not enclosed
> in quotes. This brought about a quite correct error of no reference to
> parent xxx.Meals and Entertainment. However after correcting the csv's
> it starts to exhibit the above error.
>
> 4. There is additional behaviour that is exhibited sometimes, if the
> file is corrected without following the above steps. And that is that
> it will just not import anything at all, and start any configuration
> actions.
>
> I have experienced on latest 6.0.2 sources (as of yesterday) on Linux,
> and have user reports confirmed on Windows as well. The best way to see
> this behaviour is to set a breakpoint at line 733 or 735 (or the
> exception line) in the ORM within _get_id function and step over, while
> using a bad csv file or directory.
>
> I'm not concerned that it kicks up a fuss about bad csv files and stupid
> developer symlinks and chucks an error, but that it is so difficult to
> recover after correcting - and I think the bug is more related to
> ir.model.data than csv import. Maybe just throw an exception if a
> hyphen is in the path.
>
> While I highly do not recommend this, if you want files to test, good
> files is latest rev of lp:~gdgellatly/openerp-au/l10n-nz-minima...

[Vinay Rana (OpenERP) (vra-openerp)]

Hello Graeme Gellatly,

I have found one minor problem in your chart of account module. due to that problem I am not able to install your module. The following patch will solve the problem:

=== modified file 'l10n_nz/data/account.tax.template.csv'
--- l10n_nz/data/account.tax.template.csv 2011-05-23 05:45:48 +0000
+++ l10n_nz/data/account.tax.template.csv 2011-06-03 09:34:04 +0000
@@ -1,9 +1,9 @@
 "id","description","chart_template_id:id","type_tax_use","name","type","amount","account_collected_id:id","account_paid_id:id","base_code_id:id","tax_code_id:id","ref_base_code_id:id","ref_tax_code_id:id","ref_base_sign","ref_tax_sign","parent_id:id","python_compute"
-"ST1","ST1","l10n_nz_ext","sale","Standard rate sales (15.0%)","code","0.15","21310","21310","5","1","5","1","-1","-1",,"result = price_unit * 0.15"
-"PT1","PT1","l10n_nz_ext","purchase","Standard rate purchases (15.0%)","code","0.15","21320","21320","6","3","6","3","-1","-1",,"result = price_unit * 0.15"
-"ST0","ST0","l10n_nz_ext","sale","Zero rated sales","percent","0","21310","21310","5","1","5","1","-1","-1",,
-"ST2","ST2","l10n_nz_ext","sale","Exempt sales","percent","0","21310","21310","5","1","5","1","-1","-1",,
-"PT0","PT0","l10n_nz_ext","purchase","Zero rated purchases","percent","0","21320","21320","6","3","6","3","-1","-1",,
-"PT2","PT2","l10n_nz_ext","purchase","Exempt purchases","percent","0","21320","21320","6","3","6","3","-1","-1",,
-"ST11","ST11","l10n_nz_ext","sale","Old rate sales (12.5%)","percent","0.125","21310","21310","5","1","5","1","-1","-1",,
-"PT11","PT11","l10n_nz_ext","purchase","Old rate purchases (12.5%)","percent","0.125","21320","21320","6","3","6","3","-1","-1",,
+"ST1","ST1","l10n_nz_std","sale","Standard rate sales (15.0%)","code","0.15","21310","21310","5","1","5","1","-1","-1",,"result = price_unit * 0.15"
+"PT1","PT1","l10n_nz_std","purchase","Standard rate purchases (15.0%)","code","0.15","21320","21320","6","3","6","3","-1","-1",,"result = price_unit * 0.15"
+"ST0","ST0","l10n_nz_std","sale","Zero rated sales","percent","0","21310","21310","5","1","5","1","-1","-1",,
+"ST2","ST2","l10n_nz_std","sale","Exempt sales","percent","0","21310","21310","5","1","5","1","-1","-1",,
+"PT0","PT0","l10n_nz_std","purchase","Zero rated purchases","percent","0","21320","21320","6","3","6","3","-1","-1",,
+"PT2","PT2","l10n_nz_std","purchase","Exempt purchases","percent","0","21320","21320","6","3","6","3","-1","-1",,
+"ST11","ST11","l10n_nz_std","sale","Old rate sales (12.5%)","percent","0.125","21310","21310","5","1","5","1","-1","-1",,
+"PT11","PT11","l10n_nz_std","purchase","Old rate purchases (12.5%)","percent","0.125","21320","21320","6","3","6","3","-1","-1",,

Thanks.

[Graeme Gellatly (gdgellatly)]

Thanks for pointing out. I've pushed a new revision

On Fri, Jun 3, 2011 at 9:41 PM, Vinay Rana (openerp) <
<email address hidden>> wrote:

> Hello Graeme Gellatly,
>
> I have found one minor problem in your chart of account module. due to
> that problem I am not able to install your module. The following patch
> will solve the problem:
>
> === modified file 'l10n_nz/data/account.tax.template.csv'
> --- l10n_nz/data/account.tax.template.csv 2011-05-23 05:45:48 +0000
> +++ l10n_nz/data/account.tax.template.csv 2011-06-03 09:34:04 +0000
> @@ -1,9 +1,9 @@
>
> "id","description","chart_template_id:id","type_tax_use","name","type","amount","account_collected_id:id","account_paid_id:id","base_code_id:id","tax_code_id:id","ref_base_code_id:id","ref_tax_code_id:id","ref_base_sign","ref_tax_sign","parent_id:id","python_compute"
> -"ST1","ST1","l10n_nz_ext","sale","Standard rate sales
> (15.0%)","code","0.15","21310","21310","5","1","5","1","-1","-1",,"result =
> price_unit * 0.15"
> -"PT1","PT1","l10n_nz_ext","purchase","Standard rate purchases
> (15.0%)","code","0.15","21320","21320","6","3","6","3","-1","-1",,"result =
> price_unit * 0.15"
> -"ST0","ST0","l10n_nz_ext","sale","Zero rated
> sales","percent","0","21310","21310","5","1","5","1","-1","-1",,
> -"ST2","ST2","l10n_nz_ext","sale","Exempt
> sales","percent","0","21310","21310","5","1","5","1","-1","-1",,
> -"PT0","PT0","l10n_nz_ext","purchase","Zero rated
> purchases","percent","0","21320","21320","6","3","6","3","-1","-1",,
> -"PT2","PT2","l10n_nz_ext","purchase","Exempt
> purchases","percent","0","21320","21320","6","3","6","3","-1","-1",,
> -"ST11","ST11","l10n_nz_ext","sale","Old rate sales
> (12.5%)","percent","0.125","21310","21310","5","1","5","1","-1","-1",,
> -"PT11","PT11","l10n_nz_ext","purchase","Old rate purchases
> (12.5%)","percent","0.125","21320","21320","6","3","6","3","-1","-1",,
> +"ST1","ST1","l10n_nz_std","sale","Standard rate sales
> (15.0%)","code","0.15","21310","21310","5","1","5","1","-1","-1",,"result =
> price_unit * 0.15"
> +"PT1","PT1","l10n_nz_std","purchase","Standard rate purchases
> (15.0%)","code","0.15","21320","21320","6","3","6","3","-1","-1",,"result =
> price_unit * 0.15"
> +"ST0","ST0","l10n_nz_std","sale","Zero rated
> sales","percent","0","21310","21310","5","1","5","1","-1","-1",,
> +"ST2","ST2","l10n_nz_std","sale","Exempt
> sales","percent","0","21310","21310","5","1","5","1","-1","-1",,
> +"PT0","PT0","l10n_nz_std","purchase","Zero rated
> purchases","percent","0","21320","21320","6","3","6","3","-1","-1",,
> +"PT2","PT2","l10n_nz_std","purchase","Exempt
> purchases","percent","0","21320","21320","6","3","6","3","-1","-1",,
> +"ST11","ST11","l10n_nz_std","sale","Old rate sales
> (12.5%)","percent","0.125","21310","21310","5","1","5","1","-1","-1",,
> +"PT11","PT11","l10n_nz_std","purchase","Old rate purchases
> (12.5%)","percent","0.125","21320","21320","6","3","6","3","-1","-1",,
>
>
> Thanks.
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/787256
>
> Title:
> importing a bad csv file in module, and subsequent correction causes
> issues
>
> Status...

[Olivier Dony (Odoo) (odo-openerp)]

Hi Graeme,

I think you analysis with the noupdate flag of csv imports is correct.
(Note that there is a legacy difference between CSV files that are in the 'update_xml' section of the __openerp__ vs those in the init_xml/data sections. The former are in noupdate=False mode, the latter are in noupdate=True mode).

However, AFAIK a failure during the parsing of any CSV files should rollback the whole module update/install, just like for Python or XML content. There was an old issue with CSVs import before bug 672669, but that should be fine if you're on a recent version.

So the behavior should be: if a CSV fails during import, nothing gets commited in the database, and the module is not installed at all. So the broken/partial data should never be in the database, and there is nothing to patch. If a CSV fails during update, it should also be rolled back, but it should make no difference anyway for CSV (noupdate) data, which would not have been updated anyway.
Can you confirm that you see this behavior too?

If not, then perhaps something fishy is going on with an internal cache, or something similar, causing trouble during the next update.

[Amit Parik (amit-parik)]

Hello,

Would you please look in to lp:1024114 also, because both are related to each other.
And assigning to framework team , and request to framework team would you please double check it and take the appropriate decision on it.

Thank you!