fetchmail mail_gateway don't accept attachment 6.0.3

[xavinsky]

Openerp 6.0.3. Module Fetchmail.

  It's working when I send a simple mail.
my mail is fetched.
  When my mail have an attachment.
It's fail with this message.
WARNING:pop:[02]: Error occurred while validating the field(s) name,parent_id,res_model,res_id: File name must be unique!

  In the code it's the line in mail_gateway/mail_gateway.py
attachments.append(att_obj.create(cr, uid, {'res_model':case._name,'res_id':case.id,'name': att[0], 'datas': base64.encodestring(att[1])}))
that's don't work... the name att[0] seem correct...
I need another module for that's work ? or a config ?

  Module installed :
base
base_action_rule
base_calendar
base_setup
board
crm
document
document_webdav
fetchmail
knowledge
mail_gateway
process
resource
web_livechat

[Olivier Dony (Odoo) (odo-openerp)]

The 'document' module modifies the way attachments (ir.attachment records) are stored, to make them part of a global virtual file system. File names must be unique per directory in this "virtual" file system, in order to make it compatible with WebDav, and FTP bindings.

As you're technically inclined, look at the _check_duplication() in document.py, it prevents creating 2 files with the same name attached to the same document (res_model + res_id).

I suppose your use case involves receiving multiple mails with the same file attached, related to the same resource. In this case, the fetchmail script should perhaps generate a unique file name (e.g. with a random suffix) to avoid name clashes.

PS: I did not confirm the bug yet, as I did not have time to test and reproduce.

[Ferdinand (office-chricar)]

 generate a unique file:
IMHO use timestamp + running sequence is better than random

[xavinsky (xavinsky)]

Hi Olivier.

 > File names must be unique per directory in this "virtual" file system
Yes I understood this. But it's my first file...

How to reproduce this bug :
  I create a vm for test this bug.
I install openerp and an unique french database without demonstration data
I install with wizrd crm, knowledge (with default options)
I install the module fetchmail.
  I create a new acount gmail for this test.
  I create my server mail pop.gmail.com (ssl) with login/passwd info in openerp
with modele crm.lead.
 I confirm.

  I send a first simple email without attachment. The fetch works.

I create a simple text file toto.txt
  I send a second email with toto attachment. The fetch don't works.

[2011-11-22 10:36:24,610][testmail] INFO:imap:fetchmail start checking for new emails on gmailtestfetch
[2011-11-22 10:36:25,041][testmail] WARNING:pop:[01]: ValidateError
[2011-11-22 10:36:25,042][testmail] WARNING:pop:[02]: Error occurred while validating the field(s) name,parent_id,res_model,res_id: File name must be unique!

I reproduce this bug on 2 VM (gentoo and ubuntu 11.10 server)
I reproduce this bug with and without the option add attachment.

>> I suppose your use case involves receiving multiple mails with the same file attached, related to the same resource.
As you can see, It's not the usecase now, but it's could be true later :)

The name of the file seem is detected in the att[0] in mail_gateway/mail_gateway.py
I think the module must make 2 create for this file, or somethings is wrong in detection name file...

[xavinsky (xavinsky)]

  I continue mi investigation...

  In the script fetchmail.py
the object email_server in the method fetch_mail
call process_email.

in the script mail_gateway.py
the object mailgate_tool in the method process_email

we have the code

...
 if not len(res_ids):
            new_res_id, attachment_ids = create_record(msg) # FIRST CREATE ??
            res_ids = [new_res_id]

        # Store messages
        context.update({'model' : model})
        if hasattr(model_pool, 'history'):
            model_pool.history(cr, uid, res_ids, _('receive'), history=True,
                            subject = msg.get('subject'),
                            email = msg.get('to'),
                            details = msg.get('body'),
                            email_from = msg.get('from'),
                            email_cc = msg.get('cc'),
                            message_id = msg.get('message-id'),
                            references = msg.get('references', False) or msg.get('in-reply-to', False),
                            attach = attachments.items(),
                            email_date = msg.get('date'),
                            context = context) # SECOND CREATE
...

create_record(msg) create an instance of crm.lead. and ir.attachment(s)
and history part try to make a second one, and it's didn't works.
It's could be the explain of this bug.

the methode history of the object mailgate_thread

in the object mailgate_thread the method try to do

attachments.append(att_obj.create(cr, uid, {'res_model':case._name,'res_id':case.id,'name': att[0], 'datas': base64.encodestring(att[1])}))

and it's rejeted
Error occurred while validating the field(s) name,parent_id,res_model,res_id: File name must be unique!
with name,parent_id,res_model,res_id =>toto.txt 1 crm.lead 85

before and after run the test, in sql :
 select * from crm_lead;
select * from ir_attachment;
show there is nothing created for toto.txt ...

In create_record :
create ir attachement {'partner_id': False, 'description': 'Mail attachment', 'res_model': 'crm.lead', 'parent_id': 1, 'datas_fname': 'toto.txt', 'file_size': 13, 'res_id': 85L, 'datas': 'dG90b3RvdG8K\n', 'name': 'toto.txt'}

In history :
create ir attachement {'name': 'toto.txt', 'res_model': 'crm.lead', 'parent_id': 1, 'res_id': 85L, 'file_size': 13, 'partner_id': False, 'datas': 'dG90b3RvdG8K\n'}

=> bug !!!

[xavinsky (xavinsky)]

fetchmail.py email_server.fetch_mail()
=> mail_gateway.py mailgate_tool.process_email() :

1) => create_record => crm_lead.py crm_lead.message_new()
  => self.pool.get('ir.attachment').create(cr, uid, data_attach)

2) => history => ... att_obj.create(cr, uid, {'res_model':case._name,'res_id':case.id,'name': att[0], 'datas': base64.encodestring(att[1])}...

then we have the conflict :
WARNING:pop:[02]: Error occurred while validating the field(s) name,parent_id,res_model,res_id: File name must be unique!

Now , how to fix it ?
  remove attachment in crm_lead.message_new() ? In this case how to do with other model with message_new...
  don't use specifics message_new for model and remove creation of attachment ?
  don't create attachment in history ?
  add an flag for creation of attachement in history ?

  I am new in openerp framework, then I don't know how fix this without cause others troubles...

Thx for help me.

[Olivier Dony (Odoo) (odo-openerp)]

- Confirming for 6.0.3 only, the bug does not exist in trunk.
- Setting importance to low, this only occurs when: document module is installed + mail_gateway is configured + first mail for new thread arrives with an attachment.

Problem (as explained by Xavier):
In case of new emails, mailgate_tool calls first create_record(), and then history(), and both will try to attach the mail's attachment to the new record. This fails when document module is installed, because you can't have twice the same file attached to the same record.

Recommended solution:
The ideal solution would be to cleanup the history() and message_new() (called by create_record) so there would be no duplicated functionality. However this is too risky for stable, it would change the API and requires fixing many modules (all the message_new() implementations)
So the best solution for stable is to change mailgate_tool.process_email(), and make sure it does not pass the attachments to history() when it is processing a new email: if create_record was called, then do not pass `attach` to history, otherwise pass it.

[xavinsky (xavinsky)]

I don't know if it's a fetchmail problem only or an gateway problem.

if it's a gateway problem, the very simple solution :
in mail_gateway.py, replace the line 577
                            attach = attachments.items(),
by this :
                            attach = [],
It's working for me.

But if it's can avoid side effect , because it's only a fetchmail problem,
we must add a parameter to process_mail fonction (attach_history_ignore=False)
and send it at True in fetchmail fonction.
and for in line 557
                            attach = attachments.items() if attach_history_ignore else []
thx to tell me what is the best way.

[Rifakat Husen (OpenERP) (rha-openerp)]

Landed into into Stable 6.0 branch,
Revision ID: <email address hidden>
Revision: 4872

Regards.