Invalid Mail Message Date format makes fetchmail crash permanently

[Eric Caudal - www.elico-corp.com]

A date format causes the cron job to crash and cannot recover.

[2011-07-18 08:22:40,863][company] INFO:imap:fetchmail start checking for new emails on <email address hidden>

[2011-07-18 08:22:41,245][company] DEBUG:tools:tools.misc.html2plaintext: cannot use BeautifulSoup, fallback to lxml.etree.HTMLParser

[2011-07-18 08:22:41,273][company] DEBUG:translate:no translation language detected, skipping translation for "'receive'"

[2011-07-18 08:22:41,280][company] ERROR:db.cursor:bad query: insert into "mailgate_message" (id,"description","server_type","email_bcc","email_to","email_cc","date","partner_id","server_id","user_id","name","email_from","references","model","res_id","message_id","history",create_uid,create_date) values (2714,E'&#13;

×ð¾´µÄÓû§ÄúºÃ:

&#13;

½üÆÚÒòÃÜÂë¼òµ¥¶øÔì³ÉÓÊÏäÕ˺ű»µÁÇé¿öƵ·¢£¬ÎªÁË·ÀÖ¹ÄúÕý³£µÄÓÊÏäÕ˺ű»ÓÃÓÚ·Ç·¨À¬»øÓʼþ·¢ËÍ£¬ÎÒÃǶÔϵͳÔö¼ÓÁËÈõÃÜÂë¼ì²â¹æÔò£¬&#13;

¾­ÏµÍ³¼ì²âÄúµÄÃÜÂë¹ýÓÚ¼òµ¥,¼«Ò×±»·Ç·¨·Ö×Ó½øÐÐÃÜÂëÆƽ⣬µ«ÊÇÓÉÓÚÄúµÄÃÜÂëÊôÓÚ¼ÓÃÜÐÅÏ¢£¬ÎÒÃÇÎÞ·¨²é¿´£¬ËùÒÔÇëÄú×ÔÐм°Ê±ÊÖ¶¯ÐÞ¸ÄÃÜ&#13;

Â룬·ñÔòϵͳ½«ÏÞÖÆÄúÿÈÕÍâ·¢ÓʼþµÄÊýÁ¿Îª20·â£¬ÐÞ¸ÄÍêÃÜÂëºóϵͳ½«×Ô¶¯È¡Ïû¶ÔÄúµÄÓÊÏäÏÞÖÆ!

&#13;

*ÃÜÂëÉèÖÃСÌùÊ¿*

&#13;

ÇëÄú²ÉÄÉÎÒÃÇÌṩ¸øÄúµÄÒÔϹØÓÚÃÜÂëÉèÖõÄС½¨Ò飺

&#13;

1. ÃÜÂ볤¶È±ØÐëÔÚ6-20λ

&#13;

2. ±ØÐë²ÉÓÃÎÞÐòµÄ¡°×Öĸ+Êý×Ö¡±µÄ×éºÏ·½Ê½£»¸ñʽÈ磺c1h2i3n4a5

&#13;

3. ²»Òª²ÉÓõÝÔö»òÕߵݼõµÄÄÚÈÝ£¬È磺¡°ABCD1234¡±¡¢¡°4321DCBA¡±£¬ÒòΪÕâÑùµÄÃÜÂëºÜÈÝÒ×±»ÃÜÂë¼ÆË㹤¾ßËùÆƽ⣻

&#13;

4. ÃÜÂëÖв»Òª°üº¬ÓʼþÕ˺ż°ÓʼþÓòÃû

&#13;

5. Èç¹ûÄúʹÓÿͻ§¶ËÈí¼þ£¨foxmail,outlookµÈ£©ÊÕ·¢Óʼþ£¬ÔÚwebmailÉÏÐÞ¸ÄÃÜÂëºó£¬ÇëÁ¢¼´½øÈë¿Í»§¶Ëͬ²½½øÐÐÃÜÂëÉèÖ㬷ñÔò¿Í»§¶Ë½«ÎÞ·¨Õý³£ÊÕ·¢Óʼþ¡£

&#13;

±£»¤ÄúµÄÓÊÏäÃÜÂ룬¾ÍÊDZ£»¤ÄúÆóÒµµÄÎÞÐÎ×ʲú£¬¸ÐлÄú¶ÔÖйúÍòÍø¡°ÈõÃÜÂëÓÅ»¯¡±¹¤×÷С×éµÄÖ§³Ö£¡&#13;

',E'imap',NULL,<email address hidden>',NULL,E'Date: Tue, 12 Jul 2011 01:29:56 +0800',NULL,6,10,E'ÄúµÄÃÜÂë¹ýÓÚ¼òµ¥, Ç뾡¿ìÐÞ¸Ä!',<email address hidden>;',NULL,E'crm.lead',698,E'<1310405396$19244$22878693@<email address hidden>>',E'True',10,now())

Traceback (most recent call last):

  File "/usr/local/lib/python2.6/dist-packages/openerp-server_9040/sql_db.py", line 131, in execute

    res = self._obj.execute(query, params)

DataError: invalid input syntax for type timestamp: "Date: Tue, 12 Jul 2011 01:29:56 +0800"

LINE 23: ',E'imap',NULL,<email address hidden>',NULL,E'Date: Tue, 12 J...

                                                    ^

[2011-07-18 08:22:41,281][company] WARNING:imap:[01]: invalid input syntax for type timestamp: "Date: Tue, 12 Jul 2011 01:29:56 +0800"

[2011-07-18 08:22:41,281][company] WARNING:imap:[02]: LINE 23: ',E'imap',NULL,<email address hidden>',NULL,E'Date: Tue, 12 J...

[2011-07-18 08:22:41,281][company] WARNING:imap:[03]: ^

[2011-07-18 08:22:41,282][company] ERROR:db.cursor:bad query: SELECT email_server.active,email_server.user,email_server.date,email_server.password,email_server.port,email_server.note,email_server.user_id,email_server.name,email_server.is_ssl,email_server.object_id,email_server.server,email_server.priority,email_server.attach,email_server.state,email_server.type,email_server.action_id,email_server.id FROM "email_server" WHERE email_server.id IN (7) ORDER BY id

Traceback (most recent call last):

  File "/usr/local/lib/python2.6/dist-packages/openerp-server_9040/sql_db.py", line 131, in execute

    res = self._obj.execute(query, params)

InternalError: current transaction is aborted, commands ignored until end of transaction block

[2011-07-18 08:22:41,283][company] WARNING:imap:current transaction is aborted, commands ignored until end of transaction block

[2011-07-18 08:22:41,284][company] ERROR:db.cursor:bad query: SELECT email_server.active,email_server.user,email_server.date,email_server.password,email_server.port,email_server.note,email_server.user_id,email_server.name,email_server.is_ssl,email_server.object_id,email_server.server,email_server.priority,email_server.attach,email_server.state,email_server.type,email_server.action_id,email_server.id FROM "email_server" WHERE email_server.id IN (8) ORDER BY id

Traceback (most recent call last):

  File "/usr/local/lib/python2.6/dist-packages/openerp-server_9040/sql_db.py", line 131, in execute

    res = self._obj.execute(query, params)

InternalError: current transaction is aborted, commands ignored until end of transaction block

I temporarily changed the mail_gateway/History function around line 162 as followed:
# 'date': email_date or time.strftime('%Y-%m-%d %H:%M:%S'),
                    'date': time.strftime('%Y-%m-%d %H:%M:%S'),#EC

[Vishal Parmar(Open ERP) (vpa-openerp)]

Hello Eric Caudal,

I have tested your scenario with latest updated code with stable6 and trunk both but I did't faced any problem.

I have attached screenshot for your reference so would please check it and notify us where you faced the problem.

Would try with latest updated code with new db.

Thanks and waiting for your reply.

[Eric Caudal - www.elico-corp.com (elicoidal)]

Hi Vishal,
I think you misunderstood my point. Actually I do receive mails in
Chinese into the CRM system. The problem happens with some mails that
seems to have wrong format for the date: check the trackback on my first
message.
Some mails when they arrive have the date like

'Date: Tue, 12 July 2011' Instead of 'Tue, 12 July 2011'

The problem is when this happens fecthmail tries over and over to get
this email and crashes all cron jobs.

So what we should at least check in the code is that the date is
properly formated else take the date of the day.

Eric CAUDAL
<email address hidden> <mailto:<email address hidden>>
Cell phone: + 86 186 2136 1670

Elico Corp, Shanghai
OpenERP Ready Partner
http://www.elico-corp.com

On 29/07/2011 21:00, Vishal Parmar(Open ERP) wrote:
> ** Attachment added: "mail_message.png"
> https://bugs.launchpad.net/openobject-addons/+bug/812072/+attachment/2240548/+files/mail_message.png
>
> ** Changed in: openobject-addons
> Status: New => Incomplete
>

[Vishal Parmar(Open ERP) (vpa-openerp)]

Hello Eric Caudal,

I have completely checked your issue again as per your bug specification but I did't face any trace back or

crash mail problem. One thing I have mark in your bug specification you got this 'Date: Tue, 12 July 2011' Instead of

'Tue, 12 July 2011' but I did't get 'Date: Tue, 12 July 2011' and it is working as expected.

Would you please provide me your end configuration or scenario with proper steps, video or screenshot so

we can reproduce same at my end.

Thanks and waiting for your reply.

[Eric Caudal - www.elico-corp.com (elicoidal)]

Hi,
What information do you have in the mail itself? Did your mail provider
sent you 'Date: Tue...'? Because this is what I receive in my mail and
which is blocking OpenERP. I am not saying that OpenERP is writing 'Date
: Tues': just that I receive this content in field date and then the
system doesnot handle it properly (it should detect an error in format
and take the date of the day in this case).

Eric CAUDAL

[Vishal Parmar(Open ERP) (vpa-openerp)]

Hello Eric Caudal,

I have checked again your issue with latest updated code with stable6 and trunk both but still I did't face any

problem and it is working.

Would you please provide me your database backup and try with latest updated code and new db.

Thank and waiting for your reply.

[Eric Caudal - www.elico-corp.com (elicoidal)]

Hi,
Did you receive a mail with wrong date format?

If this is not the case, even providing you backup will not help because
if I receive the right mail format, I have no problem.
Best regards

Eric CAUDAL
<email address hidden> <mailto:<email address hidden>>
Cell phone: + 86 186 2136 1670

Elico Corp, Shanghai
OpenERP Ready Partner
http://www.elico-corp.com

[Dhruti Shastri(OpenERP) (dhs-openerp)]

Hello Eric,

I have checked your scenario at my end and it is completely working fine.

According to your comment #6,
In the mail, I have changed the date formate according to your specification(ex,Wednesday 03 August 2011 10:04:39 IST) but working normally.

If problem is still harassing you, then please let us know with more information.

It would be great if you provide screenshots.It will help us to solve the problem faster.

Thanks.

[Eric Caudal - www.elico-corp.com (elicoidal)]

Sorry, but you still dont understand my meaning (or I cant express
myself clearly enough which is the same ;)).

It is not a question of format it is question that before the date,
there are the words "date :" which make crash OE because OE expects a
date and receives a string.
This is a problem of the email server sending a wrong information.

I hope this is clear enough.

Eric CAUDAL
<email address hidden> <mailto:<email address hidden>>
Cell phone: + 86 186 2136 1670

Elico Corp, Shanghai
OpenERP Ready Partner
http://www.elico-corp.com

On 03/08/2011 15:04, Dhruti Shastri(OpenERP) wrote:
> Hello Eric,
>
> I have checked your scenario at my end and it is completely working
> fine.
>
> According to your comment #6,
> In the mail, I have changed the date formate according to your specification(ex,Wednesday 03 August 2011 10:04:39 IST) but working normally.
>
> If problem is still harassing you, then please let us know with more
> information.
>
> It would be great if you provide screenshots.It will help us to solve
> the problem faster.
>
> Thanks.
>

[Vishal Parmar(Open ERP) (vpa-openerp)]

Hello Eric Caudal,

I have tested again your scenario as per your specification still I didn't face any problem.

I have attached a video for your more reference so would you please check it again and informed us where

you faced the problem and provide related video, you end configuration.

Thanks and waiting for your reply.

[Eric Caudal - www.elico-corp.com (elicoidal)]

Hi Vishal Parmar,
I am sorry, but you are not answering the question over and over. I
think you simply do not understand my issue at all.

I will repeat me again here: I have no problem receiving my emails in
CRM. No need to send a video about it: I have no problem with that aspect.
The problem I have is certain emails from the mail supplier are sent
with incorrect date formating and the information which is parsed after
in OpenERP includes an extra "Date :" in the date field when processed
by fetchmail (check my log).

Please no need to send me another useless video: escalate the bug if you
do not understand me.

Eric CAUDAL <mailto:<email address hidden>>

[Launchpad Janitor (janitor)]

[Expired for OpenERP Addons because there has been no activity for 60 days.]

[Amit Parik (amit-parik)]

Hello Eric,

Which type of character encoding you have set on your mail client?
Because I have tried with GB2312(Chinese Simplified) encoding and It didn't pass a string for date filed.

Thanks and waiting for your reply!

[Launchpad Janitor (janitor)]

[Expired for OpenERP Addons because there has been no activity for 60 days.]

[Amit Parik (amit-parik)]

Hello Eric,

Would you please try with the latest updated code.

If you still face the problem then you can reopen this with more information.

Thanks.