parse-ppa-apache-logs failing (missing files)

[Stuart Bishop]

parse-ppa-apache-access-logs.py on production is regularly failing with:

2012-03-10 07:06:49 ERROR Unhandled exception
 -> http://launchpadlibrarian.net/96185222/s9aeNuxZ3ghdgtbnm53cwfdUWYQ.txt ([Errno 2] No such file or directory: '/var/log/apache2/ppa.launchpad.net-access.log-20110912.gz')

The traceback is:

Traceback (most recent call last):
  File "/srv/launchpad.net/production/launchpad-rev-14914/lib/lp/services/scripts/base.py", line 103, in log_unhandled_exceptions_func
    return func(self, *args, **kw)
  File "/srv/launchpad.net/production/launchpad-rev-14914/lib/lp/services/scripts/base.py", line 387, in lock_and_run
    use_web_security=use_web_security, isolation=isolation)
  File "/srv/launchpad.net/production/launchpad-rev-14914/lib/lp/services/scripts/base.py", line 103, in log_unhandled_exceptions_func
    return func(self, *args, **kw)
  File "/srv/launchpad.net/production/launchpad-rev-14914/lib/lp/services/scripts/base.py", line 340, in run
    self.main()
  File "/srv/launchpad.net/production/launchpad-rev-14914/lib/lp/services/apachelogparser/script.py", line 79, in main
    for fd, position in files_to_parse:
  File "/srv/launchpad.net/production/launchpad-rev-14914/lib/lp/services/apachelogparser/base.py", line 39, in get_files_to_parse
    fd, file_size = get_fd_and_file_size(file_path)
  File "/srv/launchpad.net/production/launchpad-rev-14914/lib/lp/services/apachelogparser/base.py", line 71, in get_fd_and_file_size
    fd = gzip.open(file_path)
  File "/usr/lib/python2.6/gzip.py", line 33, in open
    return GzipFile(filename, mode, compresslevel)
  File "/usr/lib/python2.6/gzip.py", line 79, in __init__
    fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')
IOError: [Errno 2] No such file or directory: '/var/log/apache2/ppa.launchpad.net-access.log-20110912.gz'

Hopefully, this error is spurious in which case the code needs to be updated to cope and not report errors (even if it does signal a production issue, such as files being removed before they can be processed, WARNING would be a better level).

[Julian Edwards (julian-edwards)]

I wonder if it's a race condition with log rotation?

[William Grant (wgrant)]

That's been my assumption.

[Stuart Bishop (stub)]

On Mon, Mar 12, 2012 at 8:08 AM, William Grant <email address hidden> wrote:

> That's been my assumption.

Given that the missing file is 6 months old, I think we can assume it
has already been processed and we are not losing data.

--
Stuart Bishop <email address hidden>

[Launchpad QA Bot (lpqabot)]

Fixed in stable r14936 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/14936>.