log parser should not read entire file into memory

Bug #588288 reported by Michael Nelson on 2010-06-01
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Medium
Michael Nelson

Bug Description

This is related to bug 139855. Currently when parsing a log file, the entire contents to be parsed is read into memory via readlines() rather than processing each line individually.

Additionally, it would be good to allow a config option to limit the number of lines processed at a time.

Related branches

Changed in launchpad-foundations:
milestone: 10.06 → 10.05
status: In Progress → Fix Committed
tags: added: qa-needstesting
Ursula Junque (ursinha) on 2010-06-06
Changed in launchpad-foundations:
milestone: 10.05 → 10.06
Julian Edwards (julian-edwards) wrote :

Got this during QA testing, dunno if it's related:

2010-06-07 12:23:58 ERROR Error (Unable to parse: 0] "GET /ubuntu-langpack/ubuntu/dists/edgy/main/binary-i386/Packages.gz HTTP/1.1" 200 20268 "-" "Debian APT-HTTP/1.3" with the ^(\S*) (\S*) (\S*) (\[[^\]]+\]) \"([^"\\]*(?:\\.[^"\\]*)*)\" (\S*) (\S*) \"([^"\\]*(?:\\.[^"\\]*)*)\" \"([^"\\]*(?:\\.[^"\\]*)*)\"$ regular expression) while parsing "0] "GET /ubuntu-langpack/ubuntu/dists/edgy/main/binary-i386/Packages.gz HTTP/1.1" 200 20268 "-" "Debian APT-HTTP/1.3"

Julian Edwards (julian-edwards) wrote :

And that seems to prevent it scanning the file further.

Julian Edwards (julian-edwards) wrote :

see bug 590766 for this issue

Michael Nelson (michael.nelson) wrote :

I've fixed the issue (liked to bug 590766) and qa'd on df, so we can mark this qa-ok once the branch lands. I'll land a separate config change to limit the number of lines processed initially for ppa parsing.

tags: added: qa-ok
removed: qa-needstesting
Changed in launchpad-foundations:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers