log parser should not read entire file into memory

[Michael Nelson]

This is related to bug 139855. Currently when parsing a log file, the entire contents to be parsed is read into memory via readlines() rather than processing each line individually.

Additionally, it would be good to allow a config option to limit the number of lines processed at a time.

[Ursula Junque (ursinha)]

Fixed in stable r10942 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/10942>

[Julian Edwards (julian-edwards)]

Got this during QA testing, dunno if it's related:

2010-06-07 12:23:58 ERROR Error (Unable to parse: 0] "GET /ubuntu-langpack/ubuntu/dists/edgy/main/binary-i386/Packages.gz HTTP/1.1" 200 20268 "-" "Debian APT-HTTP/1.3" with the ^(\S*) (\S*) (\S*) (\[[^\]]+\]) \"([^"\\]*(?:\\.[^"\\]*)*)\" (\S*) (\S*) \"([^"\\]*(?:\\.[^"\\]*)*)\" \"([^"\\]*(?:\\.[^"\\]*)*)\"$ regular expression) while parsing "0] "GET /ubuntu-langpack/ubuntu/dists/edgy/main/binary-i386/Packages.gz HTTP/1.1" 200 20268 "-" "Debian APT-HTTP/1.3"

[Julian Edwards (julian-edwards)]

And that seems to prevent it scanning the file further.

[Julian Edwards (julian-edwards)]

see bug 590766 for this issue

[Michael Nelson (michael.nelson)]

I've fixed the issue (liked to bug 590766) and qa'd on df, so we can mark this qa-ok once the branch lands. I'll land a separate config change to limit the number of lines processed initially for ppa parsing.