date_reported in python-launchpad-bugs not accessible via bughelper

[Brian Murray]

1) 196 from bughelper.main

python-launchpad-bugs has information about the date a bug was reported (date_reported) however, this information is not accessible via bughelper. Having the ability to filter bugs based off the date they were reported would be helpful.

[Brian Murray (brian-murray)]

Needs review.

[Markus Korn (thekorn)]

Thanks Brian for working on this,
I was unable to test it until now, but had a quick look at the code.
I think you missed one important part in py-lp-bugs:
In 'datereported()' you have to ensure that the bug-object is really an instance of 'Bug' and not an integer or an instance of 'BugInfo'. You might want to have a look at the huge 'try:...except AttributeError:...'-block in 'lastcomment()' where I did what I think you also need in 'datereported()'. If you do not add such a test-block you will always get an error running bugnumbers with '--parsemode=html'.

Will have a closer look at the code later today,

Markus

[Markus Korn (thekorn)]

A common use case for this would be:
    "show me all open bugs in ubuntu which were reported since 2008-01-01"

I suggest the following work flow in py-lp-bugs (the fastest solution I could imagine):
   1.) use the html-mode for BugList (this returns us many blocks of 50 bugs and not one big list as in the text-mode)
   2.) let launchpad sort the buglist by "newest first" ('?orderby=-datereported')
   3.) stop getting bug pages when the bug was reported before 2008-01-01

This work flow can easily be adapted for "reported on 2008-01-01" or "reported before 2008-01-01"

A correct implementation of this solution is blocked by Bug 194312, as we need Buglist to be a list to respect the order of the elements)

Markus

[Markus Korn (thekorn)]

The following two attachments are implementing the idea described in comment #3, but should not work as long buglist is a set.

[Markus Korn (thekorn)]

I got the following results while testing the latest versions of the both branches added by me and as far as I can see they are looking good:

markus@thekorn:/media/sda5/bughelper/datereported$ ./bugnumbers -P bughelper --dr "=2007-03-05"
88521 (Confirmed, Wishlist) - selectively pulling attachments by name
88102 (Confirmed, Medium) - sort report by issue type rather than bug number
markus@thekorn:/media/sda5/bughelper/datereported$ ./bugnumbers -P bughelper --dr "<2007-03-05"
84723 (Confirmed, Wishlist) - RFE: grep for package version
88102 (Confirmed, Medium) - sort report by issue type rather than bug number
81291 (Confirmed, Wishlist) - RFE: Web interface to bughelper
88521 (Confirmed, Wishlist) - selectively pulling attachments by name
markus@thekorn:/media/sda5/bughelper/datereported$ ./bugnumbers -P bughelper --dr ">2007-03-05"
109547 (Confirmed, Wishlist) - support to sort out matched clues to multiple files per package (e.g. one file per task)
109628 (Confirmed, Wishlist) - allow clues to match "duplicate count > X"; e.g. to list top-crashers
185357 (In Progress, Wishlist) - date_reported in python-launchpad-bugs not accessible via bughelper
183887 (In Progress, Wishlist) - bugnumbers --stats could be more complete
131378 (New, Medium) - diff between output of feisty's and gutsy's bughelper
197037 (In Progress, Low) - bug statuses are case sensitive
196218 (Confirmed, Undecided) - No bug attribute for a bzr branch
110937 (Confirmed, Wishlist) - show status updates during long bughelper runs

I've attached the result for:
  $ ./bugnumbers -D ubuntu --dr "=2008-03-03"

Markus

[Markus Korn (thekorn)]

in the recent version of the bughelper branch I fixed an error when running --dr and --stats in one query, it's looking better now:

$ ./bugnumbers -D ubuntu --dr "=2008-03-03" --stats
reportTotal: 199

status
 In Progress: 5
 Confirmed: 37
 Fix Committed: 2
 Triaged: 1
 New: 135
 Incomplete: 19

importance
 High: 2
 Medium: 14
 Wishlist: 11
 Undecided: 161
 Low: 11

Markus

[Brian Murray (brian-murray)]

Awesome, I have this branch running in the data center now.

I wonder if package maintainers would have any interest in the using '--dr' as it is now, with BugList being a set. I could see it being useful if someone was on vacation or what have and wanted to look at all the bugs reported while they were on vacation. I thought it still worked pretty well on a per package basis, just not for the whole distribution.

[Markus Korn (thekorn)]

Added a python-launchpad-bugs task, this way we can track the necessary changes there

Markus

[Markus Korn (thekorn)]

I rewrote basebuglistfilter.datereported() in rev117 of the intrepid.merge branch. The API changed compared to the former approaches here in the bugreport, so we need to rewrite the bughelper part too.
This solution is more robust, faster and has more readable code.
I attached a example script which should show the usage and gives us a possibility to test this function, please use it ;)

Markus

[Markus Korn (thekorn)]

I forgot to mention: it's required to use buglists sorted by + or - datereported

[Markus Korn (thekorn)]

updated script

[Markus Korn (thekorn)]

updated the test-script, it was wrong, parsing date_created for tasks was never supported in the html-mode

Markus

[Brian Murray (brian-murray)]

I believe this hangs indefinitely when you use a future date. For example I took the script and just changed date_option to:

date_option = (">", LPTime("2008-06-27 00:00:01 BST"))

and it is taking longer to run than 2008-06-26.

[Markus Korn (thekorn)]

Having a date in the future here does not make any sense to me. in revision 141 I fixed this by raising a ValueError in this cases.

Markus

[Brian Murray (brian-murray)]

Even with revision 141 it is still behaving oddly for me.

2008-06-26 00:00:01 BST - works fine and returns results in about 4 minutes
2008-06-28 00:00:01 BST - raises the value error
2008-06-27 00:00:01 BST - still running after 15 minutes

[Markus Korn (thekorn)]

------------------------------------------------------------
revno: 107
committer: Markus Korn <email address hidden>
branch nick: python-launchpad-bugs
timestamp: Wed 2008-07-02 22:34:33 +0200
message:
  * Improved LPBugList and LPBugPage:
    This two classes have now a cleaner structure, filtering of bug lists
    with python-launchpad-bugs is now a lot easier. With
    LPBugList.set_progress_hook() it is now possible to visualize the
    process of fetching bug lists.
  * launchpadbugs/{tasksbase,html_bug.py,text_bug.py}: if a task is a
    bugwatch then task.remote returns the remote-url, otherwise it returns
    'None'
  * Improved error handling in python-launchpad-bugs and added
    infrastructure for unittests. This also contains some basic testcases in
    tests/ (LP: #189572)
  * Added ability to parse bugtracker overviews (LP: #203312)
  * It is now possible to filter bug lists based on milestones. This also
    implements a helper function to list all active milestones of a project
    in launchpad (LP: #200457)
  * Added function to filter buglists based on Bug.date_reported
    (LP: #185357)
  * added 'move-duplicates' written by Kjell Braden and 'README' to
    examples/ (LP: #208148)
  * launchpadbugs/attachmentsbase.py: LPAttachment.download() now also
    accepts an optional argument to specify the location where to download
    the attachment (LP: #242317)
  * Added method to let the user choose which version/server of launchpad to
    use, so it is possible to explicit use bugs.launchpad.net or
    bugs.edge.launchpad.net (LP: #188298)
  * launchpadbugs/html_bug.py: InfoTable: fixed parsing of milestones in
    read-only mode (LP: #243057)
  * Added functionality for adding new tasks to bug reports.
    With this implementation so far you are able to add 'project' and
    'distro' tasks, but unable to create bugwatches and change task
    attributes before committing the changes. (LP: #193853)
  * launchpadbugs/html_bug.py: Stephan Hermann fixed html connector to work
    with the newest launchpad rollout (LP: #243193, #244452)
    .
    Unfortunately this update breaks compatibilty. Please update your code
    to make use of the python-launchpad-bugs 0.3 API. It's worth it.
    ------------------------------------------------------------

[Brian Murray (brian-murray)]

Patch to add date reported filtering to bug numbers.

I don't think the last line of it is the best way though:

+ bug_filter.functions.append(datereported((opt_d.keys().pop(), opt_d.values().pop()), Bug))

[Markus Korn (thekorn)]

thanks Brian, this is looking good. But I think we should change the type of the value returned by _parse_date_reported(() from dict to tuple, so everything is looking cleaner, please see the attached patch.

Markus

[Brian Murray (brian-murray)]

I'm going to rename the bugnumbers option to --dc and date_created as this is really filtering on the date a task was created not the date when the bug was reported.

[Brian Murray (brian-murray)]

I've merged this into revision 214 of bughelper.