status update is so slow it blocks the two following updates

[Peter Maydell]

The update jobs for status.linaro.org appear to be taking about three hours to run. You can see from the update log:
http://status.linaro.org/update.log.txt
that although we kick off an update once an hour, the two subsequent attempts fail because the first one is still running. Looking at the timestamps we're only a few minutes away from having three blocked update attempts for every successful one.

This means that the actual update frequency is more like "every three hours" than the intended "every hour".

[Mattias Backman (mabac)]

On Tue, Oct 25, 2011 at 3:32 PM, Launchpad Bug Tracker
<email address hidden> wrote:
> You have been subscribed to a public bug by Alexander Sack (asac):
>
> The update jobs for status.linaro.org appear to be taking about three hours to run. You can see from the update log:
> http://status.linaro.org/update.log.txt
> that although we kick off an update once an hour, the two subsequent attempts fail because the first one is still running. Looking at the timestamps we're only a few minutes away from having three blocked update attempts for every successful one.
>
> This means that the actual update frequency is more like "every three
> hours" than the intended "every hour".

The scheduled used to be every two hours but since the update
eventually took more than those two hours it was in effect 4 hours
between updates. A small fix was to set the schedule to every hour so
that the next update is kicked off asap (effectively every three
hours). So even though we get a couple of lines in the update log
about this, we actually get 25% more updates in a day with this
schedule.

Granted, the update takes a really long time which increases with
every new bp and work item being added.

>
> ** Affects: launchpad-work-items-tracker
>     Importance: Undecided
>         Status: New
>
> --
> status.linaro.org update is so slow it blocks the two following updates
> https://bugs.launchpad.net/bugs/881368
> You received this bug notification because you are a member of Linaro Infrastructure, which is subscribed to the bug report.
>

[Peter Maydell (pmaydell)]

> Granted, the update takes a really long time which increases with
> every new bp and work item being added.

Yes, my point was more "updates shouldn't be taking three hours" -- clearly the code is correctly handling the "previous update still in progress" case.

[Mattias Backman (mabac)]

On Tue, Oct 25, 2011 at 5:06 PM, Peter Maydell <email address hidden> wrote:
>> Granted, the update takes a really long time which increases with
>> every new bp and work item being added.
>
> Yes, my point was more "updates shouldn't be taking three hours" --
> clearly the code is correctly handling the "previous update still in
> progress" case.

That's a valid point. I did discuss this with James a few weeks ago
and we agreed that there is no trivial fix to cut the duration of the
updates significantly. Can you provide a target for what is
reasonable?

[Peter Maydell (pmaydell)]

My typical use case is that I look at status.linaro, spot some work items I've already done or want to rearrange, drill down into the blueprint to change them, and than want to come back and see the changes reflected in status.linaro. So ideally you want launchpad to push out notifications of blueprint updates to status.linaro to give you real-time updates. (at least, my uneducated guess is that there aren't that many actual changes to blueprints and so monitoring changes is probably more efficient than rescanning all blueprints from scratch). Failing that, anything beyond "within 5 or 10 minutes" is starting to get pretty awkward. More than an hour or so and I think it's pretty much unusable as a personal "engineer's todo list" (although obviously the 'provide stats and pretty graphs to management' aspect can probably cope with longer refresh times).

That's just my personal opinion though -- other users might be more tolerant of delayed updates.

[Mattias Backman (mabac)]

On Tue, Oct 25, 2011 at 6:44 PM, Peter Maydell <email address hidden> wrote:
> My typical use case is that I look at status.linaro, spot some work
> items I've already done or want to rearrange, drill down into the
> blueprint to change them, and than want to come back and see the changes
> reflected in status.linaro. So ideally you want launchpad to push out
> notifications of blueprint updates to status.linaro to give you real-
> time updates. (at least, my uneducated guess is that there aren't that
> many actual changes to blueprints and so monitoring changes is probably
> more efficient than rescanning all blueprints from scratch). Failing
> that, anything beyond "within 5 or 10 minutes" is starting to get pretty
> awkward. More than an hour or so and I think it's pretty much unusable
> as a personal "engineer's todo list" (although obviously the 'provide
> stats and pretty graphs to management' aspect can probably cope with
> longer refresh times).

I understand. Getting down to around one hour will probably not be
possible with the current scheme of scanning blueprints to get the
background data. To get down to a few minutes will absolutely require
rewriting the entire thing and getting Launchpad to push
notifications.

I'll poke the PM:s and see if we can get this into the list of future
projects. It's down to what the purpose of s.l.o is I guess and if
this work can be motivated by the value added.

>
> That's just my personal opinion though -- other users might be more
> tolerant of delayed updates.
>
> --
> You received this bug notification because you are a member of Linaro
> Infrastructure, which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/881368
>
> Title:
>  status.linaro.org update is so slow it blocks the two following
>  updates
>
> Status in Track work items and generate burndown charts:
>  New
>
> Bug description:
>  The update jobs for status.linaro.org appear to be taking about three hours to run. You can see from the update log:
>  http://status.linaro.org/update.log.txt
>  that although we kick off an update once an hour, the two subsequent attempts fail because the first one is still running. Looking at the timestamps we're only a few minutes away from having three blocked update attempts for every successful one.
>
>  This means that the actual update frequency is more like "every three
>  hours" than the intended "every hour".
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/launchpad-work-items-tracker/+bug/881368/+subscriptions
>

[Mattias Backman (mabac)]

On Wed, Oct 26, 2011 at 11:05 AM, Mattias Backman
<email address hidden> wrote:
> On Tue, Oct 25, 2011 at 6:44 PM, Peter Maydell <email address hidden> wrote:
>> My typical use case is that I look at status.linaro, spot some work
>> items I've already done or want to rearrange, drill down into the
>> blueprint to change them, and than want to come back and see the changes
>> reflected in status.linaro. So ideally you want launchpad to push out
>> notifications of blueprint updates to status.linaro to give you real-
>> time updates. (at least, my uneducated guess is that there aren't that
>> many actual changes to blueprints and so monitoring changes is probably
>> more efficient than rescanning all blueprints from scratch). Failing
>> that, anything beyond "within 5 or 10 minutes" is starting to get pretty
>> awkward. More than an hour or so and I think it's pretty much unusable
>> as a personal "engineer's todo list" (although obviously the 'provide
>> stats and pretty graphs to management' aspect can probably cope with
>> longer refresh times).
>
> I understand. Getting down to around one hour will probably not be
> possible with the current scheme of scanning blueprints to get the
> background data. To get down to a few minutes will absolutely require
> rewriting the entire thing and getting Launchpad to push
> notifications.
>
> I'll poke the PM:s and see if we can get this into the list of future
> projects. It's down to what the purpose of s.l.o is I guess and if
> this work can be motivated by the value added.
>
>>
>> That's just my personal opinion though -- other users might be more
>> tolerant of delayed updates.

We'll just have to discuss it with the other stakeholders and see.
Would you sponsor the project to make this happen? Could you bring
this it up with the Infrastructure stakeholders
(linaro-infrastructure-stakeholders) to get the ball rolling?

>>
>> --
>> You received this bug notification because you are a member of Linaro
>> Infrastructure, which is subscribed to the bug report.
>> https://bugs.launchpad.net/bugs/881368
>>
>> Title:
>>  status.linaro.org update is so slow it blocks the two following
>>  updates
>>
>> Status in Track work items and generate burndown charts:
>>  New
>>
>> Bug description:
>>  The update jobs for status.linaro.org appear to be taking about three hours to run. You can see from the update log:
>>  http://status.linaro.org/update.log.txt
>>  that although we kick off an update once an hour, the two subsequent attempts fail because the first one is still running. Looking at the timestamps we're only a few minutes away from having three blocked update attempts for every successful one.
>>
>>  This means that the actual update frequency is more like "every three
>>  hours" than the intended "every hour".
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/launchpad-work-items-tracker/+bug/881368/+subscriptions
>>
>