Sorting jobs is time consuming

[Marc Legris]

Feature request:
When checkbox runs initially it gathers information on the machine it is running on. If this information could be stored so it only runs once this would save time. Some machines take as long as 10 minutes to run this task.

[Marc Tardif (cr3)]

The time spent during initial data gathering is mostly split between resources and jobs. As far as resources are concerned, you might like to run each of the commands under the resources.txt* file in the base checkbox package. If these are reasonably quick, then the problem is not with the resources. As far as jobs are concerned, the current algorithm for topologically sorting jobs has an order of growth of O(n^2), ie quadratic. This means that Checkbox becomes significantly slower as the number of jobs increases.

Please let me know if any of the resource commands take a significant amount of time to make sure that the problem is really with the gathering of jobs. Thanks!

[Marc Tardif (cr3)]

For your convenience, simply provide the output from running the following commands:

time /usr/share/checkbox/scripts/cpuinfo_resource >/dev/null
time /usr/share/checkbox/scripts/cdimage_resource >/dev/null
time /usr/share/checkbox/scripts/dpkg_resource >/dev/null
time /usr/share/checkbox/scripts/gconf_resource >/dev/null
time /usr/share/checkbox/scripts/lsb_resource >/dev/null
time /usr/share/checkbox/scripts/meminfo_resource >/dev/null
time /usr/share/checkbox/scripts/module_resource >/dev/null
time /usr/share/checkbox/scripts/package_resource >/dev/null
time /usr/share/checkbox/scripts/udev_resource >/dev/null
time /usr/share/checkbox/scripts/uname_resource >/dev/null

[Marc Legris (maaarc-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount)]

time /usr/share/checkbox/scripts/cpuinfo_resource >/dev/null

real 0m0.026s
user 0m0.016s
sys 0m0.012s

time /usr/share/checkbox/scripts/cdimage_resource >/dev/null

real 0m0.035s
user 0m0.028s
sys 0m0.008s

time /usr/share/checkbox/scripts/dpkg_resource >/dev/null

real 0m0.048s
user 0m0.028s
sys 0m0.020s

time /usr/share/checkbox/scripts/gconf_resource >/dev/null

real 0m0.109s
user 0m0.084s
sys 0m0.016s

time /usr/share/checkbox/scripts/lsb_resource >/dev/null

real 0m0.158s
user 0m0.136s
sys 0m0.020s

time /usr/share/checkbox/scripts/meminfo_resource >/dev/null

real 0m0.035s
user 0m0.024s
sys 0m0.008s

time /usr/share/checkbox/scripts/module_resource >/dev/null

real 0m0.022s
user 0m0.020s
sys 0m0.004s

time /usr/share/checkbox/scripts/package_resource >/dev/null

real 0m0.136s
user 0m0.116s
sys 0m0.020s

time /usr/share/checkbox/scripts/udev_resource >/dev/null

real 0m0.314s
user 0m0.208s
sys 0m0.112s

time /usr/share/checkbox/scripts/uname_resource >/dev/null

real 0m0.026s
user 0m0.024s
sys 0m0.004s

[Marc Legris (maaarc-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount)]

Even though the above commands execute quite fast, checkbox still takes about 10 minutes to "gather information about your system" on certain machines.

[Javier Collado (javier.collado)]

Please have a look at the merge proposal

[Ara Pulido (ara)]

Javier, is this still happening?

I have seen that you expressed some concerns about the patch proposed.

[Marc Legris (maaarc-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount)]

looks like it not the gathering resources that is causing the delay, but the sorting of tests. Maybe instead of displaying "gathering resources" it can instead display "compiling tests" or something similar.

[Daniel Manrique (roadmr)]

Recent improvements to Checkbox have pretty much taken care of this problem.

A job cache was implemented, this is used when doing job sorting, and basically implements the same functionality as Javier's merge proposal.

Then, file syncing/flushing is disabled during the gathering phase, for an additional speed boost.

As a result, the gathering phase for the default checklist went down from 47 seconds to a bit over 7 seconds.

I'll mark this Fix Released.