add JSON output for queued and running tests

[Andy Whitcroft]

The running.shtml has a three lots of useful data in it, queue lengths, running jobs and their output, and pending jobs.

It would be nice if this could be encoded in something more computer readable (as well) so that one could parse it to produce reports against it (think adt-ps). The queue at the bottom is already half encoded.

something like

{
'queue-depths':
'running-jobs':
'queues':
}

with the bits attached to them in order, or something would be interesting to me.

[Barry Warsaw (barry)]

Andy, can you give more information about the format of the keys and values you'd like to see in the JSON? For example, there are a couple of options.

For queue-depth, would you rather have nested dictionaries, or flattened values? E.g. we have distroseries and architecture, which we could arrange as {'trusty': {'amd64': 4, 'i386': 3}} or we could flatten that to {'trusty-amd64': 4, 'trusty-i386': ...} It sort of depends on what kind of unpacking the client is expecting to do because it's pretty easy to shuffle things around any way you want it.

As for running-jobs, how much of the information we have on hand do you want? We could essentially mirror the entire running_tests dictionary that we have internally. It currently maps:

package -> runhash -> release -> arch -> (params, duration, logtail)

E.g. do you want the logtail, which of course can get big

In any case, I'll submit a first draft of the patch to Pitti, and then we can refine it based on your feedback.

[Barry Warsaw (barry)]

Another question related to queue-depth. AFAICT, the 'queues' that you want would be exactly what's available in our queue_info, namely (afaict, a mapping from) release -> arch -> [packages]

You could calculate queue-depth from that, so unless you want the debci to collate that information, I think all you need is the 'queues' dictionary. Unless of course, I'm misunderstanding the OP!

[Barry Warsaw (barry)]

Pending feedback from Andy, here's a tentative patch.

[Andy Whitcroft (apw)]

@Barry -- the consumer here has yet to be written so as long as it is JSON and the information is in there I am sure we can get it out just fine. For queue-depth yes, if we have the list of jobs for each then we can trivially count them (as you do in the patch). The logtail is less obvious, if it isn't going to bloat the world I could see wanting that, but its no where close to a deal breaker either way.

I guess I wonder what runhash is in your description, but perhaps either deploy what you have, or indeed email a sample of the json from testing and I can try and parse it for my purposes and see if it is complete enough for that. Thanks for looking at this.

[Martin Pitt (pitti)]

@Barry: This has the right shape indeed, thanks for working on this! There's just one thing which I'd like to avoid for cleanliness: You first deep-copy the entire queue_info hash and then replace the long list of queued tests with their counts. This is really expensive. I suggest to rather build an entirely new queue_depths dict out of iterating over queue_info. But from Andy's response I take it it's not really necessary to do that even, we can just expose the queue contents and let the consumer count.

I'm mildly in favor of not publishing logtails on JSON. This is reeeeally not meant for machine parsing, just for humans, and they can become quite big.

[Barry Warsaw (barry)]

On Jan 20, 2016, at 12:13 PM, Andy Whitcroft wrote:

>@Barry -- the consumer here has yet to be written so as long as it is
>JSON and the information is in there I am sure we can get it out just
>fine. For queue-depth yes, if we have the list of jobs for each then
>we can trivially count them (as you do in the patch).

Cool, I think I will remove queue-depth and let the client count them.

>The logtail is less obvious, if it isn't going to bloat the world I could see
>wanting that, but its no where close to a deal breaker either way.

I agree with pitti that the logtails are probably going to be too huge and not
really useful for machine parsing. Let's leave those out.

>I guess I wonder what runhash is in your description, but perhaps either
>deploy what you have, or indeed email a sample of the json from testing
>and I can try and parse it for my purposes and see if it is complete
>enough for that. Thanks for looking at this.

It looks like runhash is a way to disambiguate between running tests for the
same package/release/arch combination. AFAICT, it's calculated from the test
parameters, but it's not evident from the code what in the params is
sufficient to disambiguate the runs. In any case, it seems useful to keep
since that mirrors the internal data, and you can just treat it as opaque.

I can do a simple test run and post the JSON that results, but it likely won't
be an accurate snapshot of the complex data set you'd see on production.
Still, we can land this now and once the infrastructure's in place, it should
be quite easy to tweak the JSON format.

[Barry Warsaw (barry)]

On Jan 20, 2016, at 12:49 PM, Martin Pitt wrote:

>@Barry: This has the right shape indeed, thanks for working on this!
>There's just one thing which I'd like to avoid for cleanliness: You
>first deep-copy the entire queue_info hash and then replace the long
>list of queued tests with their counts. This is really expensive. I
>suggest to rather build an entirely new queue_depths dict out of
>iterating over queue_info. But from Andy's response I take it it's not
>really necessary to do that even, we can just expose the queue contents
>and let the consumer count.

I started to go down that path, but to me, the code looked uglier, and since
the structure was essentially the same, a deepcopy made the code look cleaner
to me. It's a general problem with deeply nested dictionaries; they are
simply ugly to iterate over fully, especially when you're just replacing a
single item embedded down deep.

I'm also not entirely convinced it's more expensive than rebuilding a parallel
dictionary structure explicitly. While deepcopy() has more logic to handle
the generic structure case, it also uses the pickling protocol to effect the
copies, and for basic types, that'll be implemented in C. Of course, there's
only one way to know for sure, and that's to time it. But OTOH, this doesn't
seem to be a performance critical section, given that it's run once every 10
seconds anyway.

In any case, I agree that given Andy's response, we'll just remove
queue_depths and let the client calculate it.

>I'm mildly in favor of not publishing logtails on JSON. This is
>reeeeally not meant for machine parsing, just for humans, and they can
>become quite big.

+1

[Barry Warsaw (barry)]

Example contents of http://.../running.json

{"running_jobs": {"coreutils": {"": {"trusty": {"amd64": [{}]}}}}, "queues": {"trusty": {"amd64": ["coreutils", "gzip", "libpng", "libpng", "gzip", "coreutils", "coreutils", "gzip", "gzip", "libpng", "libpng", "libpng", "gzip", "coreutils", "gzip", "libpng", "gzip", "gzip", "coreutils"]}}}

[Barry Warsaw (barry)]

Patch, take 2

[Martin Pitt (pitti)]

Barry Warsaw [2016-01-20 15:09 -0000]:
> It looks like runhash is a way to disambiguate between running tests for the
> same package/release/arch combination. AFAICT, it's calculated from the test
> parameters, but it's not evident from the code what in the params is
> sufficient to disambiguate the runs.

The params is all that we currently have; indeed if you have multiple
runs of the exact same package with the exact same test parameters,
this will get confused as it will get alternating status updates for
two different testse. In practice this doesn't happen very often,
though.

If this becomes a problem, then the workers themselves need to
generate a random "test run ID" (an UUID or perhaps
hostname+instance/lxc name), which we can them use to replace the
runhash. This should be transparent to consumers, though.

TL/DR: I think it's necessary to add it to the JSON, but consumers
should treat it as an opaque string.

[Martin Pitt (pitti)]

Committed with slight changelog massage, and deployed. http://autopkgtest.ubuntu.com/running.json exists now. Thanks Barry!