Need ways to "publish" big build results from test runs

[Paul Sokolovsky]

With support for gcc, etc. builds via CBuild integration, there's a need to publish 100+Mb artifacts. It was confirmed that native test results attachments do not scale to such sizes. There's support to reference an attachment by (external) URL, what's missing is support for pushing an artifact to some external location.

[Paul Sokolovsky (pfalcon)]

Some mail captures:

From: Andy Doan

On 11/27/2012 10:51 AM, Paul Sokolovsky wrote:
> Hello Validation team,
>
>
> Proceeding further with CBuild/LAVA integration work, it's time to
> think how to publish gcc, etc. binary tarballs as produced by the
> builds. Initially it seemed like an easy step, but the more I think
> about it, the more it seems like complicated to do "right".
>
> The tarballs in question are ~100Mb, so I guess there cannot be talk
> about publishing them as attachment to a result bundle. Let me know if
> that's ok though, because it would resolve all complexities below.

The attachment approach indeed seems like a bad idea.

> Otherwise, is there a way to set some custom data per user, which would
> be available to all jobs submitted by that user - in a secure manner?
> The idea, that would be an access credential to use for publishing (so
> far we usually use SSH privkey to auth to publishing host).
>
> If not, is there a way to pass such information during job submission,
> but again in a secure manner (i.e. it shouldn't be part of job
> definition, as that's dumped in plain text).
>
> Any other infrastructure in LAVA (like, pre-authed publishing host) or
> suggestions how to solve this?

We currently have an SSH key in place to pull restricted builds from
mombin. My idea after thinking on this for 60 seconds is that we create
a new "publish" action for our jobs. This publish option would then push
these files somewhere. In our validation lab specific setup we could
publish somewhere on mombin using our current ssh key.

Does this sound sensible?

[Paul Sokolovsky (pfalcon)]

From: Andy Doan

On 11/29/2012 04:34 AM, Paul Sokolovsky wrote:
> Hello,
>
> On Wed, 28 Nov 2012 08:31:41 +1300
> Michael Hudson-Doyle <email address hidden> wrote:
>
> []
>
>>> I am hoping we can find some time to improve this with an API on
>>> https://snapshots.linaro.org that would be authenticated directly,
>>> but we can't make promises on when that's going to be around.
>>
>> Sounds like all the more reason to keep the job file at the fairly
>> abstract "publish" level then.
>
> Yes, I also fully agree with this idea. And it also clear that we need
> to support different methods of publishing - test/devel and production,
> old and new, etc. So, publish section in job JSON def would accept
> identifier of publishing method, like "method"/"location"/"destination",
> whatever:
>
> {
> "command": "publish",
> "parameters": {
> "location": "snapshots",
> "pattern": "build/product/*/out/*.tar.*"
> }
> }

Seems about right - without me *really* thinking into the issue :)

> That location key also shouldn't be treated as just a hostname, but
> rather as identifier of publisher config, which can be for example a
> subdirectory like:
>
> publisher/<id>/:
> publish - this gets called by lava with the list of files to do
> actual publishing
> ssh_key - a file used by publish script, can be any other files as
> needed

I'm not a huge fan of this, but I don't really have a better suggestion
to offer.

> Of course, the above is just a guess how it can be done, please use
> whatever suits LAVA paradigm well, but it would be nice to get "devel"
> publishing to mombin with existing LAVA setup soon, and "production"
> publishing later, so some way of abstracting publisher support is
> definitely needed. Please let us now if Infrastructure can help with
> org side of this, like submitting a ticket or blueprint for this.

I think you are on the right track. I think the best way you could help
here, would be to submit a patch for this.

[Dave Pigott (dpigott)]

Hi Milo - is this all done? Can you please update the bug?

[Milo Casagrande (milo)]

Support for token-based publishing landed in snapshots|releases.l.o, and the lava side bits have been done too.
More details are here:

https://cards.linaro.org/browse/LAVA-597

It needs to be tested though how well it behaves with big files, but we can close this for now.