Private Repository Support for Requirements

[John Chittum]

We are investigating using lpcraft to run tests on our Python tooling. At this time, some of our Python tools install private libraries via git in requirements.txt

However, the container / vm running lpcraft does not have credentials available to pull private git repositories during run time.

* Are there plans to support credential binding locally to support private dependencies?
* Are there plans to support credential binding in launchpad servers to support private dependencies?

[Jürgen Gmach (jugmac00)]

Thanks for reaching out.

We have been asked about secrets management in various channels, so I think it is a good idea to have an issue for that and collect all requests.

I cannot give you an adhoc answer.

I will discuss this with the team and I try to motivate the other affected parties to add their opinion on this, so we can get a better picture whether and how this would fit on our roadmap.

Could you also please elaborate on "credential binding locally"?

We already support e.g. the `--secrets` CLI argument, see https://lpcraft.readthedocs.io/en/latest/cli-interface.html#lpcraft-run-optional-arguments

[John Chittum (jchittum)]

what exactly does `--secrets` do? In our case, we need an SSH key for the user within the build, so that a `pip install ${TOOL} @ ${PRIVATE_LAUNCHPAD_GIT_REPO} succeeds.

Credential binding in a gist could be a volume mount or file push that has a FROM and TO setup, so that credentials are placed in the "correct" places as required for a run. For Launchpad, that'd mean having

* a service available to safely store the credentials
* the ability to specify those credentials
* ability to pull those credentials at run time
* mounting / pushing of those credentials into the container, into the proper location

locally, depending on how this is run (i think it's lxc, right?) doing a bind-mount could be a simple solution, but for something like SSH keys, there'll need to be some user and permission changes likely.

[Isaac True (itrue)]

Thanks @jugmac00 for letting me know about this bug.

> We have been asked about secrets management in various channels, so I think it is a good idea to have an issue for that and collect all requests.

In our case, we were hoping for secrets management in order to be able to access private PPAs on Launchpad. Git repository access isn't that important for us, but I guess it would be a similar mechanism.

[Jürgen Gmach (jugmac00)]

As mentioned in the docs, via the `--secrets` command line option you can provide a file with key/value pairs for secrets.

https://lpcraft.readthedocs.io/en/latest/cli-interface.html#lpcraft-run-optional-arguments

So, imho - at least for running lpcraft locally - this should work already, via providing the SSH key as a value (in the secrets file) and using the `run` command or even better the `run-before` command to put the SSH key in the correct location. That is certainly only valid for a local run, and not very streamlined.

I wonder whether SSH-keys are the only way to install from a private Launchpad git repository, or maybe there is an easier way via an token? @cjwatson

And yes, we do use lxc.

[Jürgen Gmach (jugmac00)]

Oh, this looks promising:
https://help.launchpad.net/Code/Git#Generating_access_tokens

[Dimitri John Ledkov (xnox)]

1) git repo access
I sort of expect to be allowed to set per-branch/repo advanced ACLs and grant read-only or write access to the "ci-jobs", and then sort of expect magic to happen in launchpad (i.e. ephemeral HTTP GIT token is created, and deployed to lpcrafts, which then is able to access all the resources needed).

I expect it to be scoped to the owner of the git repository where lpcraft is running, and not disclose such tokens to merge-requests from other people.

2) PPA access
Similarly, i expect to be allowed to declare package-repositories (using snapcraft yaml syntax for repos), and then ephemeral access is generated and injected into my lpcraft build (something similar to add-apt-repository --login, where login permissions are used of the git repo $owner)

The basic principal, is that lpcraft repo owned by $person, should be able to have read-only access to git repos & PPAs, like the said $person. inside launchpad, automatically using ephemeral auto-generated tokens/secrets.

[Colin Watson (cjwatson)]

@xnox: This sort of thing probably isn't impossible, but it would require great care to avoid information leaks, because Launchpad mustn't give access to private resources as part of CI jobs that the people who own the repository couldn't get in some other way. This gets difficult to analyse when the owner is a team: what if some members of the team have access to a private resource and some don't? What if their access changes between the time the CI job was set up and the time that it runs? We've had similar problems elsewhere in the past. It's not easy to see how all the edge cases here would work, and unfortunately getting privacy right is all about the edge cases.

The usual practice in Launchpad is that the owner of the distribution/project that contains private resources is able to see who has access to them, and withdraw access if they need to (on the +sharing page). That isn't necessarily incompatible with what you're suggesting, but it does require quite a bit of thought.

In the shorter term, I'm much more comfortable with having a way to manage a generic secret dictionary of tokens that are passed through to the build, leaving the issuing of those tokens up to whoever's setting up the CI job, than I am with having Launchpad automatically issue the tokens itself. We do already have mechanisms for issuing reasonably-suitable tokens for both git repositories and PPAs.

[Dan Ryan (techalchemy)]

We (the SOSS team generally) have use-case 1 in common with @xnox -- specifically, we need to be able to access private repositories within CI builds. Whether this is automatic (I agree this is complex) or depends upon generating tokens and storing them in a secret store, we will definitely need this at some point. Automating read access would make life a *lot* easier.

We also have encountered a number of scenarios where we have had the need to store/access credentials for systems like Artifactory, which has currently all been managed with very project/distro specific launchpad backend configurations and which should be managed with secrets.