Support Private PPAs

[Vincent Stehlé]

Software properties add-apt-repository currently does not support adding private PPAs.

software-properties should connect to the API and observe that it gets permission denied trying to read the ppa. Then it can reconnect to the API asking for authentication, which will open a browser window where you can do the openid ritual. Then using that token it ought to be possible for it to get the password etc.

ProblemType: BugDistroRelease: Ubuntu 12.04
Package: python-software-properties 0.82.4

[Vincent Stehlé (vstehle)]

Also, invoking with private-ppa gives an error:

root@vincent-desktop:~# add-apt-repository private-ppa:tiomap-dev/private-release
Error: 'private-ppa:tiomap-dev/private-release' invalid

[Martin Pool (mbp)]

To correct this (I think), software-properties would have to connect to the API and observe that it gets permission denied trying to read the ppa. Then it can reconnect to the API asking for authentication, which will open a browser window where you can do the openid ritual. Then using that token it ought to be possible for it to get the password etc.

[Robert Roth (evfool)]

Currently the apt-add-repository does not support adding private PPAs, and it already mentions this when you try to add a private PPA, so this bug as decribed in the title does not exist anymore. Based on the latest comments, I am updating this to a Wishlist bug to support Private PPA's with the method described in comment 3.
---
Ubuntu Bug Squad volunteer triager
http://wiki.ubuntu.com/BugSquad

[James Westby (james-w)]

Calling lp.people[lp.me.name].getArchiveSubscriptionURL(archive) once the archive has been found
will get the url to be used for private PPAs.

getArchiveSubscriptionURL

Get a text line that is suitable to be used for a sources.list entry.

It will create a new IArchiveAuthToken if one doesn't already exist.
Parameters
Parameter Value Description

ws.op

(required)

Fixed: getArchiveSubscriptionURL

archive

(required)
Link to a archive.

[Kip Warner (kip)]

This problem is almost 6 years old. Is anyone working on it?

[Stuart Bishop (stub)]

Private PPAs are accessed with basic auth using https: URLs to private-ppa.launchpad.net.

https://stub:<email address hidden>/teamname/ppaname/ubuntu

It would be great if add-apt-repository accepted this or a similar URL, use it to retrieve the signing key security and add it as a source.

[Julian Andres Klode (juliank)]

OK, so:

* add-apt-repository should support logging in to launchpad and adding private ppas

* private PPAs should be added with auth.conf.d for the auth data

* to avoid having to login to launchpad on a server, something like:

  - add-apt-repository should gain a --stdin switch that contains lines of repositories with
    options
  - add-apt-repository should gain a --remote/-r switch to specify a remote host, and do the auth
    locally and then run the --stdin one on the remote host.

[Carl Karsten (carlfk)]

please consider the CI case: headless process that runs periodically for months without user interaction.

something like: deploy a private key (or similarly sensitive secret) file and a publicly readable script (apt-add-repository...) and it just runs, no browser, no user, no typing.

[Dan Streetman (ddstreet)]

targeted to b/c/d/e, although all my initial work is on e only. It includes a large amount of refactoring that I don't think is appropriate to sru back, so b/c/d would get a much smaller update - if at all.

> * add-apt-repository should support logging in to launchpad and adding private ppas

added this functionality.

> * private PPAs should be added with auth.conf.d for the auth data

added this functionality.

one note on this - when a private ppa is removed, the auth file is not currently removed, i don't have the code removing it, but probably it should be; i'll look at adding that.

> * to avoid having to login to launchpad on a server, something like:
>
> - add-apt-repository should gain a --stdin switch that contains lines of repositories with
> options

have not implemented this yet; however add-apt-repository does support providing a full sources.list line, so it could be called one-line-at-a-time. however, the way just drops the line directly into the main sources.list file, and doesn't parse out the login:password to put into an auth.conf.d/ file.

> - add-apt-repository should gain a --remote/-r switch to specify a remote host, and do the auth
> locally and then run the --stdin one on the remote host.

have not implemented this yet

[Dan Streetman (ddstreet)]

my working git repo is
https://code.launchpad.net/~ddstreet/ubuntu/+source/software-properties/+git/software-properties/+ref/lp645404

test builds at
https://launchpad.net/~ddstreet/+archive/ubuntu/lp645404

[Dan Streetman (ddstreet)]

one note to mention on changing software-properties over to using launchpadlib for lp logins; launchpadlib will create cache files in ~ and when running as sudo ~ isn't changed (without -i or -H). This results in bug 1828208

launchpadlib is not the only thing to suffer from this feature of sudo; I'm going to start an email list discussion about ubuntu's handling of $HOME under sudo.

for now, if testing with my ppa build, add-apt-repository should be run with 'sudo -H' (or sudo -i), and it has a temporary check to prevent running under plain 'sudo'.

[Dan Streetman (ddstreet)]

moved my git repo over to
https://code.launchpad.net/~ddstreet/software-properties/+git/software-properties

[Dan Streetman (ddstreet)]

@carlfk, can you elaborate on:

> deploy a private key (or similarly sensitive secret) file and a publicly
> readable script (apt-add-repository...) and it just runs, no browser,
> no user, no typing

for a private ppa, the script will only generate 3 files (e.g. for ppa:ddstreet/private on disco):
1) /etc/apt/sources.list.d/ddstreet-ubuntu-private-disco.list
2) /etc/apt/auth.conf.d/ddstreet_ubuntu_private.conf
3) /etc/apt/trusted.gpg.d/ddstreet_ubuntu_private.gpg

you could just copy those directly to whatever system you want, and not have to run anything at all on the system (besides 'sudo apt update' and then upgrade/install). Are you asking for something different?

[Dan Streetman (ddstreet)]

> * add-apt-repository should support logging in to launchpad and adding private ppas

added this functionality.

> * private PPAs should be added with auth.conf.d for the auth data

added this functionality.

still does not remove this auth data when the corresponding sources.list entry is removed.

> * to avoid having to login to launchpad on a server, something like:
>
> - add-apt-repository should gain a --stdin switch that contains lines of repositories with
> options

implemented standard '-' to read a line from stdin

> - add-apt-repository should gain a --remote/-r switch to specify a remote host, and do the auth
> locally and then run the --stdin one on the remote host.

implemented this as well, although I am not sure this is the best design. It will fail if the remote add-apt-repository isn't at a newer version (i.e. that supports the '-' param). Additionally, the gpg has to be manually added, since (for private ppas) that can't be queried without logging into lp, which is what the remote session is trying to avoid.

I think simply creating the resulting files - sourceparts, netrcparts, and trustedparts - on the remote system is going to be more reliable and simpler than using the remote system's installed add-apt-repository to try to process things into place. Calling apt-config on the remote system to gather the right directories to place the files should be simple; I think the hardest part will be figuring out how to do it without multiple subprocess runs to call ssh to 1) gather apt-config on the remote system and then 2) scp/sftp or ssh create-from-stdin the 3 files, and then 3) make sure they are the correct mode.

[Julian Andres Klode (juliank)]

So multiple ssh subprocesses should not be a huge issue, you could after all use ControlMaster and stuff to reuse the connection.

That said, it seems reasonable to create a shell script, or a Python script and execute that on the remote from stdin.

python3 and python3-apt should be available on all systems I think to do config lookups, and the content of files can be embedded into a generated script in base64.

That said, it would be an entirely different code path which I think might be worse than it maybe not working if the remote is at an older version.

[Dan Streetman (ddstreet)]

> So multiple ssh subprocesses should not be a huge issue, you could after all
> use ControlMaster and stuff to reuse the connection.

nice, I hadn't used ControlMaster before; that's useful.

> That said, it seems reasonable to create a shell script, or a Python script and
> execute that on the remote from stdin.

the problem with that, or even with directly calling add-apt-repository on the remote system, is it needs to be called under sudo, and there is no guarantee the sudo call is passwordless on the remote system, so the request for password (stdin/stdout) needs to be provided to the user. I don't see any clean way of doing that, but also piping commands or a script into stdin for execution on the remote side, without significant complications, or pulling in an external lib like paramiko.

> That said, it would be an entirely different code path which I think might be worse
> than it maybe not working if the remote is at an older version.

Not sure how it would be worse. Apt is quite clear on what needs to be in place, and where, to add a new repo; place the apt source server conf in Dir::Etc::sourceparts/d, place the apt archive's gpg key in Dir::Etc::trustedparts/d, and place the apt archive login details (if any) in Dir::Etc::netrcparts/d. Is any of that incorrect, and/or will any of that ever change? If not, simply creating the proper files in-place on the remote host should be all that's needed; all the logic of figuring out the naming and content of the files takes place on the local host.

[Dan Streetman (ddstreet)]

I changed the MP over to 'Needs Review' as I think the code is in a state ready for review.

Note that I removed the --remote support from it; while that is definitely something that should be added, I believe it's separate and complex enough it should be done separately, later.

[Dan Streetman (ddstreet)]

To be usable under sudo, the changes for this also require bug 1862948

[Dan Streetman (ddstreet)]

gave up on the MR corresponding to this bug.

moved devel work into
https://code.launchpad.net/~ubuntu-support-team/software-properties/+git/software-properties

daily builds at:
https://launchpad.net/~ubuntu-support-team/+archive/ubuntu/add-apt-repository

if anyone is interested in this please feel free to test with that ppa and let me know if it works and/or if there are any issues.

[Sebastien Bacher (seb128)]

The bug is 10 years old and there is no indication it's a priority for bionic or focal, I'm going to wontfix for those series for now, we can always reopen if someone end up working on the issue

[Dan Streetman (ddstreet)]

> if someone end up working on the issue

*if* someone works on it?

https://code.launchpad.net/~ubuntu-support-team/software-properties/+git/software-properties/+merge/379928
https://salsa.debian.org/apt-team/python-apt/-/merge_requests/43
https://launchpad.net/~ubuntu-support-team/+archive/ubuntu/add-apt-repository

I have a feeling however, that it will be easier for me to just create an entirely new tool than to get my add-apt-repostiory and python-apt changes merged :-/

As far as this bug, I gave up on any reviews for my MR months ago (and my other MR has moved beyond this bug). Feel free to leave it wontfix.

[Julian Andres Klode (juliank)]

I was about ready to get back to reviewing this (or just saying meh, I don't see anything obviously wrong, so just upload it) 3 months ago when you declared that you gave up on the MR.

I don't see how creating a new tool is easier than just uploading your changes. Both have the same effect of ignoring feedback.

I do believe the changes are too big to be backported into older release series.

It's next to impossible to review the merge requests because it just keeps growing and growing, and mixing all sort of work together in one gigantic diff.

[Dan Streetman (ddstreet)]

@juliank, please provide any feedback you have; all I saw in the original MR was a single comment. The new MR has been open for 3 months with no feedback at all, until just today.

[Colin Watson (cjwatson)]

The history is a bit confusing, but I think this was landed in software-properties 0.99.0, and so this bug should probably be closed. Am I right?

[Junien Fridrick (axino)]

see also bug 1991553