[SRU] Authentication/Authorization broken due to GitHub platform changes

[Chai T. Rex]

[Impact]

 * Gist upload (arguable as the core function of the package) is not functioning. Package versions prior to 5.1.0 provide user's access token as a query (URL) parameter, however GitHub changes now require it to be provided as a HTTP(S) header: https://developer.github.com/changes/2019-11-05-deprecated-passwords-and-authorizations-api/#authenticating-using-query-parameters

 * --login is not functioning. Package versions prior to 6.0.0 use an authentication endpoint that has been shut down since November 2020: https://developer.github.com/changes/2020-02-14-deprecating-oauth-auth-endpoint/

[Test Plan]

 * rm ~/.gist # stored credentials
 * gist-paste --login

Currently (5.0.0-4 focal) fails; output:
Obtaining OAuth2 access_token from github.
GitHub username: username
GitHub password:
RuntimeError: Got Net::HTTPNotFound from gist: {"message":"Not Found","documentation_url":"https://docs.github.com/rest"}

Expected web-based OAuth; output:
Requesting login parameters...
Please sign in at https://github.com/login/device
  and enter code: DEAD-BEEF
Success! https://github.com/settings/connections/applications/402bac389df41f24c62f

 * echo 'class Test {}' > Test.java
 * gist-paste -f Test.java -t java -p -d 'Fast method tester' -R Test.java

Currently (5.0.0-4 focal) fails; output:
Error: Got Net::HTTPBadRequest from gist: {"message":"Must specify access token via Authorization header. https://developer.github.com/changes/2020-02-10-deprecating-auth-through-query-param","documentation_url":"https://docs.github.com/v3/#oauth2-token-sent-in-a-header"}

Expected randomly-generated Gist link; output:
https://gist.github.com/username/eed178872769488d84378b13de8bb698/raw

[Where problems could occur]

 * The SRU requires a rewrite of authentication workflow, with a new OAuth (web-based) approach.

   The `--login` invocation previously accepted two inputs over stdin, however it now waits for user to do carry out manual steps based on instructions displayed (opening a page in web browser, and entering a code, as visible in Test Plan above). Although automated scripts should not be invoking `--login`, as the relevant token is stored persistently in user's home, if in any case they do so then it could halt further processing of the script.

[Other Info]

 * These changes have been tested as part of package release on prior Ubuntu versions, as well as landing in Debian stable:
   - Gist 5.1.0-1 was published in Groovy (20.10) with relevant HTTP(S) header change.
   - Gist 6.0.0-1 was published in Hirsute (21.04) with relevant changes for OAuth workflow (--login).

[Christian Ehrhardt  (paelzer)]

Hi,
it seems that the individual change to overcome
https://developer.github.com/changes/2020-02-10-deprecating-auth-through-query-param/
would be
https://github.com/defunkt/gist/commit/635b1437a513e9a13367827ee3f74fbbdaa54aa8

But a bit more wide-reaching the old auth flow in general is also deprecated and going away (depending on your account type) already.
Therefore one most likely also wants all of:
https://github.com/defunkt/gist/pull/323

Because without the new Auth flow it seems logins via the tool are failing now as well:
Old:
gist-paste --login
Obtaining OAuth2 access_token from github.
GitHub username: ...
GitHub password:
Traceback (most recent call last):
...

New:
gist-paste --login
Requesting login parameters...
Please sign in at https://github.com/login/device
  and enter code: XXXX-YYYY
Success! https://github.com/settings/connections/applications/xxxxxxxxxxxxxxxxxxxx
root@i:~# gist-paste -f test -p -d 'test'
(type a gist. <ctrl-c> to cancel, <ctrl-d> when done)
Foo
https://gist.github.com/xxxxxxxxxxxxxxxxxxxxxxxxx

And yes those two changes are in v6.0 and later which matches Ubuntu 21.04 and later.

I'm not sure if this is more an SRU [1] or an item for the newly re-forming backports team [2].
I'd wonder if there are use cases left that still work fine in the current version and that might be broke/changed by upgrading - if that is the case then more backports. If not then an SRU might apply indeed.

P.S. I was only triaging this, but not working on it. So anyone else that comes by, feel free to work on it - there will be no collision with me :-)

[1]: https://wiki.ubuntu.com/StableReleaseUpdates
[2]: https://lists.ubuntu.com/archives/ubuntu-devel/2021-August/041587.html

[Valters Jansons (sigv)]

The patch for v5.0.0 (Focal) looks good to me, and test case passes locally -- I can log in and upload a Gist as expected.

I am currently not planning on providing a patch for v4.6.1 (Bionic).

[Chai T. Rex (chaitrex)]

Can someone please publish this to the universe repository? The old version is still the current version according to apt install and https://packages.ubuntu.com/focal/gist.

[Utkarsh Gupta (utkarsh)]

Sponsored the upload. Thank you for your work on this! \o/

[Brian Murray (brian-murray)]

Hello Chai, or anyone else affected,

Accepted gist into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/gist/5.0.0-4ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chai T. Rex (chaitrex)]

I installed it from focal-proposed. `gist --login` works through the new login method. After that, `gist-paste -pd 'Example file' /etc/apt/preferences.d/proposed-updates` creates a new private gist with a description of "Example file" that is a copy of the specified file.

$ apt-cache policy gist
gist:
  Installed: 5.0.0-4ubuntu1
  Candidate: 5.0.0-4ubuntu1
  Version table:
 *** 5.0.0-4ubuntu1 400
        400 http://archive.ubuntu.com/ubuntu focal-proposed/universe amd64 Packages
        400 http://archive.ubuntu.com/ubuntu focal-proposed/universe i386 Packages
        100 /var/lib/dpkg/status
     5.0.0-4 500
        500 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages
        500 http://archive.ubuntu.com/ubuntu focal/universe i386 Packages

[Launchpad Janitor (janitor)]

This bug was fixed in the package gist - 5.0.0-4ubuntu1

---------------
gist (5.0.0-4ubuntu1) focal; urgency=medium

  * Update workflows for authentication and authorization,
    due to GitHub endpoint deprecations (LP: #1940907):
    - d/patches/auth-header: Add patch.
      Use header-based authentication, instead of query-parameter-based.
    - d/patches/auth-oauth: Add patch.
      Use web OAuth flow, instead of removed Authorizations endpoints.
    - d/patches/webmock: Remove patch.
      Upstream change (in auth-oauth patch) incorporates the diff.
    - d/patches/series: Add two patches, remove one patch.

 -- Valters Jansons <email address hidden> Fri, 17 Sep 2021 09:03:16 +0300

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for gist has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.