gpg throws out my secret keys after upgrade to 18.04

[Phillip Susi]

I recently upgraded a server from 16.04 to 18.04 and gpg has started throwing out my secret keys. I even tried deleting the .gnupg directory and re-importing or just copying the secring from another working system, and it still throws out the secret keys.

Working system:

C:\Users\psusi\AppData\Roaming\gnupg>gpg --version
gpg (GnuPG) 2.0.28 (Gpg4win 2.2.5)
C:\Users\psusi\AppData\Roaming\gnupg>gpg -K
C:/Users/psusi/AppData/Roaming/gnupg/secring.gpg
------------------------------------------------
sec# 2048R/A70FB705 2011-12-13
uid Phillip Susi <email address hidden>
uid Phillip Susi <email address hidden>
ssb 2048R/51FEF1C9 2011-12-13
ssb 2048R/FA9EEEF9 2011-12-14
ssb 2048R/3348AAF0 2013-11-26
ssb 2048R/BDCC7F92 2013-11-26
ssb 2048R/9C8E5E51 2014-10-29
ssb 2048R/93A02CCD 2014-10-29
ssb 2048R/5CBBA516 2015-10-05
ssb 2048R/10850B71 2015-10-05
ssb 2048R/6100FE84 2017-01-06
ssb 2048R/0F60068B 2017-01-06

Broken system:

psusi@devserv:~$ gpg --version
gpg: WARNING: unsafe permissions on homedir '/home/psusi/.gnupg'
gpg (GnuPG) 2.2.4
psusi@devserv:~$ gpg -K
gpg: WARNING: unsafe permissions on homedir '/home/psusi/.gnupg'
/home/psusi/.gnupg/pubring.kbx
------------------------------
sec# rsa2048 2011-12-13 [SCA]
      1B49F933916A37A3F45A1812015F4DD4A70FB705
uid [ultimate] Phillip Susi <email address hidden>
uid [ultimate] Phillip Susi <email address hidden>

[Dimitri John Ledkov (xnox)]

what are the permissions of /home/psusi ? and /home/psusi/.gnupg ?

it is not safe if .gnupg; or the directory that contains it; are group|other writable.

new gpg uses keybox, whilst old gpg uses pub/secring.gpg. whenever new gpg was first executed against this .gnupg the old secring got auto converted to a keybox and from that point on the secrings & keybox no longer maintained in sync.

Please check .gnupg/private-keys-v1.d/* folder and if that contains expected subkeys. If not try to use old gpg to export the subkeys; and re-import them with a new gpg, such that they migrate into the keybox structure.

[Phillip Susi (psusi)]

The permissions are probably world readable because as I said, I tried deleting the whole directory and recreating it by hand, then copying the secring over from the working system. This was after deleting the whole directory and letting gpg recreate it, then importing the keys exported from a working system. Whether by import or auto conversion to the new format, gpg discards the private keys. This may be because the primary key is not included because I normally keep that offline.

[Dimitri John Ledkov (xnox)]

can you provide commands to generate test keys/subkeys, to recreate this?

i do not appear to loose any subkeys at all, and i use subkeys only by default, without master key.

[Phillip Susi (psusi)]

Generating a new keypair and exporting only the subkey and then importing it into 18.04 seems to work. There must be something particular about my existing keychain. I guess I'll have to start git bisecting.

[Phillip Susi (psusi)]

I noticed something different about the new version of gpg is that I get a full screen prompt for my password when importing, but the old version does not do that.

[Phillip Susi (psusi)]

So I cloned the upstream git repo and built 2.0.31 and it works. 2.2.4 does not. Anywhere in the 2.1 area in between seems to have some protection turned out to tell you not to use production keys with a development branch and it refuses to import *any* secret key. Any idea how to bypass this?

[Phillip Susi (psusi)]

So I went back to 2.0.31, deleted ~/.gnupg, imported my keys, made sure they all showed up, then upgraded to 2.1.1. It converted to the new keybox format, and... my private keys are all gone.

[Phillip Susi (psusi)]

Ahah, gpg -K -v shows them... it seems to think they are all expired. It lists the expiration date on my current key as 2018-1-6. I believe that was the *original* expiration date, but then I extended it. gpg 2.1 seems to be failing to recognize the extension.

[Dimitri John Ledkov (xnox)]

Is the public key, with the new expiry date, and that selfsig, imported?

Sounds like an upstream bug.

Can we replicate that bug now with fresh keys? e.g. make key past-expiry (possibly under faketime), make subkey, extend master key expiry, export subkeys, import subkeys, see them expired? or something like that?

Re: full screen stuff, that is mandatory usage of pinentry/gpg-agent.

[Phillip Susi (psusi)]

Here is the upstream bug report, which launchpad apparently does not recognize:

https://dev.gnupg.org/T3101

The workaround is to re-import only the public key, or in my case, I just did a --recv-keys to fetch it from keyserver.ubuntu.com and that pulled in the updated selfsig.

I understand that it now uses gpg-agent, but what I don't understand is why it bothers to do so just to import. It doesn't need to decrypt the key at that point; only copy it into the keyring.

[Dimitri John Ledkov (xnox)]

So yeah, my suggestion to reimport public key was right. It is an odd corner case. I think it has to do with something how the upstream choose to reorganize public & secret keyrings; such that the split of which bits need to be where is now different, and the export of old is either incomplete for what the new one wants or the new one does import all the right things during migration.

I think it is prudent to refresh / have an up to date public key, wherever private keys are. I've seen this trip up a few people, but it is a bit of a self-service that one must do when migrating to 2.1 unfortunately.

It could be "fixed" by automatic refreshing of public keys.... but we are not going to do that.

[Phillip Susi (psusi)]

Yea, the tripping point for me was that I was importing a single file with both private and public keys in it, and that wasn't fixing it. The import has to be of *only* the public keys. Hopefully upstream will fix that part so the upgrade ( which automatically does an export/import ) will then work correctly.

[Dimitri John Ledkov (xnox)]

Oh wow. fail. I'm not sure it's worth tracking here, given it is an upstream issue. I guess we can set this bug to trianged.

[Phillip Susi (psusi)]

Would be nice if launchpad understood their bug tracker so it would link. Whenever they do fix it we may want to cherry pick it so this doesn't bite other people upgrading to 18.04.