gpg secret keys not migrated after upgrade to gnupg 2.1

[Jamie Strandboge]

After upgrading to gnupg 2.1 I can no longer see my keys in seahorse (Passwords and Keys in apps scope in unity) and evolution cannot find my gpg keys. Someone said this might be related to https://www.gnupg.org/faq/whats-new-in-2.1.html#autostart and https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796931

Reading the Debian bug I see someone mentioned that running this might help:
$ gpg2 --import < ./.gnupg/secring.gpg

I did that and gpg2 went through each of my private keys to import them. When done I logged out and back in and seahorse saw my keys. I'm not sure why gpg2 didn't prompt me before.

[Mario Limonciello (superm1)]

In trying to understand where the issue is, if you select the "View" menu in the seahorse app and change the "Show" setting from "Personal" to "All" do the keys start to show up?

[Jamie Strandboge (jdstrand)]

Removing the evolution and seahorse tasks.

[Mario Limonciello (superm1)]

Can you comment on what version of GPG you started from on this system? Was this system dist-upgraded a few times and did a GPG 1.x to 2.x transition at some point?

There should be a .gnupg/.gpg-v21-migrated file. Does that exist for you?

[Jamie Strandboge (jdstrand)]

This system started with vivid, upgraded to wily and upgraded to xenial a few months ago. I have .gnupg/.gpg-v21-migrated but it has the timestamp of when I ran 'gpg2 --import < ./.gnupg/secring.gpg'-- I don't know if it was created initially and retouched when I ran the import command later.

[dkg (dkg0)]

Mario pointed me to this bug, and i'm surprised that this happened. I also am not sure how to debug it because it's not something i've been able to reproduce myself, and it sounds like once it's fixed for someone, they have no incentive to go back and reproduce it themselves.

Can anyone provide a set of reproducible steps to make this happen?

For example, as a standard user, i can do:

 gpg --gen-key

and then the new key is visible via both "gpg --list-keys" and "gpg --list-secret-keys"

if i then do:

 gpg2 --list-keys

i can also see the new key. and if i do:

 gpg2 --list-secret-keys

then i see:

gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/home/testuser/.gnupg/secring.gpg' to gpg-agent
gpg: key 57B78432: secret key imported
gpg: migration succeeded

and subsequent attempts to use the gpg2 secret key work just fine.

do you have any keys on smartcards, any keys ancient or weirdly-formatted keys, or any other unusual configuration that might have triggered this problem?

without a way to reproduce the error, i'm kind of at a loss.

[Mario Limonciello (superm1)]

thanks for your comments daniel.

I've tried to reproduce this as well and couldn't either.
I started with a Xenial VM that had an older version of gnupg2 than 2.1 installed.
I generated a key, and verified I could see it with both gpg and gpg2 as well as in seahorse.

I then upgraded to gnupg2 2.1 and checked with gpg2 --list-secret-keys and the import process worked properly.

[Jamie Strandboge (jdstrand)]

I don't have any smartcards. I have three old private keys (1024D keys; one is revoked) that I don't use any more (and therefore are not unlocked) but 'gpg2 --import < ~/.gnupg/secring.gpg' noticed all of them and prompted me for a password with each.

[dkg (dkg0)]

In the default import process, i don't think i ended up needing to enter any passwords at all. I can understand why you might need to enter passwords when importing secret keys that weren't already in your keyring, in general, but it seems like if being able to import them cleanly (without password entry) works in one case, it ought to be able to work in the other too.

anyway, i'd like to help, but without any way to reproduce this behavior, i'm sort of stuck :(

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Timo Aaltonen (tjaalton)]

I'm seeing this as well on two machines that I use, broke enigmail

[Timo Aaltonen (tjaalton)]

both machines were xenial -> xenial upgrades, but .gnupg directory is older and has same origin on both (basically just copied to the laptop at some point)

I have a third system to test things on, and I'll try to reproduce by removing the migration file

[Timo Aaltonen (tjaalton)]

trying to import keys gives:

:: tjaalton@deckard:~> gpg2 --import < ./.gnupg/secring.gpg
gpg: key A88984DC: "Timo Aaltonen <email address hidden>" not changed
gpg: key A88984DC/A88984DC: error sending to agent: Permission denied
gpg: error building skey array: Permission denied
gpg: Total number processed: 3
gpg: unchanged: 1
gpg: secret keys read: 3

[Timo Aaltonen (tjaalton)]

and here's what happens after killing the agent and removing migration file

gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/home/tjaalton/.gnupg/secring.gpg' to gpg-agent
gpg: key A88984DC/A88984DC: error sending to agent: Permission denied
gpg: error building skey array: Permission denied
gpg: migration succeeded
gpg: key A88984DC: "Timo Aaltonen <email address hidden>" not changed
gpg: key A88984DC/A88984DC: error sending to agent: Permission denied
gpg: error building skey array: Permission denied
gpg: Total number processed: 3
gpg: unchanged: 1
gpg: secret keys read: 3

I'd say that it's too optimistic about the migration being successful

[Mario Limonciello (superm1)]

Since you mentioned this came from another box, can you show the permissions for the files in .gnupg? That seems the most likely reason for being unable to pull the files out.

[Timo Aaltonen (tjaalton)]

:: tjaalton@wilson:~/.gnupg> ls -al
total 1092
drwx------ 3 tjaalton tjaalton 4096 huhti 8 18:21 .
drwxr-xr-x 42 tjaalton tjaalton 4096 huhti 8 18:25 ..
-rw------- 1 tjaalton tjaalton 8081 maali 20 2015 gpg.conf
-rw-rw-r-- 1 tjaalton tjaalton 0 huhti 8 00:09 .gpg-v21-migrated
drw------- 2 tjaalton tjaalton 4096 maali 20 2015 private-keys-v1.d
-rw------- 1 tjaalton tjaalton 1669 maali 20 2015 public.key
-rw------- 1 tjaalton tjaalton 517605 maali 20 2015 pubring.gpg
-rw------- 1 tjaalton tjaalton 600 maali 17 22:44 random_seed
-rw------- 1 tjaalton tjaalton 7322 maali 20 2015 secring.gpg
srwxrwxr-x 1 tjaalton tjaalton 0 huhti 8 00:37 S.gpg-agent
-rw------- 1 tjaalton tjaalton 4520 maali 20 2015 trustdb.gpg

don't see anything wrong there

[dkg (dkg0)]

Over on https://bugs.launchpad.net/bugs/1565963, Timo Aaltonen has found
a repeatable scenario where the secret keyring has not been successfully
migrated properly when switching over to gnupg 2.1:

On Fri 2016-04-08 12:35:05 -0300, Timo Aaltonen <email address hidden> wrote:
> :: tjaalton@wilson:~/.gnupg> ls -al
> total 1092
> drwx------ 3 tjaalton tjaalton 4096 huhti 8 18:21 .
> drwxr-xr-x 42 tjaalton tjaalton 4096 huhti 8 18:25 ..
> -rw------- 1 tjaalton tjaalton 8081 maali 20 2015 gpg.conf
> -rw-rw-r-- 1 tjaalton tjaalton 0 huhti 8 00:09 .gpg-v21-migrated
> drw------- 2 tjaalton tjaalton 4096 maali 20 2015 private-keys-v1.d
> -rw------- 1 tjaalton tjaalton 1669 maali 20 2015 public.key
> -rw------- 1 tjaalton tjaalton 517605 maali 20 2015 pubring.gpg
> -rw------- 1 tjaalton tjaalton 600 maali 17 22:44 random_seed
> -rw------- 1 tjaalton tjaalton 7322 maali 20 2015 secring.gpg
> srwxrwxr-x 1 tjaalton tjaalton 0 huhti 8 00:37 S.gpg-agent
> -rw------- 1 tjaalton tjaalton 4520 maali 20 2015 trustdb.gpg
>
> don't see anything wrong there

It's a little unusual to have ~/.gnupg/private-keys-v1.d not be u+x, as
that would imply that the directory isn't listable. This is probably
causing problems for the gpg-agent.

When i test with this setup, i can verify that the migration doeesn't
happen properly, although .gpg-v21-migrated gets created anyway.

from a new user account, with gpg1 as 1.4.20 and gpg2 as 2.1.11, i ran
the following three commands:

  gpg1 --gen-key
  mkdir -m 0600 ~/.gnupg/private-keys-v1.d
  gpg2 --list-secret-keys

The final command returns an error code of 2 and produces these messages
to the terminal:

gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/home/demouser/.gnupg/secring.gpg' to gpg-agent
gpg: key C93913FC/C93913FC: error sending to agent: Permission denied
gpg: error building skey array: Permission denied
gpg: migration succeeded

I have no idea how this directory got the u+x bit cleared, but maybe
that's something that either:

 a) gpg-agent could clean up on its own, or

 b) should cause gpg-agent to not create the .gpg-v21-migrated marker file

wdyt?

     --dkg

[Timo Aaltonen (tjaalton)]

yeah, thanks for spotting that.. chmod u+x .gnupg/private-keys-v1.d fixed it. Must've done a hasty chmod 600 at some point in the past!

[dkg (dkg0)]

On Sat 2016-04-09 05:53:36 -0400, Werner Koch <email address hidden> wrote:
> On Sat, 9 Apr 2016 01:37, <email address hidden> said:
>
>> It's a little unusual to have ~/.gnupg/private-keys-v1.d not be u+x, as
>> that would imply that the directory isn't listable. This is probably
>> causing problems for the gpg-agent.
>
> Yes, gpg-agent provide commands to list private keys and we may
> eventually use that feature to speed up the --list-secret-keys command
> in certain cases.

Makes sense. At any rate, the lack of u+x appears to make gpg-agent
fail to do the initial key import.

>> I have no idea how this directory got the u+x bit cleared, but maybe
>> that's something that either:
>>
>> a) gpg-agent could clean up on its own, or
>
> That is a good idea. ~/.gnupg is anyway a property of GnuPG and thus
> gpg-agent should be allowed to change the permissions.

Shall i open an issue in https://bugs.gnupg.org/ about this?

   --dkg

[Mario Limonciello (superm1)]

Yes that would be great, thanks

On Sat, Apr 9, 2016, 14:05 dkg <email address hidden> wrote:

> On Sat 2016-04-09 05:53:36 -0400, Werner Koch <email address hidden> wrote:
> > On Sat, 9 Apr 2016 01:37, <email address hidden> said:
> >
> >> It's a little unusual to have ~/.gnupg/private-keys-v1.d not be u+x, as
> >> that would imply that the directory isn't listable. This is probably
> >> causing problems for the gpg-agent.
> >
> > Yes, gpg-agent provide commands to list private keys and we may
> > eventually use that feature to speed up the --list-secret-keys command
> > in certain cases.
>
> Makes sense. At any rate, the lack of u+x appears to make gpg-agent
> fail to do the initial key import.
>
> >> I have no idea how this directory got the u+x bit cleared, but maybe
> >> that's something that either:
> >>
> >> a) gpg-agent could clean up on its own, or
> >
> > That is a good idea. ~/.gnupg is anyway a property of GnuPG and thus
> > gpg-agent should be allowed to change the permissions.
>
> Shall i open an issue in https://bugs.gnupg.org/ about this?
>
> --dkg
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1565963
>
> Title:
> gpg secret keys not migrated after upgrade to gnupg 2.1
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/gnupg2/+bug/1565963/+subscriptions
>

[dkg (dkg0)]

On Sun 2016-04-10 08:39:48 -0400, Werner Koch wrote:
> On Sat, 9 Apr 2016 20:57, <email address hidden> said:
>
>> Shall i open an issue in https://bugs.gnupg.org/ about this?
>
> better do that.

ok, i've created this:

  https://bugs.gnupg.org/gnupg/issue2312

Thanks for the followup, Werner.

       --dkg

[Jon (kvc)]

dkg, you are a lifesaver as always. Setting u+x private-keys-v1.d fixed this for me as well.

In my case, I had to do a laptop restore. I kept my gpg keys on separate backup media - which I hadn't realized at the time was formatted as FAT32. I no doubt "fixed" my permissions upon restore incorrectly.

[Fergus Whyte (fergusw)]

Thanks dkg, I had this issue also and setting u+x private-keys-v1.d fixed it for me as well. As far as I know I never changed the permissions in the .gnupg directory but you never know, could have been years ago.

[Carlo Wood (carlo-alinoe)]

Hi, I have the same problem; I upgraded from ubuntu 16.04 to 18.04 (one LTS release to the next). To my astonishment afterwards I couldn't get into my bank accounts anymore because I can't decrypt my files!

The output of the original conversion is:

>gpg --decrypt digid.nl.gpg
gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/home/carlo/.gnupg/secring.gpg' to gpg-agent
gpg: key 29BD5E6C7C8DEF37: secret key imported
gpg: key 3232F9CFE80A9EC5: secret key imported
gpg: key 027353B08370CB05: secret key imported
gpg: migration succeeded
gpg: encrypted with RSA key, ID 6FD2C61D624ACAD5
gpg: decryption failed: No secret key

After that I tried running:

>gpg --import < /home/carlo/.gnupg/secring.gpg
gpg: key 29BD5E6C7C8DEF37: "TXXXXX3" not changed
gpg: key 29BD5E6C7C8DEF37: secret key imported
gpg: key 3232F9CFE80A9EC5: "<email address hidden>>" not changed
gpg: key 3232F9CFE80A9EC5: secret key imported
gpg: key 027353B08370CB05: "Carlo (XXXXXX) <email address hidden>" not changed
gpg: key 027353B08370CB05: secret key imported
gpg: Total number processed: 5
gpg: skipped PGP-2 keys: 2
gpg: unchanged: 3
gpg: secret keys read: 3
gpg: secret keys unchanged: 3

Note the 'skipped PGP-2 keys: 2' ?!

What, why? How do I get those back?!

[Seth Arnold (seth-arnold)]

On Thu, Aug 16, 2018 at 06:31:41PM -0000, Carlo Wood wrote:
> Note the 'skipped PGP-2 keys: 2' ?!
>
> What, why? How do I get those back?!

Hello Carlo, upstream GnuPG 2.x dropped support for the PGP-2 keys
several years ago:
https://gnupg.org/faq/whats-new-in-2.1.html#nopgp2

You can install the gnupg1 package to get an older version of GnuPG that
still supports the PGP-2 keys. (You may still need to go to some effort to
get gpg1 to work with such old keys or any content encrypted or signed
with them.)

Thanks