Failed to import ssh key

[Andreas Hasenack]

Image from Apr 25th, 2018. There is no build id in /etc/cloud, not sure how to better identify it.

I gave the installer my launchpad id ("ahasenack") for the ssh key import.

At the end, there was no /home/andreas/.ssh directory, and the logs confirmed that the import failed for some reason:
2018-04-25 20:47:38,799 - cc_ssh_import_id.py[DEBUG]: Importing ssh ids for user andreas.
2018-04-25 20:47:38,799 - util.py[DEBUG]: Running command ['sudo', '-Hu', 'andreas', 'ssh-import-id', 'lp:ahasenack'] with allowed return codes [0] (shell=False, capture=False)
2018-04-25 20:47:39,036 - util.py[WARNING]: Failed to run command to import andreas ssh ids
2018-04-25 20:47:39,036 - util.py[DEBUG]: Failed to run command to import andreas ssh ids
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ssh_import_id.py", line 104, in import_ssh_ids
    util.subp(cmd, capture=False)
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1957, in subp
    cmd=args)
cloudinit.util.ProcessExecutionError: Unexpected error while running command.
Command: ['sudo', '-Hu', 'andreas', 'ssh-import-id', 'lp:ahasenack']
Exit code: 1
Reason: -
Stdout: -
Stderr: -
2018-04-25 20:47:39,040 - util.py[WARNING]: ssh-import-id failed for: andreas ['lp:ahasenack']
2018-04-25 20:47:39,040 - util.py[DEBUG]: ssh-import-id failed for: andreas ['lp:ahasenack']
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ssh_import_id.py", line 79, in handle
    import_ssh_ids(import_ids, user, log)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ssh_import_id.py", line 107, in import_ssh_ids
    raise exc
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ssh_import_id.py", line 104, in import_ssh_ids
    util.subp(cmd, capture=False)
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1957, in subp
    cmd=args)
cloudinit.util.ProcessExecutionError: Unexpected error while running command.
Command: ['sudo', '-Hu', 'andreas', 'ssh-import-id', 'lp:ahasenack']
Exit code: 1
Reason: -
Stdout: -
Stderr: -

And later again:
2018-04-25 20:47:39,041 - handlers.py[DEBUG]: finish: modules-config/config-ssh-import-id: FAIL: running config-ssh-import-id with frequency once-per-instance
2018-04-25 20:47:39,041 - util.py[WARNING]: Running module ssh-import-id (<module 'cloudinit.config.cc_ssh_import_id' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ssh_import_id.py'>) failed
2018-04-25 20:47:39,041 - util.py[DEBUG]: Running module ssh-import-id (<module 'cloudinit.config.cc_ssh_import_id' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ssh_import_id.py'>) failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 792, in _run_modules
    freq=freq)
  File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
    return self._runners.run(name, functor, args, freq, clear_on_fail)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 187, in run
    results = functor(*args)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ssh_import_id.py", line 86, in handle
    raise elist[0]
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ssh_import_id.py", line 79, in handle
    import_ssh_ids(import_ids, user, log)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ssh_import_id.py", line 107, in import_ssh_ids
    raise exc
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ssh_import_id.py", line 104, in import_ssh_ids
    util.subp(cmd, capture=False)
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1957, in subp
    cmd=args)
cloudinit.util.ProcessExecutionError: Unexpected error while running command.
Command: ['sudo', '-Hu', 'andreas', 'ssh-import-id', 'lp:ahasenack']
Exit code: 1
Reason: -
Stdout: -
Stderr: -

[Andreas Hasenack (ahasenack)]

In another attempt, I tried a github key, but it also failed.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1766980

[Michael Hudson-Doyle (mwhudson)]

Was this the same system that you reported bug 1766286 about? Because the lack of proxy config could explain this.

In any case, the real fix for this is for subiquity to fetch the keys and shove the key material into the cloud init config rather than telling cloud-init to import the keys.

[Andreas Hasenack (ahasenack)]

On Wed, Apr 25, 2018, 19:01 Michael Hudson-Doyle <email address hidden>
wrote:

> Was this the same system that you reported bug 1766286 about? Because
> the lack of proxy config could explain this.
>

Same system, yes, but the proxy is not mandatory in this network. There is
unrestricted egress access. The proxy is there for caching purposes.

In any case, the real fix for this is for subiquity to fetch the keys
> and shove the key material into the cloud init config rather than
> telling cloud-init to import the keys.
>

It's a cloud-init bug then?

[Michael Hudson-Doyle (mwhudson)]

On 26 April 2018 at 10:41, Andreas Hasenack <email address hidden> wrote:

> On Wed, Apr 25, 2018, 19:01 Michael Hudson-Doyle <email address hidden>
> wrote:
>
> > Was this the same system that you reported bug 1766286 about? Because
> > the lack of proxy config could explain this.
> >
>
> Same system, yes, but the proxy is not mandatory in this network. There is
> unrestricted egress access. The proxy is there for caching purposes.

Ah OK.

In any case, the real fix for this is for subiquity to fetch the keys
> > and shove the key material into the cloud init config rather than
> > telling cloud-init to import the keys.
> >
>
> It's a cloud-init bug then?

I have no idea, the logs are very unhelpful :( but I guess that would be my
guess.

Cheers,
mwh

[Ryan Harper (raharper)]

I'm pretty sure this is because networking was not up yet:

i-info: +++++++++++++++++++++++Net device info++++++++++++++++++++++++
ci-info: +--------+------+-----------+-----------+-------+------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+-----------+-----------+-------+------------+
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: +--------+------+-----------+-----------+-------+------------+

That should have your nic.

I noticed in a previous install of subiquity, that it keeps the 'optional' flag on in the netplan configuration, which is fine for subiquity environment, but user environment should not use the optional flag.

Can you collect /etc/netplan/50-cloud-init.yaml to confirm?

[Andreas Hasenack (ahasenack)]

Here is my netplan config file:

# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        enp0s25:
            addresses: []
            dhcp4: true
            optional: true
    version: 2

[Ryan Harper (raharper)]

Yes, subiquity shouldn't mark the interface optional.

That means that network-online won't wait for the network to be up, and cloud-config/cloud-final run after network is online since many operations do use networking.

[Andreas Hasenack (ahasenack)]

Does this have any implication on packages I install afterwards on this
server and that depend on the network-online target?

On Thu, Apr 26, 2018, 12:26 Ryan Harper <email address hidden> wrote:

> Yes, subiquity shouldn't mark the interface optional.
>
> That means that network-online won't wait for the network to be up, and
> cloud-config/cloud-final run after network is online since many
> operations do use networking.
>
>
> ** Changed in: subiquity
> Status: Incomplete => Triaged
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1766980
>
> Title:
> Failed to import ssh key
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/subiquity/+bug/1766980/+subscriptions
>

[Steve Langasek (vorlon)]

> That means that network-online won't wait for the network to be up,
> and cloud-config/cloud-final run after network is online since many
> operations do use networking.

That is not the agreed meaning of network-online. network-online should only be true if at least one interface, and a default route, and DNS, are available; and only after all non-optional interfaces are up or timed out. In the case where you only have one interface at install time, 'optional' should make no difference (and if it does, that needs to be fixed elsewhere still). In the case where you have more than one interface, there's a design question of whether we should default to waiting for all interfaces to be up.

So I'm ok with us changing subiquity to make the interfaces not-optional on install as that seems to be the more robust solution.

[Ryan Harper (raharper)]

On Thu, Apr 26, 2018 at 12:40 PM, Steve Langasek
<email address hidden> wrote:
>> That means that network-online won't wait for the network to be up,
>> and cloud-config/cloud-final run after network is online since many
>> operations do use networking.
>
> That is not the agreed meaning of network-online. network-online should
> only be true if at least one interface, and a default route, and DNS,
> are available; and only after all non-optional interfaces are up or
> timed out. In the case where you only have one interface at install
> time, 'optional' should make no difference (and if it does, that needs
> to be fixed elsewhere still).

Are you suggesting that for a system with only one interface and that
it's marked optional networkd-wait-online should wait for it?

> In the case where you have more than one interface, there's a design
> question of whether we should default to waiting for all interfaces to be up.

I'm certainly not asking for all interfaces to be up; We're in
violent agreement that at least *one* should be up.
In this bug, there was only one, *and* it was marked optional.

>
> So I'm ok with us changing subiquity to make the interfaces not-optional
> on install as that seems to be the more robust solution.

I think whatever the user accepts or configures should NOT include optional.
That's sufficient to resolve this bug.

If there are additional nics that don't get configured in subiquity ui, those
could either stay unconfigured, or marked optional. We may want to include
them as optional in the config to communicate that the installer saw the
interfaces but the user chose not to configure it.

That's likely out of scope for this bug, but worth discussing long term.

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1766980
>
> Title:
> Failed to import ssh key
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/subiquity/+bug/1766980/+subscriptions

[Michael Hudson-Doyle (mwhudson)]

So the optional: true stuff is fixed in dailies but the bug as reported is also "superfixed" because the SSH key is fetched in the installer rather than on first boot now.