SSH AllowUsers Settings on Compute Node Causes Live Migration Failure

Bug #1899996 reported by Zhu Shengli
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Zhu Shengli

Bug Description

Description
===========
If sshd is configured with AllowUsers, for example:

~~~
AllowUsers 'heat-admin tripleo-admin'
~~~

then live migration will fail because `nova_migration` which is used as ssh user during live migration is not in the allow list.

Steps to reproduce

* Configure AllowUsers via SshServerOptions parameter
* Deploy the overcloud
* Try live migration

Expected result
===============
Live migration succeed.

Actual result
=============
Migration failed due to SSH connection failure.

Environment
===========
OSP 16

Logs & Configs
==============

Relevant heat template parameters.

```
parameter_defaults:
  SshServerOptions:
    AllowUsers: 'heat-admin tripleo-admin'
```

nova-compute.log on the source host:

```
2020-10-15 15:17:11.970 7 ERROR nova.virt.libvirt.driver [-] [instance: 7f700c41-12d6-4b4f-848d-27560e6b83a7] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+ssh://<email address hidden>:2022/system?keyfile=/etc/nova/migration/identity: Cannot recv data:
Received disconnect from 172.16.1.34 port 2022:2: Too many authentication failures^M
Disconnected from 172.16.1.34 port 2022: Connection reset by peer: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+ssh://<email address hidden>:2022/system?keyfile=/etc/nova/migration/identity: Cannot recv data:
```

Relevant `sshd_config` settings for `nova_migration_target` service.

```
$ cat /var/lib/config-data/puppet-generated/nova_libvirt/etc/ssh/sshd_config

...
...

Match LocalAddress 172.16.1.34 User nova_migration
    AllowTcpForwarding no
    AuthorizedKeysFile /etc/nova/migration/authorized_keys
    ForceCommand /bin/nova-migration-wrapper
    PasswordAuthentication no
    X11Forwarding no
Match LocalAddress !172.16.1.34
    DenyUsers nova_migration
```

Zhu Shengli (akarei)
summary: - SSH AllowUsers Settings on Compute Node Causes Live Migration Fail
+ SSH AllowUsers Settings on Compute Node Causes Live Migration Failure
tags: added: puppet-tripleo
tags: added: tripleo tripleo-heat-templates
description: updated
Revision history for this message
Zhu Shengli (akarei) wrote :

This problem is pretty difficult to debug, because there is absolutely no error message from the user interface(either Horizon Dashboard or command line) when you execute the live migration. The instance just stays on the original host with active status.

I think it's better to add AllowUsers to the match block

```
        $allow_name = 'nova_migration'

        ssh::server::match_block { 'nova_migration allow':
          name => $allow_name,
          type => $allow_type,
          order => 1,
          options => {
+ 'AllowUsers' => $allow_name,
            'ForceCommand' => $wrapper_command,
            'PasswordAuthentication' => 'no',
            'AllowTcpForwarding' => 'no',
            'X11Forwarding' => 'no',
            'AuthorizedKeysFile' => '/etc/nova/migration/authorized_keys'
          },
          notify => Service['sshd']
        }

```

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (master)

Fix proposed to branch: master
Review: https://review.opendev.org/758458

Changed in tripleo:
assignee: nobody → Zhu Shengli (akarei)
status: New → In Progress
Changed in tripleo:
importance: Undecided → High
milestone: none → victoria-3
tags: added: train-backport-potential ussuri-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on puppet-tripleo (master)

Change abandoned by Takashi Kajinami (<email address hidden>) on branch: master
Review: https://review.opendev.org/758458

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (master)

Reviewed: https://review.opendev.org/758458
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=7fb4d982dc51928e31397bfcf72d8f4f8c077ca4
Submitter: Zuul
Branch: master

commit 7fb4d982dc51928e31397bfcf72d8f4f8c077ca4
Author: Zhu Sheng Li <email address hidden>
Date: Thu Oct 15 23:51:39 2020 +0800

    Add explicit AllowUsers options for nova_migration

    If sshd on the compute nodes is configured with AllowUsers via Heat
    template, then nova_migation user could be rejected.

    Change-Id: I1a380cdcc325e1ae19fb9510a56a96eb503f51f5
    Closes-Bug: #1899996

Changed in tripleo:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.