sshd-ConsoleKit integration patch causes abrupt termination of multichannel sessions

[Nikita Shulga]

Attached to this report is a small app the reproduces the issue by creating multiple channels within one sessions. (I fist hit this issue using JSCH library)
Following information is printed to stdout if sshd is launched with -d flags (I've modified the sshd code to print PIDs next to debug level):
debug3(7526): monitor_read: checking request 156
debug3(7526): mm_answer_consolekit_register entering
debug1(7526): session_by_tty: session 0 tty /dev/pts/1
debug1(7573): server_input_channel_req: channel 2 request pty-req reply 1
debug1(7573): session_by_channel: session 2 channel 2
debug1(7573): session_input_channel_req: session 2 req pty-req
debug1(7573): Allocating pty.
debug3(7573): mm_request_send entering: type 28
debug3(7573): mm_pty_allocate: waiting for MONITOR_ANS_PTY
debug3(7573): mm_request_receive_expect entering: type 29
debug3(7573): mm_request_receive entering
debug1(7526): Unable to open session: The name org.freedesktop.ConsoleKit was not provided by any .service files
debug3(7526): mm_request_send entering: type 157
debug3(7526): mm_request_receive entering
debug3(7526): monitor_read: checking request 28
debug3(7526): mm_answer_pty entering
debug2(7526): session_new: allocate (allocated 2 max 10)
debug3(7526): session_unused: session id 2 unused
debug1(7526): session_new: session 2
mm_request_receive_expect: read: rtype 157 != type 29

From the log, it looks as if two processes use the same set of pipes to communicate to the monitor, one of them is sending MONITOR_REQ_CONSOLEKIT_REGISTER, while another MONITOR_REQ_PTY, and because they use same set of pipes, and because both processes use same pipe FDs, monitor reply is sometimes delivered to the wrong process.

I can reproduce the failure 90% of time using attached app, which expects first argument to be host name, second - login and the third - password, that are defaulted to ubuntu:ubuntu@localhost:
$ ./ssh-multisession
Connecting to localhost...Done!
Authenticating as ubuntu...Done.
Starting channel 0
Starting channel 1
ssh_channel_request_pty()=-1: Socket error: disconnected
Starting channel 2
ssh_channel_open_session()=-1: Writing packet: error on socket (or connection closed): Operation now in progress
Starting channel 3

The same script works fine if sshd is compiled without consolekit.patch:
./ssh-multisession debian-host debian debian
Connecting to 172.17.162.237...Done!
Authenticating as debian...Done.
Starting channel 0
Starting channel 1
Starting channel 2
Starting channel 3
Starting channel 4
Starting channel 5
Starting channel 6
Starting channel 7
Starting channel 8
Starting channel 9

P.S. To compile a test app one needs to have libssh-dev package installed on your system.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Stefan Spühler (bauruine)]

I can confirm this bug on Ubuntu 14.04 x64.

[Luke Reeves (luke-reeves)]

Also confirmed, this makes using configuration management tools like Ansible very problematic as they SSH will intermittently fail on some of the connections.

[Christopher May-Townsend (s-chrzs-q)]

Want to add my weight to this, been hitting this as part of Ansible, working on a workaround now (beyond just running Ansible one box at a time).

[Shay Rojansky (roji)]

Confirmed on my end too, when trying to use ansible.

[usterk (usterk-2)]

I'm having the same problem while using ansible:

- nagios: action=silence host={{ inventory_hostname }}
  delegate_to: "xxxxx"

[JanMalte (janmalte)]

Is there any information needed to fix it?

[brucellino (brucellino)]

This is now affecting me on 15.10 too.

[Colin Watson (cjwatson)]

I'll drop this patch in the next xenial upload.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssh - 1:6.9p1-3

---------------
openssh (1:6.9p1-3) unstable; urgency=medium

  * ssh_config(5): Fix markup errors in description of GSSAPITrustDns
    (closes: #799271).
  * Fix dh_install and dh_fixperms overrides to work properly with an
    architecture-independent-only build (closes: #806090).
  * Do much less work in architecture-independent-only builds.
  * Drop ConsoleKit session registration patch; it was only ever enabled for
    Ubuntu, which no longer needs it (LP: #1334916, #1502045).

 -- Colin Watson <email address hidden> Tue, 24 Nov 2015 22:48:53 +0000

[Tom (nothoover)]

Is there any way to get visibility as to when this will be added to the updates for Ubuntu releases, for instance for 15.10? I've just done a fresh install of that dist and upgraded everything, and it still appears to be at version 1:6.9p1-2ubuntu0.1.

[Andy Piper (andy-piper-gmail)]

Is this bug fix ever going to be released for 14.04 LTS?

[Linus van Geuns (nirkus)]

+1 on an ETA for 14.04
openssh 6.6p1-2ubuntu2.8 on trusty seems to be affected by this or a similar issue as well.

[Tony Caduto (snorkel12068)]

just a FYI, this is still broken on 14.04 LTS.
We recently updated from 12.04 and all our paramiko SSH python scripts that use multiple channels refused to work.

I had to update just SSH from 16.04 to get it to work as a quick fix, this is not idea since I now have a ton of broken packages when I try to do a do-release-upgrade.

[Tony Caduto (snorkel12068)]

is there a way to update SSH on 14.04 without massively messing up apt?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sebastian Schenzel (seb1281)]

I have the problem too!

On ubuntu server 16.04.2 LTS (GNU/Linux 4.8.0-56-generic x86_64)

[Dave (dlwb)]

I can confirm the existence of the problem.
Occured with Ansible on Ubuntu 16.04.3 LTS (with Ansible 2.4.0.0-1ppa~xenial).