Backup restore fails on Ubuntu 14.04

Bug #1343344 reported by Greg Lucas
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
Fix Released
High
Nikhil Manchanda

Bug Description

Creating an instance from a backup fails on Ubuntu 14.04: there is an unhandled error from pexpect.close(force=True). The restore strategy needs to handle the pexpect error and continue on to 'sudo kill' the remaining processes as it does today.

The root cause appears to be a change in kill behavior when killing sudo processes between Ubuntu 12.04 and 14.04 such that the same pexpect call raises an error only on 14.04. The underlying change can be reproduced without python -- in the following examples note that kill succeeds on 12.04 and not 14.04:

* Ubuntu 12.04.4 LTS

$ sudo tail -f /var/log/syslog &
[1] 13407
$ ps -ef | grep tail
root 13407 1863 0 15:14 pts/0 00:00:00 sudo tail -f /var/log/syslog
root 13408 13407 0 15:14 pts/0 00:00:00 tail -f /var/log/syslog
$ kill -9 13407
[1]+ Killed sudo tail -f /var/log/syslog
$ ps -ef | grep tail
root 13408 1 0 15:14 pts/0 00:00:00 tail -f /var/log/syslog

* Ubuntu 14.04 LTS

$ sudo tail -f /var/log/syslog &
[1] 5915
$ ps -ef | grep tail
root 5915 5202 0 15:14 pts/0 00:00:00 sudo tail -f /var/log/syslog
root 5916 5915 0 15:14 pts/0 00:00:00 tail -f /var/log/syslog
$ kill -9 5915
-bash: kill: (5915) - Operation not permitted
$ ps -ef | grep tail
root 5915 5202 0 15:14 pts/0 00:00:00 sudo tail -f /var/log/syslog
root 5916 5915 0 15:14 pts/0 00:00:00 tail -f /var/log/syslog

Greg Lucas (glucas-q)
Changed in trove:
assignee: nobody → Greg Lucas (glucas-q)
Revision history for this message
Greg Lucas (glucas-q) wrote :

The pexpect exception:

  File "/usr/local/lib/python2.7/dist-packages/pexpect/__init__.py", line 774, in close
    raise ExceptionPexpect('Could not terminate the child.')

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to trove (master)

Fix proposed to branch: master
Review: https://review.openstack.org/108044

Changed in trove:
status: New → In Progress
Revision history for this message
Greg Lucas (glucas-q) wrote :

There are a couple other places trove calls pexpect.close and expects to terminate a process which may be a sudo process. The proposed patch applies the same fix there.

Revision history for this message
Greg Lucas (glucas-q) wrote :

See also: https://review.openstack.org/#/c/100353

That patch address the mysql restore issue in a way that does not use pexpect's close(force). We could consider using that approach -- mysqladmin shutdown -- for the mysql restore case, but I believe we still want to address the other places in the codebase where close(force) is used.

Revision history for this message
Greg Lucas (glucas-q) wrote :

Reviewing the changelogs for sudo, I've found what seems the likely point at which this behavior changed:

http://www.sudo.ws/sudo/changes.html --> 2012-02-06 entry:

* src/sudo.c:
 Set real uid to root before calling sudo_edit() or run_command() so
 that the monitor process is owned by root and not by the user.
 Otherwise, on AIX at least, the monitor process shows up in ps as
 belonging to the user (and can be killed by the user).

Changed in trove:
importance: Undecided → High
milestone: none → juno-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to trove (master)

Reviewed: https://review.openstack.org/108044
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=7395c077f40bfae5c224e86ab277a1a823b1c092
Submitter: Jenkins
Branch: master

commit 7395c077f40bfae5c224e86ab277a1a823b1c092
Author: Greg Lucas <email address hidden>
Date: Fri Jul 18 11:08:00 2014 -0400

    Handle exception from pexpect child.close

    When pexpect child.close(force=True) is called to close a sudo process, an
    exception may be raised on certain operating systems. Terminating a sudo
    process as a non-root user succeeds on Ubuntu 12.04 but fails on 14.04.

    Handle this exception and try to kill the child process using 'sudo kill'.

    See related change https://review.openstack.org/#/c/100353 that addresses
    another instance of the same issue.

    Change-Id: I4bd9e4b98479e33604545713337440ce81ca64ee
    Partial-Bug: 1343344

Changed in trove:
assignee: Greg Lucas (glucas-q) → Nikhil Manchanda (slicknik)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/100353
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=0a7c8d4858720e85d781f2b4d3affad835747134
Submitter: Jenkins
Branch: master

commit 0a7c8d4858720e85d781f2b4d3affad835747134
Author: Nikhil Manchanda <email address hidden>
Date: Fri Jun 13 11:16:00 2014 -0700

    Fixed restore to work correctly with pexpect

    Fixed an issue with restore where it was failing with the latest
    version of pexpect in Ubuntu 14.04.

    pexpect attempts to stop the temp mysqld process which fails due to
    a difference in privilege level (since the process was run with sudo).
    We now use mysqladmin shutdown to stop the temp mysqld process.

    Closes-Bug: 1343344

    Change-Id: Iba81648bb1654db97de876dd9a0dbfb04b8811d1

Changed in trove:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in trove:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in trove:
milestone: juno-3 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.