Latest patch breaks command line 'restart all'

Bug #1786910 reported by Krister Landen on 2018-08-14
350
This bug affects 61 people
Affects Status Importance Assigned to Milestone
monit (Ubuntu)
Undecided
Unassigned

Bug Description

Running 'monit restart all' on command line was working yesterday 13.8.2018 but after latest security patch was installed it gives an error 'Invalid action “action=restart”'

Ubuntu 16.04.4 LTS
monit:
 Installed: 1:5.16-2ubuntu0.1

CVE References

Jeffrey Gelens (h1-jeffrey) wrote :

We're seeing the same issue. This breaks a lot on running machines.

Fernando Álvarez (fern4lvarez) wrote :

The root cause is the `action` parameter for the HTTP POST request to `localhost:2812/_doaction` being `action=restart`, whereas it should be just `restart`. So in the request payload you find something like `...&action=action=restart...`. This is obviously wrong and should be fixed asap.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in monit (Ubuntu):
status: New → Confirmed
Naresh Sherwal (naresh1919) wrote :

We are facing the same issue.

henry (henryoswald) wrote :

To confirm we are seeing this on all our servers which is causing quite a lot of problems for us. Details on how to roll this back would be welcome.

Fernando Álvarez (fern4lvarez) wrote :

@henryoswald for the time being, you need to revert the monit package to 1:5.16-2, try manually in a server with

    sudo apt-get install monit=1:5.16-2

If your servers are automatically provisioned with some third party tool (chef, puppet, bash scripts, etc) then you would need to update them accordingly.

Rui David (rfdavid) wrote :

Same here, several automated servers affected.

Sam Granieri (samgranieri) wrote :

Same here, this program broke a few things for me

Other actions are also affected:

$ monit restart tomcat
Invalid action "action=restart"

$ monit stop tomcat
Invalid action "action=stop"

$ monit start tomcat
Invalid action "action=start"

$ monit monitor tomcat
Invalid action "action=monitor"

$ monit unmonitor tomcat
Invalid action "action=unmonitor"

Kyle Schutt (kschutt) on 2018-08-14
description: updated
Kyle Schutt (kschutt) wrote :

Is there any update to this?

Encore Shao (encoreshao) wrote :

we have the same problem here!

taehun (tk0221) wrote :

Same issue
```
      01 Invalid action "action=stop"
      01 Invalid action "action=stop"
```

Derek Kniffin (dkniffin) wrote :

We have the same problem. It's affecting several of our projects and causing issues with our deployment process.

Kyle Schutt (kschutt) wrote :

For those of you using Chef with Ubuntu, the following worked for us as part of our deploy recipes:

package 'Install Monit 1:5.16-2' do
  package_name 'monit'
  version '1:5.16-2'
  options '--allow-downgrades'
end

Jason Brodie (drmcnasty) wrote :

As stated above you can roll back a version and it seems to be happy again.

sudo apt-get install monit=1:5.16-2

FedeX (fedex) wrote :

Same problem here, above workaround works for us too. Thanks!

Rafael Dalprá (rafaeldalpra) wrote :

Same problem here, @kschutt's workaround works here too. Thanks

Same problem here, Downgrading works fine.

Thanks @Jason Brodie

Jimmy Thrasher (jjthrash) wrote :

I'm seeing this issue as well. I believe the severity to be medium to high for the following reason:

- I have automatic security updates turned on, and I want to keep it that way
- I don't know of a way to pin the version of monit
- every morning my system upgrades monit to the bad version and I have to manually downgrade on multiple systems (Ansible makes it less painful, but human error is still a factor)

Peter Schiffer (pschiffe) wrote :

Jimmy: after you downgrade the monit package, you can pin it to the current version with command: `apt-mark hold monit`

Jimmy Thrasher (jjthrash) wrote :

@pschiffe Excellent suggestion thanks. I'll do that.

I will say, though, that now I have potential human error in the process of remembering to un-hold it when a fix for this package comes out. I can live with it, though.

MuniBilling (munibilling) wrote :

Another option if you want to run the latest version: The pre-compiled generic monit package from the monit project works OK without this problem, but Ubuntu stores the config files in a non-default location so you have to specify the path to /etc/monit/monitrc with each command.

MuniBilling (munibilling) wrote :

To add to my last comment, you can download the monit sourcecode here:
https://mmonit.com/monit/dist/monit-5.25.2.tar.gz

You can specify the non-default destination for the monitrc config file with this:
./configure --sysconfdir /etc/monit
./make

Then you can make this into a .deb file with:
checkinstall

Then you just `apt-get remove monit` to remove the bad monit installation and `dpkg -i monit_5.25.2-1_amd64.deb` to install the one you just built.

So this version of monit will accept commands like `monit unmonitor all`, etc. However, running `systemctl restart monit` will not start the daemon.

Alex C (acatighera) wrote :

Also encountered this issue, pschiffe solution seems to work temporarily for us.

taehun (tk0221) wrote :

There is unattended-upgrade running background and every its in `/etc/apt/apt.conf.d/50unattended-upgrades`

It does upgrade if
Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}";
        "${distro_id}:${distro_codename}-security";

is available

you can comment
${distro_id}:${distro_codename}-security"; # (do it manually before release to prevent incident like this)

if you check /var/log/unattended-upgrades/unattended-upgrades.log
you can find monit is upgraded.

This bug is tormenting, my servers do not stop sending emails to me, to warn me of this bug.
I have crontab that warns me every 3 minutes about this errors like:

Invalid action "action=start"
Invalid action "action=monitor"

I tryed downgrad withi:

sudo apt-get install monit=1:5.16-2 -y --allow-downgrades

but without success :/

JK (m0d) wrote :

How could this BS pass QA? All monit commands are broken, so even the most simple test should have failed!

This patch breaks a lot of servers and the only current workaround is to downgrade to a vulnerable version and either pin that version or disable unattended security upgrades. But "Importance" is still "Undecided" and no fix in sight... WTF?

Carlos Peñas (theist) wrote :

I think all the problem is in the latest CVE-2016-7067.patch which features this change like this:

- "%s",
+ "securitytoken=%s&action=%s",
+ token,

the %s comes from a var which already has an "action=" in it

I tried locally compile the package with a new patch with only this hunk:

--- monit-5.16.orig/src/control.c
+++ monit-5.16/src/control.c
@@ -449,7 +449,7 @@ boolean_t control_service_daemon(List_T
                          "Content-Length: %d\r\n"
                          "%s"
                          "\r\n"
- "securitytoken=%s&action=%s",
+ "securitytoken=%s&%s",
                          token,
                          strlen("securitytoken=") + strlen(token) + 1 +
                          StringBuffer_length(sb),

And the resulting package seems to work ok for me. I didn't tested it extensively. Also I'm just playing with source code, I do not know how to submit a proper patch.

notromda (mortonda) wrote :

ansible plays to downgrade and place on hold:

  tasks:
    - name: downgrade monit
      apt: name=monit=1:5.16-2 force=yes state=present
    - dpkg_selections:
        name: monit
        selection: hold

Carlos Peñas (theist) wrote :

This is the same patch I mentioned in comment #28

This works for me in a test environment, this is not extensively tested, however.

The attachment "deduplicate 'action=' in CLI http request" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Jimmy Thrasher (jjthrash) wrote :

I have successfully pinned all my relevant systems to the working version of monit. Now I have a dilemma:
- Do I want to have a secure system, or
- Do I want to be able to use monit (thereby allowing me to deploy code, etc)

I don't know how exposed I am now. If I'm not exposed much, then it's not high severity for me. Does anybody have a sense of how risky it is to be running this older version of monit?

Carlos Peñas (theist) wrote :

@jjtrash According to the changelog [1] and the Debian CVE database [2], it seems that monit CLI issues its commands to monit thru an HTTP server that can be accessible from outside. The security patch tries to leverage it by adding a CSRF token to the HTTP call. Without it may be possible to send commands to monit with a curl from outside.

But, by default this HTTP server, unless configured to do so, binds only to 127.0.0.1, in this case for a non-shared server should be safe.

* [1] http://changelogs.ubuntu.com/changelogs/pool/universe/m/monit/monit_5.16-2ubuntu0.1/changelog
* [2] https://security-tracker.debian.org/tracker/CVE-2016-7067

Jimmy Thrasher (jjthrash) wrote :

@theist Thanks, that helps.

Derek Kniffin (dkniffin) wrote :

It looks like theist provided a good patch. What is the next step necessary to get that patch integrated and release a new version? And about how long would that take? I'm debating whether I need to put in a better workaround on my servers, or if I can hold off until a fix is released. I'd even be willing to put in some effort to get this into a new version, but I have no idea what that entails.

TigerWolf (hiddentiger) wrote :

Trying to make a decision if we make a change to our Chef scripts to make this workaround permanent as there isn't much activity on this bug. We are holding off automated deployments as we were hoping for a fix.

Any idea on when this patch by theist will make it in?

Nazar Hussain (nazarhussain) wrote :

I am facing the same issue as well. Not with "all" command but with any service. e.g.

monit restart webserver

does not work now, it was working before.

Simon Tideswell (stideswell) wrote :

I'm also affected for all the same reasons mentioned above. Would be good if Canonical could resolve this ASAP.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers