Latest patch breaks command line 'restart all'

[Krister Landen]

Running 'monit restart all' on command line was working yesterday 13.8.2018 but after latest security patch was installed it gives an error 'Invalid action “action=restart”'

Ubuntu 16.04.4 LTS
monit:
 Installed: 1:5.16-2ubuntu0.1

[Jeffrey Gelens (h1-jeffrey)]

We're seeing the same issue. This breaks a lot on running machines.

[Fernando Álvarez (fern4lvarez)]

The root cause is the `action` parameter for the HTTP POST request to `localhost:2812/_doaction` being `action=restart`, whereas it should be just `restart`. So in the request payload you find something like `...&action=action=restart...`. This is obviously wrong and should be fixed asap.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Naresh Sherwal (naresh1919)]

We are facing the same issue.

[henry (henryoswald)]

To confirm we are seeing this on all our servers which is causing quite a lot of problems for us. Details on how to roll this back would be welcome.

[Fernando Álvarez (fern4lvarez)]

@henryoswald for the time being, you need to revert the monit package to 1:5.16-2, try manually in a server with

    sudo apt-get install monit=1:5.16-2

If your servers are automatically provisioned with some third party tool (chef, puppet, bash scripts, etc) then you would need to update them accordingly.

[Rui David (rfdavid)]

Same here, several automated servers affected.

[Sam Granieri (samgranieri)]

Same here, this program broke a few things for me

[Clemens Fuchslocher (clemens-fuchslocher)]

Other actions are also affected:

$ monit restart tomcat
Invalid action "action=restart"

$ monit stop tomcat
Invalid action "action=stop"

$ monit start tomcat
Invalid action "action=start"

$ monit monitor tomcat
Invalid action "action=monitor"

$ monit unmonitor tomcat
Invalid action "action=unmonitor"

[Kyle Schutt (kschutt)]

Is there any update to this?

[Encore Shao (encoreshao)]

we have the same problem here!

[taehun (tk0221)]

Same issue
```
      01 Invalid action "action=stop"
      01 Invalid action "action=stop"
```

[Derek Kniffin (dkniffin)]

We have the same problem. It's affecting several of our projects and causing issues with our deployment process.

[Kyle Schutt (kschutt)]

For those of you using Chef with Ubuntu, the following worked for us as part of our deploy recipes:

package 'Install Monit 1:5.16-2' do
  package_name 'monit'
  version '1:5.16-2'
  options '--allow-downgrades'
end

[Jason Brodie (drmcnasty)]

As stated above you can roll back a version and it seems to be happy again.

sudo apt-get install monit=1:5.16-2

[FedeX (fedex)]

Same problem here, above workaround works for us too. Thanks!

[Rafael Dalprá (rafaeldalpra)]

Same problem here, @kschutt's workaround works here too. Thanks

[Tushar Chhabhaiya (tushar-chhabhaiya)]

Same problem here, Downgrading works fine.

Thanks @Jason Brodie

[Jimmy Thrasher (jjthrash)]

I'm seeing this issue as well. I believe the severity to be medium to high for the following reason:

- I have automatic security updates turned on, and I want to keep it that way
- I don't know of a way to pin the version of monit
- every morning my system upgrades monit to the bad version and I have to manually downgrade on multiple systems (Ansible makes it less painful, but human error is still a factor)

[Peter Schiffer (pschiffe)]

Jimmy: after you downgrade the monit package, you can pin it to the current version with command: `apt-mark hold monit`

[Jimmy Thrasher (jjthrash)]

@pschiffe Excellent suggestion thanks. I'll do that.

I will say, though, that now I have potential human error in the process of remembering to un-hold it when a fix for this package comes out. I can live with it, though.

[MuniBilling (munibilling)]

Another option if you want to run the latest version: The pre-compiled generic monit package from the monit project works OK without this problem, but Ubuntu stores the config files in a non-default location so you have to specify the path to /etc/monit/monitrc with each command.

[MuniBilling (munibilling)]

To add to my last comment, you can download the monit sourcecode here:
https://mmonit.com/monit/dist/monit-5.25.2.tar.gz

You can specify the non-default destination for the monitrc config file with this:
./configure --sysconfdir /etc/monit
./make

Then you can make this into a .deb file with:
checkinstall

Then you just `apt-get remove monit` to remove the bad monit installation and `dpkg -i monit_5.25.2-1_amd64.deb` to install the one you just built.

So this version of monit will accept commands like `monit unmonitor all`, etc. However, running `systemctl restart monit` will not start the daemon.

[Alex C (acatighera)]

Also encountered this issue, pschiffe solution seems to work temporarily for us.

[taehun (tk0221)]

There is unattended-upgrade running background and every its in `/etc/apt/apt.conf.d/50unattended-upgrades`

It does upgrade if
Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}";
        "${distro_id}:${distro_codename}-security";

is available

you can comment
${distro_id}:${distro_codename}-security"; # (do it manually before release to prevent incident like this)

if you check /var/log/unattended-upgrades/unattended-upgrades.log
you can find monit is upgraded.

[herculano souza (herculano-sou)]

This bug is tormenting, my servers do not stop sending emails to me, to warn me of this bug.
I have crontab that warns me every 3 minutes about this errors like:

Invalid action "action=start"
Invalid action "action=monitor"

I tryed downgrad withi:

sudo apt-get install monit=1:5.16-2 -y --allow-downgrades

but without success :/

[JK (m0d)]

How could this BS pass QA? All monit commands are broken, so even the most simple test should have failed!

This patch breaks a lot of servers and the only current workaround is to downgrade to a vulnerable version and either pin that version or disable unattended security upgrades. But "Importance" is still "Undecided" and no fix in sight... WTF?

[Carlos Peñas (theist)]

I think all the problem is in the latest CVE-2016-7067.patch which features this change like this:

- "%s",
+ "securitytoken=%s&action=%s",
+ token,

the %s comes from a var which already has an "action=" in it

I tried locally compile the package with a new patch with only this hunk:

--- monit-5.16.orig/src/control.c
+++ monit-5.16/src/control.c
@@ -449,7 +449,7 @@ boolean_t control_service_daemon(List_T
                          "Content-Length: %d\r\n"
                          "%s"
                          "\r\n"
- "securitytoken=%s&action=%s",
+ "securitytoken=%s&%s",
                          token,
                          strlen("securitytoken=") + strlen(token) + 1 +
                          StringBuffer_length(sb),

And the resulting package seems to work ok for me. I didn't tested it extensively. Also I'm just playing with source code, I do not know how to submit a proper patch.

[notromda (mortonda)]

ansible plays to downgrade and place on hold:

  tasks:
    - name: downgrade monit
      apt: name=monit=1:5.16-2 force=yes state=present
    - dpkg_selections:
        name: monit
        selection: hold

[Carlos Peñas (theist)]

This is the same patch I mentioned in comment #28

This works for me in a test environment, this is not extensively tested, however.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "deduplicate 'action=' in CLI http request" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Jimmy Thrasher (jjthrash)]

I have successfully pinned all my relevant systems to the working version of monit. Now I have a dilemma:
- Do I want to have a secure system, or
- Do I want to be able to use monit (thereby allowing me to deploy code, etc)

I don't know how exposed I am now. If I'm not exposed much, then it's not high severity for me. Does anybody have a sense of how risky it is to be running this older version of monit?

[Carlos Peñas (theist)]

@jjtrash According to the changelog [1] and the Debian CVE database [2], it seems that monit CLI issues its commands to monit thru an HTTP server that can be accessible from outside. The security patch tries to leverage it by adding a CSRF token to the HTTP call. Without it may be possible to send commands to monit with a curl from outside.

But, by default this HTTP server, unless configured to do so, binds only to 127.0.0.1, in this case for a non-shared server should be safe.

* [1] http://changelogs.ubuntu.com/changelogs/pool/universe/m/monit/monit_5.16-2ubuntu0.1/changelog
* [2] https://security-tracker.debian.org/tracker/CVE-2016-7067

[Jimmy Thrasher (jjthrash)]

@theist Thanks, that helps.

[Derek Kniffin (dkniffin)]

It looks like theist provided a good patch. What is the next step necessary to get that patch integrated and release a new version? And about how long would that take? I'm debating whether I need to put in a better workaround on my servers, or if I can hold off until a fix is released. I'd even be willing to put in some effort to get this into a new version, but I have no idea what that entails.

[TigerWolf (hiddentiger)]

Trying to make a decision if we make a change to our Chef scripts to make this workaround permanent as there isn't much activity on this bug. We are holding off automated deployments as we were hoping for a fix.

Any idea on when this patch by theist will make it in?

[TigerWolf (hiddentiger)]

Found bug link on monit tracker: https://bitbucket.org/tildeslash/monit/issues/766/running-monit-restart-all-on-command-line

[Nazar Hussain (nazarhussain)]

I am facing the same issue as well. Not with "all" command but with any service. e.g.

monit restart webserver

does not work now, it was working before.

[Simon Tideswell (stideswell)]

I'm also affected for all the same reasons mentioned above. Would be good if Canonical could resolve this ASAP.

[Rich Sutton (clippermadness)]

This is also affecting my company. We use monit on all boxes for process management and have had to pin on a downrev version.

Do we know if the package maintainer knows that this is an issue? Can we get some confirmation of that?

[grzegorz (strange3studio)]

Confirmed it broke monit the same way on all our live 16.04 servers.
Also em unable to re-monitor service that once went un-monitored due to failure attempts.
Gives the dreaded:
`Invalid action "action=monitor"`
message.
Haven't yet tested how that affects general reportability and recovereability of affected systems however.

[nyet (nyetwurk)]

This makes zero sense.

I see this in the changelog:
monit (1:5.16-2ubuntu0.1) xenial-security; urgency=medium

  * SECURITY UPDATE: CSRF vulnerability
    - debian/patches/CVE-2016-7067.patch: The following http services
      are no longer implemented for GET method and require CSRF
      protected POST: _doaction, _viewlog
    - CVE-2016-7067

 -- Eduardo Barretto <email address hidden> Fri, 10 Aug 2018 12:24:19 -0300

but monit shows CVE 2016-7067 already patched long before then:

https://bitbucket.org/tildeslash/monit/commits/c6ec3820e627f85417053e6336de2987f2d863e3?at=master

[nyet (nyetwurk)]

Is see that monit is up to 5.25.3

Updating to upstream would make more sense than doing a poorly written and untested backport to an ancient version.

[Eduardo Barretto (ebarretto)]

Thanks for reporting this bug and helping make Ubuntu better.

I'm sorry this affected you all.

I would like you to ask the reporter and all the involved people in the thread to always include the last person listed as Maintainer for the package (you can check this in the debian/changelog) in the bug tickets, otherwise the tickets might never get the attention of those responsible for it and we want to support you all.

Thanks Nye Liu for adding me to the ticket!

As you may know packages in Universe are community based and we're trying to give them security updates from time to time. Regressions may happen from time to time, and this was the case here.

Updating to upstream is not always an option as there might be ABI and API changes.

For now the temporary solution (and every time you find a problem in a package) is to downgrade the package.

I will be taking a look at it from now on and will release an update as soon as possible.

[nyet (nyetwurk)]

Question: is it better to cherry-pick monit's CVE patch or patch the existing patch with https://bugs.launchpad.net/ubuntu/+source/monit/+bug/1786910/comments/28?

They seem to take very different approaches.

Also, it looks like monit's official patch will not work against 5.16, it wasn't incorporated util 5.20

[Eduardo Barretto (ebarretto)]

Has anyone seen the same problem in Trusty (Ubuntu 14.04)?

[James Gregory-Monk (jamgregory)]

In our experience, Trusty appears to be fine - it is just Xenial that is affected.

[nyet (nyetwurk)]

Interestingly, monit 5.18 and up no longer works with the ansible monit module:
https://github.com/ansible/ansible/issues/29322

[Eduardo Barretto (ebarretto)]

Thanks to Carlos Peñas for proposing the fix.

Can anyone test the new version?
You can download it from here:
https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+packages

This new version if approved will be released on Monday, as we don't want to release today and not having anyone during the weekend to respond in case of a problem.

Thanks!

[Simon Tideswell (stideswell)]

I've tested (on a single host) and early indications are that the issue is resolved. If I notice any bad stuff I will report back.

[andrew bezella (abezella)]

did some quick testing (also on a single host) and the "Invalid action" error is not appearing with the 1:5.16-2ubuntu0.2 version.

[Adam Thorn (alt36)]

I've tried 1:5.16-2ubuntu0.2 and it looks to fix the reported issue - i.e., commands such as "monit status", "monit restart all", "monit restart all" work as intended, rather than producing the "Invalid action" error message

[Eduardo Barretto (ebarretto)]

Thanks for testing the package and giving feedback!
I really appreciate it.

So based on your feedback and on my tests, we just released monit 1:5.16-2ubuntu0.2 to the repository.
It should be available for upgrade in a few minutes depending on the mirrors.

If you encounter any problems, please do report and add me to the ticket.

Thanks again!

[Shao-Chieh Chiang (shaochieh.chiang)]

$ dpkg -l |grep monit
ii monit 1:5.16-2ubuntu0.2 amd64 utility for monitoring and managing daemons or similar programs

all those start, stop, monitor, unmonitor, restart actions against certain component is working fine just "monit restart all" is throwing this:
"cannot parse response"

[Eduardo Barretto (ebarretto)]

Hi shaochieh.chiang,

I appreciate you taking the time to report it and helping make Ubuntu better.

My tests didn't give the "cannot parse response", and from the feedback received above, it appears that no one faced this so far.

So could you give more information?
Which are the steps to reproduce the message you see?
How easy can you reproduce it?

Thanks

[Shao-Chieh Chiang (shaochieh.chiang)]

Us that thrown by your code? Under what circumstances will that be thrown?
I'll sift through the logs and find out. Do you have a way to turn on
verbose logs?

Thanks,

Don

Eduardo dos Santos Barretto <email address hidden> 於 2018年10月8日 週一
下午9:55 寫道：

> Hi shaochieh.chiang,
>
> I appreciate you taking the time to report it and helping make Ubuntu
> better.
>
> My tests didn't give the "cannot parse response", and from the feedback
> received above, it appears that no one faced this so far.
>
> So could you give more information?
> Which are the steps to reproduce the message you see?
> How easy can you reproduce it?
>
> Thanks
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1786910
>
> Title:
> Latest patch breaks command line 'restart all'
>
> Status in monit package in Ubuntu:
> Fix Released
>
> Bug description:
> Running 'monit restart all' on command line was working yesterday
> 13.8.2018 but after latest security patch was installed it gives an
> error 'Invalid action “action=restart”'
>
> Ubuntu 16.04.4 LTS
> monit:
> Installed: 1:5.16-2ubuntu0.1
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/monit/+bug/1786910/+subscriptions
>

[Shao-Chieh Chiang (shaochieh.chiang)]

from our /var/log/monit.log:

[UTC Oct 8 03:45:56] error : 'nginx' failed protocol test [HTTP] at [localhost]:80/api/wmsnode/services/health/ [TCP/IP] -- HTTP: Error receiving data -- Resource temporarily unavailable
[UTC Oct 8 03:45:56] error : Mail: No mail servers are defined -- see manual for 'set mailserver' statement
[UTC Oct 8 03:45:56] info : Adding event to the queue file /var/lib/monit/events/1538970356_e7f1f0 for later delivery
[UTC Oct 8 03:45:56] info : 'nginx' exec: /bin/bash
[UTC Oct 8 03:47:35] error : cannot parse response
[UTC Oct 8 03:47:40] info : 'strapd' stop on user request
[UTC Oct 8 03:47:40] info : Monit daemon with PID 629 awakened
[UTC Oct 8 03:47:40] info : Awakened by User defined signal 1

the following logs were produced right after I ran "monit restart all":
[UTC Oct 8 15:34:49] info : Processing queued event /var/lib/monit/events/1538970463_ecdad0
[UTC Oct 8 15:34:49] error : Mail: No mail servers are defined -- see manual for 'set mailserver' statement
[UTC Oct 8 15:34:49] error : Alert handler failed, retry scheduled for next cycle
[UTC Oct 8 15:34:49] error : 'nginx' failed protocol test [HTTP] at [localhost]:80/api/wmsnode/services/health/ [TCP/IP] -- HTTP error: Server
returned status 404
[UTC Oct 8 15:34:49] error : Mail: No mail servers are defined -- see manual for 'set mailserver' statement
[UTC Oct 8 15:34:49] info : Adding event to the queue file /var/lib/monit/events/1539012889_e7f1f0 for later delivery
[UTC Oct 8 15:35:07] error : cannot parse response

is it from somewhere in your code?

[Eduardo Barretto (ebarretto)]

Hi shaochieh.chiang,

Could you try to downgrade the package version as below:
sudo apt-get install monit=1:5.16-2

And see if you can reproduce the error?

I've also found this on monit bug tracker:
https://bitbucket.org/tildeslash/monit/issues/327

It might be related to what you're facing.

[Shao-Chieh Chiang (shaochieh.chiang)]

Hi Eduardo,

so we should roll back to 1:5.16-2 then? yes once rolled back that error is gone.

[Eduardo Barretto (ebarretto)]

Hi shaochieh.chiang,

Thanks for getting back to me.

I still need more information, how many services, processes and so on are you monitoring?
Can you share your monitrc configuration?

Your log also contain errors from nginx ... have you tried to solve them?

I'm still not convinced that your problem is related to the "restart all" problem reported here.

The "cannot parse response" error message comes from util.c:2072 (Util_parseMonitHttpResponse) and it comes up when the HTTP response is larger than 300 ... so a 404 (not found) or any other error response code could end up generating the "cannot parse response".

So I believe that there's some problem between your monit and nginx.

Since you are the only person reporting this problem so far, I also would like you to send the information directly to my email (you can get from my launchpad profile), as I am considering your problem as a different problem, not related to the main topic here.
And I don't want people to get confused by our discussion.

As soon as we get more information, we can open a new bug ticket here in launchpad to track your issue, or update this one if there's any relation.

Thanks