RabbitMQ default user's password isn't changed

[Jimmy McCrory]

The default RabbitMQ user, guest, has administrator privileges through the management plugin and a very simple password by default.

[Ian Cordasco (icordasc)]

Here's an untested patch for master, I'm working on backporting that to kilo, juno and icehouse.

[Ian Cordasco (icordasc)]

The above patch works for master & kilo.

And here's an untested patch for juno & icehouse. I'm going to start spinning up an AIO without the patch to confirm the issue and then I'm going to apply the patch and re-run the rabbitmq playbooks to confirm this fixes it.

[Kevin Carter (kevin-carter)]

When running on a multi node environment that was already clustered the removal of this user on one node results in the remove of the user on all nodes and due to the speed at which this task is being executed there is a race condition once the user lookup has been completed. We are likely going to need to add something like "failed_when: false" to the task to simply all it to always continue regardless of the task state.

TASK: [rabbitmq_server | Ensure default rabbitmq guest user is removed] *******
changed: [aio1_rabbit_mq_container-5d9b0cfc]
failed: [aio1_rabbit_mq_container-e8e4279d] => {"cmd": "/usr/sbin/rabbitmqctl -q -n rabbit delete_user guest", "failed": true, "rc": 2}
stderr: Error: no_such_user: guest

msg: Error: no_such_user: guest
failed: [aio1_rabbit_mq_container-bb27526c] => {"cmd": "/usr/sbin/rabbitmqctl -q -n rabbit delete_user guest", "failed": true, "rc": 2}
stderr: Error: no_such_user: guest

msg: Error: no_such_user: guest

FATAL: all hosts have already failed -- aborting

[Ian Cordasco (icordasc)]

I just came to the same conclusion Kevin

[Ian Cordasco (icordasc)]

Here's the latest set of patches with "failed_when: false"

[Christopher H. Laco (claco)]

Swallowing failures seems bad[ish]. Can we pin this task to infra[0] and run it post clustering?

[Kevin Carter (kevin-carter)]

Sadly the rabbitmq user module doesn't return data consistently in both a success and failure scenario so there's no easy way to intelligently allow the module to fail if there is an issue. In this case, because we're looking to simply nuke the guest user, I'm thinking the `failed_when: false` option is the best way forward but I'm also open to a better way if we can think of one.

Data from the module for success and failure:
ok: [aio1_rabbit_mq_container-bb27526c] => {
    "var": {
        "rabbitguest": {
            "cmd": "/usr/sbin/rabbitmqctl -q -n rabbit delete_user guest",
            "failed": false,
            "failed_when_result": false,
            "invocation": {
                "module_args": "",
                "module_name": "rabbitmq_user"
            },
            "msg": "Error: no_such_user: guest",
            "rc": 2,
            "stderr": "Error: no_such_user: guest\n",
            "stdout": "",
            "stdout_lines": []
        }
    }
}
ok: [aio1_rabbit_mq_container-5d9b0cfc] => {
    "var": {
        "rabbitguest": {
            "changed": true,
            "failed": false,
            "failed_when_result": false,
            "invocation": {
                "module_args": "",
                "module_name": "rabbitmq_user"
            },
            "state": "absent",
            "user": "guest"
        }
    }
}
ok: [aio1_rabbit_mq_container-e8e4279d] => {
    "var": {
        "rabbitguest": {
            "cmd": "/usr/sbin/rabbitmqctl -q -n rabbit delete_user guest",
            "failed": false,
            "failed_when_result": false,
            "invocation": {
                "module_args": "",
                "module_name": "rabbitmq_user"
            },
            "msg": "Error: no_such_user: guest",
            "rc": 2,
            "stderr": "Error: no_such_user: guest\n",
            "stdout": "",
            "stdout_lines": []
        }
    }
}

[Ian Cordasco (icordasc)]

> Swallowing failures seems bad[ish]. Can we pin this task to infra[0] and run it post clustering?

We could

[Ian Cordasco (icordasc)]

We could move with failed_when and perhaps make the module better upstream?

[Christopher H. Laco (claco)]

Even host pin, the second run would fail. derp. the only way would be to check for the user, then conditional, which is likely overkill.

[Kevin Carter (kevin-carter)]

the task will succeed on a second run as is and on a new deployment it seems to work. its only in the case of an already cluster environment where this fails. swallowing failures is bad however it would seem that the upstream module needs work and IMO i think we start there to make this better for everyone however in the mean time to fix this I say we fire and forget but maybe add a comment to the role indicating why and that we intend to revise when the module is made better upstream.

[Christopher H. Laco (claco)]

NM. My brain was on Kevins use of cmd, but forgot we use the module. So yeah, seems like host pin in the setup would do it.

[Kevin Carter (kevin-carter)]

i think that we can do what claco says with two tasks one with a when condition pinning to the first host and the others excluding the first host.

Example:

- name: Ensure default rabbitmq guest user is removed
  rabbitmq_user:
    user: guest
    state: absent
  when: inventory_hostname == rabbitmq_all[0]
  tags:
    - rabbitmq-user

- name: Ensure default rabbitmq guest user is removed
  rabbitmq_user:
    user: guest
    state: absent
  when: inventory_hostname != rabbitmq_all[0]
  tags:
    - rabbitmq-user

This would side step the race and make sure that the user is eradicated on all hosts no matter how they we're setup.

[Kevin Carter (kevin-carter)]

@claco applying patch,

Download patch to /tmp/rabbit-patch.diff

cd /opt/os-ansible-deployment
git apply /tmp/rabbit-patch.diff
# run rabbit playbooks.

[Christopher H. Laco (claco)]

Sounds like Ian was able to test the patches in up to date environments. In the old lab we have access to atm, it's a tad older, and complaint about "ERROR: rabbitmq_user is not a legal parameter in an Ansible Playbook" even though the patch applied cleanly. I put that square on an old install.

Aside, we have a manual mitigation until things are solid if we need it (and thank you rabbitmqctl for crap exit codes):

ansible rabbit[0] -m shell --args="/usr/sbin/rabbitmqctl -q -n rabbit delete_user guest; true"

We tested this with a login test before and after.

[Ian Cordasco (icordasc)]

Based on the docs, the rabbitmq_user module should absolutely be there (http://docs.ansible.com/rabbitmq_user_module.html#synopsis "New in version 1.1")

[Kevin Carter (kevin-carter)]

+1 Ian,

 I've looked back through the various releases and there is no reason that anyone should be running ansible < 1.5.0 and running OSAD which the `rabbitmq_user` module is definitely there in any Ansible version that we'd support [ https://github.com/ansible/ansible/tree/release1.5.0/library/messaging ]. That said, If there is an issue with ansible for some reason or another the ansible install can be repaired with the script `./scripts/bootstrap-ansible.sh`

[Kevin Carter (kevin-carter)]

A private CVE has been requested

[Andy McCrae (andrew-mccrae)]

I've applied the patch it works fine except for 1 change:
The "when" lines in the change need to be as follows:

when: inventory_hostname == groups['rabbitmq_all'][0]
when: inventory_hostname != groups['rabbitmq_all'][0]

Otherwise you get:

fatal: [node1_rabbit_mq_container-7a62b615] => error while evaluating conditional: inventory_hostname == rabbitmq_all[0]

Still need to test the icehouse/juno version but LGTM.

[Kevin Carter (kevin-carter)]

CVE-2015-4708 has been assigned, the NDB has not be updated at this time.

[Kevin Carter (kevin-carter)]

Good catch Andy. We'll need to iterate on that.

[Christopher H. Laco (claco)]

Also, from a Juno/Icehouse impact, removing guest outright breaks the RabbitMQ maas checks.

https://github.com/rcbops/rpc-openstack/issues

We should be able to just pass in the password from the variables? https://github.com/rcbops/rpc-openstack/blob/master/rpcd/playbooks/roles/rpc_maas/tasks/local.yml#L307

[Kevin Carter (kevin-carter)]

Kilo master patch

[Kevin Carter (kevin-carter)]

Icehouse juno patch

[Kevin Carter (kevin-carter)]

Everyone. please review the latest patches when you can.

[Jimmy McCrory (jimmy-mccrory)]

The latest Juno patch is erroring out for me.
TASK: [rabbit_user | Ensure default rabbitmq guest user is removed] ***********
fatal: [infra1_rabbit_mq_container-b7893314] => error while evaluating conditional: inventory_hostname == groups['rabbitmq_all'][0]

Also, for the maas rabbitmq check will the 'Ensure rabbitmq user' task need to tag that user with an administrator role?

[Andy McCrae (andrew-mccrae)]

For juno we don't need the when: line in the role itself.

The way it was in juno we call the role from within a play that only targets rabbit[0].
So these when: statements are redundant for juno - Should be good to just remove them.

[Kevin Carter (kevin-carter)]

Patch updated to run the rabbit_user command in the bootstrap process as a "post_task" instead of within the role. This should ensure that we're not running up against the race condition from before.

[Kevin Carter (kevin-carter)]

@jimmy im not sure that the openstack user needs to be an administrator to monitor the cluster however I'll see about trying to get someone to confirm.

[Andy McCrae (andrew-mccrae)]

It does need administrator privileges unfortunately - I'm creating a patch for the rpc-openstack side work as this is rackspace specific, this will create a new user with random password and administrator privileges.

It's probably better to have a monitoring specific user regardless, but it does require administrator privileges.

[Christopher H. Laco (claco)]

What's the upgrade impact to this? Will we need to run pwgen again to fill in the new users password, or update anything from the prior version to this version?

[Andy McCrae (andrew-mccrae)]

@claco, for the rpc-openstack maas side?
Well our maas checks don't update - so you would need to remove the existing rabbit checks and re-run the maas setup play, after adding the password to user_extras_secrets.yml and populating it.

Aside from that the maas-setup play handles the user creation etc.

There isn't anyway around that regardless, since the maas checks dont update.

[Kevin Carter (kevin-carter)]

I guess from the perspective of simplicity we could assign the administrator tag to the openstack user which would allow the maas checks to be deployed with the changed user and credentials however in review I'm in agreement with Andy that the monitoring user should be different than the user that the OpenStack cluster relies on. If the checks need to be removed/replaced anyway adding an additional task to create a use shouldn't be so terrible and we can move that additional monitoring user task down into the rabbitmq/maas checks play instead of attempting to work into the rabbit roles for juno/icehouse.

[Andy McCrae (andrew-mccrae)]

I'd say we shouldn't do that - I think it makes sense to have a separate "maas" user for this in anycase, and if the openstack services don't require administrator any deployments that arent using "rax" specific maas plugins won't require this admin user.

The overhead is pretty small, just create an additional user, and I think this separates concerns so is a good idea in anycase. (My 2c at least). The new user is created with a random password as the openstack user is, so from that perspective it is the same.

[Christopher H. Laco (claco)]

I act not that I'm married to any specific solution, but that we need to make sure it gets documented in the release notes that a delete/recreate will need to happen on the check, etc.

[Andy McCrae (andrew-mccrae)]

Agree with the tag, that will need to be there regardless of the solution.

[Andy McCrae (andrew-mccrae)]

This is the rpc-openstack (github) patch for master/juno. Will need to be made as a PR in github not gerrit.

[Andy McCrae (andrew-mccrae)]

This is the juno/icehouse patch for the maas change. NB As before the rabbit monitor will need to be manually removed/changed and the new vars will need to be copied over (they are in the user_variables file for juno/icehouse, and in the user_extras_secrets/variables in kilo/master).

This patch goes in os-ad since the maas work had not been removed at the point of juno/icehouse.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/198786
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=7dee99f3941ff7e3326858742c0944b5caa07b70
Submitter: Jenkins
Branch: kilo

commit 7dee99f3941ff7e3326858742c0944b5caa07b70
Author: kevin <email address hidden>
Date: Mon Jul 6 10:57:24 2015 -0500

    Fixes RabbitMQ guest user creation

    This change removes the "guest" user. This patch fixes an issues
    where the guest user was not being cared by default which could
    leave a deployment vulnerable to should someone gain access to the
    internal network that the messaging system runs within. To fix this
    issue the guest user is simply being removed as its not needed by the
    rest of the stack.

    Closes-Bug: #1466982
    Change-Id: I81f9295daf7923b828ad884b6707bb3f74b0684a
    (cherry picked from commit 980938ab011ae773cd0e732f47b2d52d0afdcd9f)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Nolan Brubaker (<email address hidden>) on branch: juno
Review: https://review.openstack.org/198888
Reason: Abandoned, this is a duplicate of https://review.openstack.org/#/c/198792

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/198792
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=026466a1849c648bfea5ef777da9a83ceb7aca10
Submitter: Jenkins
Branch: juno

commit 026466a1849c648bfea5ef777da9a83ceb7aca10
Author: kevin <email address hidden>
Date: Mon Jul 6 11:10:50 2015 -0500

    Fixes RabbitMQ guest user creation

    This change removes the "guest" user. This patch fixes an issues
    where the guest user was not being cared by default which could
    leave a deployment vulnerable to should someone gain access to the
    internal network that the messaging system runs within. To fix this
    issue the guest user is simply being removed as its not needed by the
    rest of the stack. The rpc-maas setup was updated to now support the
    creation of the new monitoring user, which is being created as an
    administrator

    Change-Id: Ic91199abbf85a681ccb9f7eac60de1f0b582b748
    Closes-Bug: #1466982

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/198794
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=ed5fd2a205b388cb7d5f544ff75fde850ef27b4d
Submitter: Jenkins
Branch: icehouse

commit ed5fd2a205b388cb7d5f544ff75fde850ef27b4d
Author: kevin <email address hidden>
Date: Mon Jul 6 11:10:50 2015 -0500

    Fixes RabbitMQ guest user creation

    This change removes the "guest" user. This patch fixes an issues
    where the guest user was not being cared by default which could
    leave a deployment vulnerable to should someone gain access to the
    internal network that the messaging system runs within. To fix this
    issue the guest user is simply being removed as its not needed by the
    rest of the stack. The rpc-maas setup was updated to now support the
    creation of the new monitoring user, which is being created as an
    administrator

    Change-Id: Ic91199abbf85a681ccb9f7eac60de1f0b582b748
    Closes-Bug: #1466982
    (cherry picked from commit 405a7f8b52c2e04fa13611bb7ae75e895fd237e5)

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.11 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 11.2.12 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.14 release.