keystone-manage mapping_engine is broken: TypeError

[Daniel Speichert]

Running `keystone-manage mapping_engine` (with parameters) is broken:

2017-03-30 16:09:11.982 13513 CRITICAL keystone [-] TypeError: __init__() takes exactly 3 arguments (2 given)
2017-03-30 16:09:11.982 13513 ERROR keystone Traceback (most recent call last):
2017-03-30 16:09:11.982 13513 ERROR keystone File "/usr/bin/keystone-manage", line 10, in <module>
2017-03-30 16:09:11.982 13513 ERROR keystone sys.exit(main())
2017-03-30 16:09:11.982 13513 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/cmd/manage.py", line 44, in main
2017-03-30 16:09:11.982 13513 ERROR keystone cli.main(argv=sys.argv, config_files=config_files)
2017-03-30 16:09:11.982 13513 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/cmd/cli.py", line 1270, in main
2017-03-30 16:09:11.982 13513 ERROR keystone CONF.command.cmd_class.main()
2017-03-30 16:09:11.982 13513 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/cmd/cli.py", line 1143, in main
2017-03-30 16:09:11.982 13513 ERROR keystone rp = mapping_engine.RuleProcessor(rules['rules'])
2017-03-30 16:09:11.982 13513 ERROR keystone TypeError: __init__() takes exactly 3 arguments (2 given)

Affects mitaka, newton and ocata stable (centos-release).

[Lance Bragstad (lbragstad)]

Hi Dan,

Can you provide an example mapping and arguments that you used to recreate the issue?

[Daniel Speichert (dasp)]

This is my mapping.json file:
[
    {
        "local": [
            {
                "user": {
                    "name": "{0}",
                    "domain": {
                        "id": "default"
                    },
                    "type": "local"
                },
            }
        ],
        "remote": [
            {
                "type": "uid"
            }
        ]
    }
]

This is the test.txt input file:
uid: abc

Then simply running:
keystone-manage mapping_engine --rules mapping.json --input test.txt

The error I posted earlier only shows up in /var/log/keystone/keystone.log

[Anthony Washington (anthony-washington)]

Hey Daniel, after looking at your "mapping.json" file there's an error with it. If you remove the comma after the "type": "local }" block -- it should work.

[Daniel Speichert (dasp)]

Hi Anthony, good catch but that did not fix the issue. The command doesn't return any useful output (no errors either). The error in keystone.log happens when the JSON is valid.

[Lance Bragstad (lbragstad)]

Anthony, were you able to recreate this to some extent?

[Darin Sorrentino (dsorrent)]

I ran into the same issue. The problem appears to be in:

keystone/cmd/cli.py

This block of code:

        rules = MappingEngineTester.read_rules(CONF.command.rules)
        rules = MappingEngineTester.normalize_rules(rules)
        mapping_engine.validate_mapping_structure(rules)

        assertion = MappingEngineTester.read_file(CONF.command.input)
        assertion = MappingEngineTester.normalize_assertion(assertion)
        rp = mapping_engine.RuleProcessor(rules['rules'])
        print(jsonutils.dumps(rp.process(assertion), indent=2))

I was able to get it working with this change to the 2nd to last line:

        rules = MappingEngineTester.read_rules(CONF.command.rules)
        rules = MappingEngineTester.normalize_rules(rules)
        mapping_engine.validate_mapping_structure(rules)

        assertion = MappingEngineTester.read_file(CONF.command.input)
        assertion = MappingEngineTester.normalize_assertion(assertion)
        rp = mapping_engine.RuleProcessor(assertion, rules['rules']) # Passing the assertion too
        print(jsonutils.dumps(rp.process(assertion), indent=2))

I don't know if this is the 'correct' fix, but this is what got it working for me.

[Lance Bragstad (lbragstad)]

Hi Darin,

What version of the code are you running? I'm looking through master and stable/ocata but I can't seem to locate the code your referencing in comment #6 [0].

[0] https://github.com/openstack/keystone/blob/d01c7f1b962ef75cfacc63d8a5c2f4d5e05017cf/keystone/cmd/cli.py#L1119-L1237

[Daniel Speichert (dasp)]

Lance,

I checked again on stable/ocata running right now and it does not have this problem anymore. I'm quite certain that stable/newton is still experiencing this issue.

[Lance Bragstad (lbragstad)]

I can confirm that this is still a problem on stable/newton. It has likely *always* been a problem on stable newton due to insufficient testing. The RuleProcessor object expects a mapping_id to be passed with the rules [0]. We are passing both of those values in all version except stable/newton [1].

Looks like a fix was merged in during the Ocata development cycle [2]. There is a backport up for review to stable/newton [3] that will fix the issue.

I'll propose a new stable/newton release as soon as it merged.

[0] https://github.com/openstack/keystone/blob/stable/newton/keystone/federation/utils.py#L415
[1] https://github.com/openstack/keystone/blob/stable/newton/keystone/cmd/cli.py#L1143
[2] https://github.com/openstack/keystone/commit/f2d0f8c9ab38172a6e37b02339eac59da911435c
[3] https://review.openstack.org/#/c/466873/1

[Edward Hope-Morley (hopem)]

@fnordahl having just realised that the patch you are backporting is from another bug I'm afraid I'll have to ask you to stick to our usual process for submitting SRUs i.e. always use the LP bug that the patch is actually associated with. Please also ensure that all of the following are extant in the bug when submit; (a) a link to the upstream stable branch patch(es) if applicable (b) nominations for Ubuntu releases being targeted (you can also add the cloud-archive project if you want but we can take of that) and (c) the SRU template as you have provided here. If you need to reference this bug for clarity that's fine. Thanks.

[Frode Nordahl (fnordahl)]

Sorry about the noise in this bug, further process is tracked in bug 1655182

[Morgan Fainberg (mdrnstm)]

newton is EOL.