Invalid scenario argument: ''dict' object has no attribute 'services''` in rally jobs

[YAMAMOTO Takashi]

eg. http://logs.openstack.org/69/442169/6/check/gate-networking-midonet-rally-dsvm-ml2-ubuntu-xenial-nv/4ea2c4d/console.html

2017-03-07 04:11:07.296484 | 2017-03-07 04:11:07.296 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine [-] Input task is invalid!
2017-03-07 04:11:07.297677 | 2017-03-07 04:11:07.297 |
2017-03-07 04:11:07.298965 | 2017-03-07 04:11:07.298 | Subtask NetworkPlugin.create_networks[0] has wrong configuration\Subtask configuration:
2017-03-07 04:11:07.300274 | 2017-03-07 04:11:07.299 | {'runner': {"type": "rps", "times": 100, "rps": 1}, 'args': {"network_create_args": {}}, 'sla': {"max_seconds_per_iteration": 5, "failure_rate": {"max": 0}}, 'context': {"users": {"tenants": 2, "users_per_tenant": 2}, "quotas": {"neutron": {"network": -1}}}}
2017-03-07 04:11:07.301392 | 2017-03-07 04:11:07.301 |
2017-03-07 04:11:07.302766 | 2017-03-07 04:11:07.302 | Reason:
2017-03-07 04:11:07.304097 | 2017-03-07 04:11:07.303 | Invalid scenario argument: ''dict' object has no attribute 'services''
2017-03-07 04:11:07.305276 | 2017-03-07 04:11:07.305 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine Traceback (most recent call last):
2017-03-07 04:11:07.306597 | 2017-03-07 04:11:07.306 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine File "/opt/stack/new/rally/rally/task/engine.py", line 376, in validate
2017-03-07 04:11:07.307983 | 2017-03-07 04:11:07.307 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine self._validate_config_semantic(self.config)
2017-03-07 04:11:07.309421 | 2017-03-07 04:11:07.309 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine File "/opt/stack/new/rally/rally/common/logging.py", line 197, in wrapper
2017-03-07 04:11:07.311202 | 2017-03-07 04:11:07.310 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine result = f(self, *args, **kwargs)
2017-03-07 04:11:07.312461 | 2017-03-07 04:11:07.312 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine File "/opt/stack/new/rally/rally/task/engine.py", line 354, in _validate_config_semantic
2017-03-07 04:11:07.313853 | 2017-03-07 04:11:07.313 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine workloads_with_users, self.deployment)
2017-03-07 04:11:07.315143 | 2017-03-07 04:11:07.314 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine File "/opt/stack/new/rally/rally/task/engine.py", line 305, in _validate_config_semantic_helper
2017-03-07 04:11:07.316287 | 2017-03-07 04:11:07.316 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine raise exceptions.InvalidTaskConfig(**kw)
2017-03-07 04:11:07.317539 | 2017-03-07 04:11:07.317 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine InvalidTaskConfig: Input task is invalid!
2017-03-07 04:11:07.318906 | 2017-03-07 04:11:07.318 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine
2017-03-07 04:11:07.320103 | 2017-03-07 04:11:07.319 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine Subtask NetworkPlugin.create_networks[0] has wrong configuration\Subtask configuration:
2017-03-07 04:11:07.321324 | 2017-03-07 04:11:07.321 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine {'runner': {"type": "rps", "times": 100, "rps": 1}, 'args': {"network_create_args": {}}, 'sla': {"max_seconds_per_iteration": 5, "failure_rate": {"max": 0}}, 'context': {"users": {"tenants": 2, "users_per_tenant": 2}, "quotas": {"neutron": {"network": -1}}}}
2017-03-07 04:11:07.322799 | 2017-03-07 04:11:07.322 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine
2017-03-07 04:11:07.323889 | 2017-03-07 04:11:07.323 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine Reason:
2017-03-07 04:11:07.325077 | 2017-03-07 04:11:07.324 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine Invalid scenario argument: ''dict' object has no attribute 'services''
2017-03-07 04:11:07.326276 | 2017-03-07 04:11:07.326 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine
2017-03-07 04:11:07.327607 | 2017-03-07 04:11:07.327 | Task config is invalid: `Input task is invalid!
2017-03-07 04:11:07.328773 | 2017-03-07 04:11:07.328 |
2017-03-07 04:11:07.330123 | 2017-03-07 04:11:07.329 | Subtask NetworkPlugin.create_networks[0] has wrong configuration\Subtask configuration:
2017-03-07 04:11:07.331312 | 2017-03-07 04:11:07.331 | {'runner': {"type": "rps", "times": 100, "rps": 1}, 'args': {"network_create_args": {}}, 'sla': {"max_seconds_per_iteration": 5, "failure_rate": {"max": 0}}, 'context': {"users": {"tenants": 2, "users_per_tenant": 2}, "quotas": {"neutron": {"network": -1}}}}
2017-03-07 04:11:07.332419 | 2017-03-07 04:11:07.332 |
2017-03-07 04:11:07.333634 | 2017-03-07 04:11:07.333 | Reason:
2017-03-07 04:11:07.335119 | 2017-03-07 04:11:07.334 | Invalid scenario argument: ''dict' object has no attribute 'services''`

[Antonio Ojea (aojea)]

The rally job was working fine on May 1st, https://review.openstack.org/#/c/199387/, and the jobs that we run internally with rally stable branch are working.

This commit seems to affect the way that rally validates tasks
https://github.com/openstack/rally/commit/4dcb597949bebaaaad3f2eacdce2f6bb9b480cb3

anyway, we need to investigate more, but most likely is because of an incompatible change in rally.

[Antonio Ojea (aojea)]

Thanks Takashi for show me the logstash tool, it's very useful.

I could check with the following query http://logstash.openstack.org/#dashboard/file/logstash.json?query=(message%3A%20%5C%22%5BZuul%5D%20Job%20complete%2C%20result%3A%5C%22%20OR%20message%3A%20%5C%22To%20get%20raw%20JSON%20output%20of%20task%20results%2C%20run%5C%22%20OR%20message%3A%20%5C%22Input%20task%20is%20invalid%5C%22)%20AND%20build_name%3A%20gate-networking-midonet-rally-dsvm-ml2-ubuntu-xenial-nv

that the last rally job that worked was with this commit

9e3ccfa [CI] Update hacking and fix pep8 errors

and the first job failing has this rally commit

732bbc1 Merge "[Core] Make task engine more abstract”

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/443058

[Antonio Ojea (aojea)]

Workaround https://review.openstack.org/443058 until we fix our custom plugins

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/443058
Committed: https://git.openstack.org/cgit/openstack/networking-midonet/commit/?id=a487c8439d058004e5c0cf72c18ec348d658f8fa
Submitter: Jenkins
Branch: master

commit a487c8439d058004e5c0cf72c18ec348d658f8fa
Author: Antonio Ojea <email address hidden>
Date: Wed Mar 8 11:23:31 2017 +0100

    Disable custom plugin scenarios in rally

    The rally jobs are failing because they fail to validate
    the subtasks that use our custom plugins. It seems that
    something has changed with rally upstream and we need to
    fix our plugins in order to met the new requirements.

    This patchs remove the scenarios that are using our custom
    plugins.

    Related-Bug: #1670577

    Change-Id: I7d8430dc988a68bc6f465f7e6035237da5fb261b
    Signed-off-by: Antonio Ojea <email address hidden>

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/467563

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/467563
Committed: https://git.openstack.org/cgit/openstack/networking-midonet/commit/?id=76e932cda6f575bcd0de723fef09d781f155798d
Submitter: Jenkins
Branch: stable/ocata

commit 76e932cda6f575bcd0de723fef09d781f155798d
Author: Antonio Ojea <email address hidden>
Date: Wed Mar 8 11:23:31 2017 +0100

    Disable custom plugin scenarios in rally

    The rally jobs are failing because they fail to validate
    the subtasks that use our custom plugins. It seems that
    something has changed with rally upstream and we need to
    fix our plugins in order to met the new requirements.

    This patchs remove the scenarios that are using our custom
    plugins.

    Related-Bug: #1670577

    Change-Id: I7d8430dc988a68bc6f465f7e6035237da5fb261b
    Signed-off-by: Antonio Ojea <email address hidden>
    (cherry picked from commit a487c8439d058004e5c0cf72c18ec348d658f8fa)

[YAMAMOTO Takashi (yamamoto)]

https://review.openstack.org/#/c/492376/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/492376
Committed: https://git.openstack.org/cgit/openstack/networking-midonet/commit/?id=fb024e587dc073b3bc0cdb80b79edfad25dd4bf4
Submitter: Jenkins
Branch: master

commit fb024e587dc073b3bc0cdb80b79edfad25dd4bf4
Author: YAMAMOTO Takashi <email address hidden>
Date: Thu Aug 10 13:51:22 2017 +0900

    rally: Adapt to the new API

    See the following rally changes for details.

        Ife436d2167d0768a7e7fcc83460338fba2ebf236
        I539838186d2e809143adb5b4c25dfedecbf2764b
        Ib05cc0a04f9874904e9f42f4ca982bbe4e798c0b
        Ia6841068eb1d5a39a8f64f160160f1fcc1e05ac5

    Closes-Bug: #1670577
    Change-Id: I1be253557adb1857ddf870045345cef5e4871b1b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/492374
Committed: https://git.openstack.org/cgit/openstack/networking-midonet/commit/?id=0eba6c8bb436970307651f5b19358cd9e0627cbd
Submitter: Jenkins
Branch: master

commit 0eba6c8bb436970307651f5b19358cd9e0627cbd
Author: YAMAMOTO Takashi <email address hidden>
Date: Thu Aug 10 04:31:22 2017 +0000

    Revert "Disable custom plugin scenarios in rally"

    This reverts commit a487c8439d058004e5c0cf72c18ec348d658f8fa.

    Also,

    - Remove unused context from midonet scenarios.

    - Enlarge the subnet for for NetworkPlugin.create_routers to avoid
      "Unexpected error: increment exceeds address boundary!"

    - Comment out sla sections for some failing scenarios because their
      failure ratios seem too high for a voting job.

    Related-Bug: #1670577
    Related-Bug: #1710589
    Change-Id: I10f49177e64acb64820831f3f32eb26eca05f153

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-midonet 6.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/447054
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.