Create policy rule fails with APIC error

[venkat akkina]

Policy rule creation failing , while Inserting Fw+Lb scenario in E-W using ATF

Neutron server log:
===============

2015-11-12 00:57:48.265 29163 DEBUG apicapi.apic_client [-] Response: [{u'error': {u'attributes': {u'text': u'vz::EntryMo (Dn0) - non-IP Ethertype cannot be combined with other l4 properties Dn0=uni/tn-_noirolab_ea103dbebbb44a1ca7e7f73ce89fc2a0/flt-atf-prj1_prs_rule_4/e-os-entry, ', u'code': u'105'}}}] _send /usr/lib/python2.7/site-packages/apicapi/apic_client.py:415
2015-11-12 00:57:48.265 29163 ERROR gbpservice.neutron.services.grouppolicy.policy_driver_manager [-] Policy driver 'apic' failed in create_policy_rule_postcommit
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager Traceback (most recent call last):
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib/python2.7/site-packages/gbpservice/neutron/services/grouppolicy/policy_driver_manager.py", line 119, in _call_on_drivers
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager getattr(driver.obj, method_name)(context)
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib/python2.7/site-packages/gbpservice/neutron/services/grouppolicy/drivers/cisco/apic/apic_mapping.py", line 508, in create_policy_rule_postcommit
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager **attrs)
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager self.gen.next()
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib/python2.7/site-packages/apicapi/apic_client.py", line 867, in transaction
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager result = transaction.commit()
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib/python2.7/site-packages/apicapi/apic_client.py", line 773, in commit
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager *self.root_params)
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib/python2.7/site-packages/apicapi/apic_client.py", line 480, in post_body
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager return self._send(self.session.post, url, data=data)
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib/python2.7/site-packages/apicapi/apic_client.py", line 435, in _send
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager err_text=err_text, err_code=err_code)
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager ApicResponseNotOk: APIC responded with HTTP status 400: Bad Request, Request: '/mo/uni/tn-_noirolab_ea103dbebbb44a1ca7e7f73ce89fc2a0.json, data={"fvTenant": {"attributes": {"rn": "tn-_noirolab_ea103dbebbb44a1ca7e7f73ce89fc2a0"}, "children": [{"vzFilter": {"attributes": {"rn": "flt-atf-prj1_prs_rule_4"}, "children": [{"vzEntry": {"attributes": {"rn": "e-os-entry", "etherT": "unspecified"}, "children": []}}]}}]}}', APIC error code 105: vz::EntryMo (Dn0) - non-IP Ethertype cannot be combined with other l4 properties Dn0=uni/tn-_noirolab_ea103dbebbb44a1ca7e7f73ce89fc2a0/flt-atf-prj1_prs_rule_4/e-os-entry,
2015-11-12 00:57:48.265 29163 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager
2015-11-12 00:57:48.267 29163 ERROR gbpservice.neutron.services.grouppolicy.plugin [-] policy_driver_manager.create_policy_rule_postcommit failed, deleting policy_rule 7a95a5db-eeaa-4d63-9055-48ade8057a2c
2015-11-12 00:57:48.267 29163 TRACE gbpservice.neutron.services.grouppolicy.plugin Traceback (most recent call last):
2015-11-12 00:57:48.267 29163 TRACE gbpservice.neutron.services.grouppolicy.plugin File "/usr/lib/python2.7/site-packages/gbpservice/neutron/services/grouppolicy/plugin.py", line 1064, in create_policy_rule
2015-11-12 00:57:48.267 29163 TRACE gbpservice.neutron.services.grouppolicy.plugin policy_context)
2015-11-12 00:57:48.267 29163 TRACE gbpservice.neutron.services.grouppolicy.plugin File "/usr/lib/python2.7/site-packages/gbpservice/neutron/services/grouppolicy/policy_driver_manager.py", line 280, in create_policy_rule_postcommit
2015-11-12 00:57:48.267 29163 TRACE gbpservice.neutron.services.grouppolicy.plugin self._call_on_drivers("create_policy_rule_postcommit", context)
2015-11-12 00:57:48.267 29163 TRACE gbpservice.neutron.services.grouppolicy.plugin File "/usr/lib/python2.7/site-packages/gbpservice/neutron/services/grouppolicy/policy_driver_manager.py", line 134, in _call_on_drivers
2015-11-12 00:57:48.267 29163 TRACE gbpservice.neutron.services.grouppolicy.plugin method=method_name
2015-11-12 00:57:48.267 29163 TRACE gbpservice.neutron.services.grouppolicy.plugin GroupPolicyDriverError: create_policy_rule_postcommit failed.

2015-11-12 00:57:48.267 29163 TRACE gbpservice.neutron.services.grouppolicy.plugin
2015-11-12 00:57:48.268 29163 DEBUG gbpservice.neutron.services.grouppolicy.plugin [-] gbpservice.neutron.services.grouppolicy.plugin.GroupPolicyPlugin method delete_policy_rule called with arguments (<neutron.context.Context object at 0x5da4750>, '7a95a5db-eeaa-4d63-9055-48ade8057a2c') {} wrapper /usr/lib/python2.7/site-packages/neutron/common/log.py:33

[Ivar Lazzaro (mmaleckk)]

What kind of rule was about to be created? How does the PR look like? Is it reproducible only on service chain scenarios?

[venkat akkina (venkat-akkina)]

Following are the steps which we did when we saw the issue.

    Create a tenant(member).
    Create 4 classifiers as following
        icmp : icmp classifier
        udp: udp bi-directional
        tcp: tcp bi-directional
        redirect_class: classifier with no protocol and port(Created this from CLI)
        tcp80: tcp port 90, bi-directional
    Create 4 allow actions(allow1, allow2, allow3, allow4) and 1 redirect action with FW service chain spec(to insert FW in E-W)
    Create 5 rules as following
        pr1: icmp+allow1
        pr2: tcp+allow2
        pr3: udp+allow3
        pr4: redirect_class+allow4
        pr5: tcp80+ redirect action
    Created PRS 'prs1' using rules pr1, pr2, pr3, pr4 and pr5.
    Create consumer and provider PTG to use the above PRS 'prs1'. FW service was inserted between the consumer and provider groups.
    Create another 4 classifiers as following:
        icmp1: icmp classifier
        tcp1: tcp classifier
        udp1: udp classifier
        redirect_class1: classifier with no protocol and port (created from CLI)
    Create 4 allow actions(allow5, allow6, allow7, allow8)
    Create 3 policy rules as following:
        pr6: tcp1+allow5
        pr7: udp1+allow7
        pr8: icmp1+allow6
    Now tried creating another allow rule using redirect_class1 classifier and allow8 action. Observed the policy rule create issue.

Note: After sometime, when tried adding the same rule again, the rule creation was successful.

Neutron server.log and the host-report output of OS controller and compute nodes are copied to "/root/pr_create_fail_log/" location on OS contoller node(10.30.120.97)

Snapshot of neutron server log when the error is seen:
2015-11-17 05:12:05.219 9976 ERROR gbpservice.neutron.services.grouppolicy.policy_driver_manager [-] Policy driver 'apic' failed in create_policy_rule_postcommit
2015-11-17 05:12:05.219 9976 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager Traceback (most recent call last):
2015-11-17 05:12:05.219 9976 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib/python2.7/site-packages/gbpservice/neutron/services/grouppolicy/policy_driver_manager.py", line 119, in _call_on_drivers
2015-11-17 05:12:05.219 9976 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager getattr(driver.obj, method_name)(context)
2015-11-17 05:12:05.219 9976 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib/python2.7/site-packages/gbpservice/neutron/services/grouppolicy/drivers/cisco/apic/apic_mapping.py", line 589, in create_policy_rule_postcommit
2015-11-17 05:12:05.219 9976 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager **attrs)
2015-11-17 05:12:05.219 9976 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2015-11-17 05:12:05.219 9976 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager self.gen.next()
2015-11-17 05:12:05.219 9976 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib/python2.7/site-packages/apicapi/apic_client.py", line 867, in transacti...