Multiqueue is not enabled in ovs in dpdk deployment

[Vladimir Khlyunev]

When multiqueue is enabled in image metadata dpdk fails to create network:
/var/log/openvswitch/ovs-vswitchd.log
2016-11-22T14:26:37.058Z|00082|dpdk(vhost_thread2)|ERR|vHost Device '/var/run/openvswitch/vhu6770cf96-ca' 0 can't be added - too many queues 2 > 1

Steps to repro:
1 - Deploy cluster with enabled dpdk (1 controller 1 compute is enough)
2 - Update TestVM image metadata with "hw_vif_multiqueue_enabled=true"
3 - Boot instance

Result - instance is not receiving IP via dhcp, error mentioned above in OVS log

Diag snapshot is not available but I can provide env with this error

Workaround: execute following command on each compute:
"ovs-vsctl set open_vswitch . other_config:n-dpdk-rxqs=X"
where X is queues count

[Dmitry Teselkin (teselkin-d)]

http://wiki.qemu.org/Documentation/vhost-user-ovs-dpdk#Enabling_multi-queue

[Oleksiy Molchanov (omolchanov)]

I think this should be added to documentation.

Also seems automating this is not a bug and should be covered as blueprint.

[Dmitry Klenov (dklenov)]

This is configuration issue - user doesn't have to deal with additional configuration parameters. I agree that generally it is a small feature, although lack of it makes entire multiqueue story unusable.
I would suggest to fix it at least partially - let's set X to the number of cores in the system and do not implement its configuration by users.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/402322

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/403635

[OpenStack Infra (hudson-openstack)]

Change abandoned by Anastasia Balobashina (<email address hidden>) on branch: master
Review: https://review.openstack.org/402322
Reason: This problem is not relevant for ovs 2.6.1

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Polenchuk (<email address hidden>) on branch: master
Review: https://review.openstack.org/403635
Reason: not applicable for ovs2.6+

[Vladimir Khlyunev (vkhlyunev)]

fixed by updating ovs to 2.6

[TatyanaGladysheva (tgladysheva)]

Verified on 9.2 snapshot #634.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/413585

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/414991

[Nastya Urlapova (aurlapova)]

Due to open reviews move to "in progress"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/415161

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/415458

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/415666

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mikhail Zhnichkov (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/413585

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/415765

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415666
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=8ed8caf30a2c9ef63bfbb5f653271cd77eb5f823
Submitter: Jenkins
Branch: master

commit 8ed8caf30a2c9ef63bfbb5f653271cd77eb5f823
Author: Mikhail <email address hidden>
Date: Mon Dec 26 18:05:19 2016 +0400

    Get number of maximum queues for NIC

    It's needed for configure multiq for dpdk

    Change-Id: Iffa1a6c872b46230de6e9ab3b8f34cb727661571
    Partial-Bug: #1643920

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/416970

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/416970
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=f7169a931478fb70994951904dca121d86ddcd1d
Submitter: Jenkins
Branch: stable/newton

commit f7169a931478fb70994951904dca121d86ddcd1d
Author: Mikhail <email address hidden>
Date: Mon Dec 26 18:05:19 2016 +0400

    Get number of maximum queues for NIC

    It's needed for configure multiq for dpdk

    Change-Id: Iffa1a6c872b46230de6e9ab3b8f34cb727661571
    Partial-Bug: #1643920
    (cherry picked from commit 8ed8caf30a2c9ef63bfbb5f653271cd77eb5f823)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/414991
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=012310b2b1a13b3822661edd463a35f3b2bb1e2d
Submitter: Jenkins
Branch: stable/mitaka

commit 012310b2b1a13b3822661edd463a35f3b2bb1e2d
Author: Mikhail <email address hidden>
Date: Mon Dec 26 18:05:19 2016 +0400

    Get number of maximum queues for NIC

    It's needed for configure multiq for dpdk

    Change-Id: Iffa1a6c872b46230de6e9ab3b8f34cb727661571
    Partial-Bug: #1643920
    (cherry picked from commit 8ed8caf30a2c9ef63bfbb5f653271cd77eb5f823)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415161
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=d56ab1b07d5c9ce33e2ce42e87349b0d738e1d5a
Submitter: Jenkins
Branch: master

commit d56ab1b07d5c9ce33e2ce42e87349b0d738e1d5a
Author: Anastasiya <email address hidden>
Date: Tue Dec 27 13:07:12 2016 +0400

    Serialize max queues count for dpdk interfases

    We need to configure queues count on dpdk interfaces.
    The count should be limited by max supported queues
    value on DPDK NIC and number of CPUs for DPDK usage.

    Change-Id: I9b3160b7e0659d11445ca3927aa38e523a93b3a9
    Partial-Bug: #1643920
    Depends-On: Iffa1a6c872b46230de6e9ab3b8f34cb727661571

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/417355

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/417355
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=e08a69d2c3c074ea0525efab69603a4a4e5688ee
Submitter: Jenkins
Branch: stable/newton

commit e08a69d2c3c074ea0525efab69603a4a4e5688ee
Author: Anastasiya <email address hidden>
Date: Tue Dec 27 13:07:12 2016 +0400

    Serialize max queues count for dpdk interfases

    We need to configure queues count on dpdk interfaces.
    The count should be limited by max supported queues
    value on DPDK NIC and number of CPUs for DPDK usage.

    Change-Id: I9b3160b7e0659d11445ca3927aa38e523a93b3a9
    Partial-Bug: #1643920
    Depends-On: Iffa1a6c872b46230de6e9ab3b8f34cb727661571
    (cherry picked from commit d56ab1b07d5c9ce33e2ce42e87349b0d738e1d5a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415458
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=d8675cacc7ce4d54c441b717c91cc5bb16c0a3f8
Submitter: Jenkins
Branch: stable/mitaka

commit d8675cacc7ce4d54c441b717c91cc5bb16c0a3f8
Author: Anastasiya <email address hidden>
Date: Tue Dec 27 13:07:12 2016 +0400

    Serialize max queues count for dpdk interfases

    We need to configure queues count on dpdk interfaces.
    The count should be limited by max supported queues
    value on DPDK NIC and number of CPUs for DPDK usage.

    Change-Id: I9b3160b7e0659d11445ca3927aa38e523a93b3a9
    Partial-Bug: #1643920
    Depends-On: Iffa1a6c872b46230de6e9ab3b8f34cb727661571
    (cherry picked from commit 07a4615641e9f8da052af7ddcc149ffac1c42afe)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/417933

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/417934

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415765
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b28edfb2a7697ad5f5990b39d2e4c90dee558b05
Submitter: Jenkins
Branch: master

commit b28edfb2a7697ad5f5990b39d2e4c90dee558b05
Author: Mikhail <email address hidden>
Date: Fri Dec 30 02:43:03 2016 +0400

    Configure number of multiq for dpdk

    Change-Id: I34781a752fba5cc35cbe4190619a8869d07c4993
    Closes-Bug: 1643920
    Depends-On: I9b3160b7e0659d11445ca3927aa38e523a93b3a9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/417933
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1a32cc7d1d3998bfe111c6e8a57b8ce2b16390eb
Submitter: Jenkins
Branch: stable/newton

commit 1a32cc7d1d3998bfe111c6e8a57b8ce2b16390eb
Author: Mikhail <email address hidden>
Date: Fri Dec 30 02:43:03 2016 +0400

    Configure number of multiq for dpdk

    Change-Id: I34781a752fba5cc35cbe4190619a8869d07c4993
    Closes-Bug: 1643920
    Depends-On: I9b3160b7e0659d11445ca3927aa38e523a93b3a9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/417934
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e2595ca27da39fb64620b552f7e706263db8bd16
Submitter: Jenkins
Branch: stable/mitaka

commit e2595ca27da39fb64620b552f7e706263db8bd16
Author: Mikhail <email address hidden>
Date: Fri Dec 30 02:43:03 2016 +0400

    Configure number of multiq for dpdk

    Change-Id: I34781a752fba5cc35cbe4190619a8869d07c4993
    Closes-Bug: 1643920
    Depends-On: I9b3160b7e0659d11445ca3927aa38e523a93b3a9

[TatyanaGladysheva (tgladysheva)]

Verified on 9.2 snapshot #759.

Actual results:
http://paste.openstack.org/show/595247/

[Dmitry Klenov (dklenov)]

Reopening as multiqueue is not always configured correctly in runtime.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/423325

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/423390

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/423391

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/423627

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/423628

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/423325
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=643e172d484f8bae6a2c5469c937c7372010d583
Submitter: Jenkins
Branch: master

commit 643e172d484f8bae6a2c5469c937c7372010d583
Author: Sergey Vasilenko <email address hidden>
Date: Fri Jan 20 17:17:07 2017 +0300

    Add interface_propertiesi hash merge for

    interfaces members of bond

    CI disabled, because next patch in the chain should be passed.
    Whole chain should be merged at one moment.

    Change-Id: Ie7804dac600b2ea4747d0117f4951f811ef834a1
    Partial-bug: #1643920
    Fuel-CI: disabled

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421765
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=fc4269db1bf1adcab37a8ddf082f2af7fb799de3
Submitter: Jenkins
Branch: master

commit fc4269db1bf1adcab37a8ddf082f2af7fb799de3
Author: Mikhail <email address hidden>
Date: Wed Jan 18 12:37:02 2017 +0400

    Fix multiq configuration for dpdk in runtime

    Change-Id: Ib4048eb037b34d0d8f0c651783b540922a7cfd98
    Closes-Bug: #1643920

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/423391
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4d7526901272a509e8dc30dd3948ce7fe2d57e7d
Submitter: Jenkins
Branch: stable/newton

commit 4d7526901272a509e8dc30dd3948ce7fe2d57e7d
Author: Sergey Vasilenko <email address hidden>
Date: Fri Jan 20 17:17:07 2017 +0300

    Add interface_propertiesi hash merge for

    interfaces members of bond

    CI disabled, because next patch in the chain should be passed.
    Whole chain should be merged at one moment.

    Change-Id: Ie7804dac600b2ea4747d0117f4951f811ef834a1
    Partial-bug: #1643920
    Fuel-CI: disabled

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/423628
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=3529c56f8e967dff0443f28b2ea24c54beff12af
Submitter: Jenkins
Branch: stable/newton

commit 3529c56f8e967dff0443f28b2ea24c54beff12af
Author: Mikhail <email address hidden>
Date: Wed Jan 18 12:37:02 2017 +0400

    Fix multiq configuration for dpdk in runtime

    Change-Id: Ib4048eb037b34d0d8f0c651783b540922a7cfd98
    Closes-Bug: #1643920

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/423390
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=9fb1c276a274ae21d5f42b547885e0ac3c5e5aa1
Submitter: Jenkins
Branch: stable/mitaka

commit 9fb1c276a274ae21d5f42b547885e0ac3c5e5aa1
Author: Sergey Vasilenko <email address hidden>
Date: Fri Jan 20 17:17:07 2017 +0300

    Add interface_propertiesi hash merge for

    interfaces members of bond

    CI disabled, because next patch in the chain should be passed.
    Whole chain should be merged at one moment.

    Change-Id: Ie7804dac600b2ea4747d0117f4951f811ef834a1
    Partial-bug: #1643920
    Fuel-CI: disabled

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/423627
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a78b06ede0a3ad68d2839b1255ccc9664941cc0c
Submitter: Jenkins
Branch: stable/mitaka

commit a78b06ede0a3ad68d2839b1255ccc9664941cc0c
Author: Mikhail <email address hidden>
Date: Wed Jan 18 12:37:02 2017 +0400

    Fix multiq configuration for dpdk in runtime

    Change-Id: Ib4048eb037b34d0d8f0c651783b540922a7cfd98
    Closes-Bug: #1643920

[TatyanaGladysheva (tgladysheva)]

Verified on 9.2 snapshot #801.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/438928

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/439483

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/439484

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/439485

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/438928
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=d5fd08f78118a36a2cde8df4f5d216ca7b042bcf
Submitter: Jenkins
Branch: master

commit d5fd08f78118a36a2cde8df4f5d216ca7b042bcf
Author: Sergey Vasilenko <email address hidden>
Date: Tue Feb 28 14:34:25 2017 +0300

    Support bonds of subinterfaces while merge interface_properties

    Change-Id: I27ede232a37c4e8c2d669f5dd8936a60baf04b40
    Related-bug: #1643920
    Closes-bug: #1666229

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/439483
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a5eca380199738c5df0e497099145de0e5f5a824
Submitter: Jenkins
Branch: stable/mitaka

commit a5eca380199738c5df0e497099145de0e5f5a824
Author: Sergey Vasilenko <email address hidden>
Date: Tue Feb 28 14:34:25 2017 +0300

    Support bonds of subinterfaces while merge interface_properties

    Change-Id: I27ede232a37c4e8c2d669f5dd8936a60baf04b40
    Related-bug: #1643920
    Closes-bug: #1666229

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/439484
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b43956ce2cc129a5e9aa9c236e683a7b4a706b24
Submitter: Jenkins
Branch: stable/newton

commit b43956ce2cc129a5e9aa9c236e683a7b4a706b24
Author: Sergey Vasilenko <email address hidden>
Date: Tue Feb 28 14:34:25 2017 +0300

    Support bonds of subinterfaces while merge interface_properties

    Change-Id: I27ede232a37c4e8c2d669f5dd8936a60baf04b40
    Related-bug: #1643920
    Closes-bug: #1666229

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/439485
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ade66c382645b8ee10bab3950fde914d4898ba8d
Submitter: Jenkins
Branch: stable/ocata

commit ade66c382645b8ee10bab3950fde914d4898ba8d
Author: Sergey Vasilenko <email address hidden>
Date: Tue Feb 28 14:34:25 2017 +0300

    Support bonds of subinterfaces while merge interface_properties

    Change-Id: I27ede232a37c4e8c2d669f5dd8936a60baf04b40
    Related-bug: #1643920
    Closes-bug: #1666229
    (cherry picked from commit d5fd08f78118a36a2cde8df4f5d216ca7b042bcf)