tempest-dsvm jobs don't pull required projects from source

[Boden R]

The failures of the legacy-tempest-dsvm-networking-bagpipe and legacy-tempest-dsvm-networking-bgpvpn-bagpipe jobs in [1] indicate that networking-sfc (a dependency of bagpipe) is importing the neutron.db.api module that doesn't exist [2].

However this use of neutron.db.api was removed in sfc with [3].

That said it doesn't appear these legacy jobs are pulling the dependencies from source for devstack. I chatted with the infra team [4] and they recommend that these legacy jobs [5][6] for bagpipe be migrated to zuul v3 format.

There's a migration guide for that [7] and as much as I might be willing to help, this is a bit beyond my current scope for bagpipe.

[1] https://review.openstack.org/#/c/636962/
[2] http://logs.openstack.org/62/636962/2/check/legacy-tempest-dsvm-networking-bagpipe/4bef871/logs/devstacklog.txt.gz#_2019-02-19_14_31_51_802
[3] https://github.com/openstack/networking-sfc/commit/eb72322943c111df7eaaa472857383ac2b2f2012#diff-dedc1f04413287f9620970b90ea536e3
[4] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-02-19.log.html#t2019-02-19T15:14:04
[5] https://github.com/openstack-infra/openstack-zuul-jobs/tree/master/playbooks/legacy/tempest-dsvm-networking-bagpipe
[6] https://github.com/openstack-infra/openstack-zuul-jobs/tree/master/playbooks/legacy/tempest-dsvm-networking-bgpvpn-bagpipe
[7] https://docs.openstack.org/infra/manual/zuulv3.html#

[Thomas Morin (tmmorin-orange)]

Last time I checked, devstack-based josb weren't inheriting the magic behavior of required-projects, which was inherited via tox-siblings. Are we sure that simply rewriting the job as a non-legacy job would be sufficient ?

(moving to a zuulv3 job is certainly a good idea, I'm just wondering if it will be sufficient to solve this bug)

[Boden R (boden)]

I can't say for certain, but seems to be what the infra team was saying in [4] from the description.

[Boden R (boden)]

Looking more into this, I think the playbooks need to include the required projects on their PROJECTS env var in the playbook like [1]. These projects would also be required-projects for the job definition.

Although I haven't done these playbooks before, I can try to get something started for this.

[1] https://github.com/openstack/oslo.messaging/blob/master/playbooks/oslo.messaging-telemetry-dsvm-integration-amqp1/run.yaml#L37

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/638429

[Boden R (boden)]

I sent a question to the ML [1]. It seems like some additional magic is needed if a project requires a dependency from source and also has it in requirements.txt.

[1] http://lists.openstack.org/pipermail/openstack-discuss/2019-February/003010.html

[Boden R (boden)]

Appears to be the same issue in https://review.openstack.org/#/c/638099

[Boden R (boden)]

I was able to get things working for tricircle [1], but it required to separate out the "required projects" that are pulled from source into their own requirements file so they are not picked up by devstack.

IIUC the above solution implies each project that does this would need to add those required projects back into requirements.txt with the proper versions for each openstack release.

Still hoping for a better solution on the ML (see comment #5), but so far nothing.

[1] https://review.openstack.org/#/c/638099/10

[Boden R (boden)]

I still haven't gotten an answer on this topic.
I created a QA bug [1] in hopes someone on that team can help with a better solution.

[1] https://bugs.launchpad.net/tempest/+bug/1817555

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/639363

[Akihiro Motoki (amotoki)]

In https://review.openstack.org/639363, I proposed another solution to update networking-sfc dependency to stein b1.
networking-bagpipe gate still pulls networking-sfc from PyPI, but we can now consume networking-sfc which is compatible with the latest stein neutron and neutron-lib. I hope this will fix the gate failure.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/639363
Committed: https://git.openstack.org/cgit/openstack/networking-bagpipe/commit/?id=d02b173d170edb1002f36aa44fe507752a8c47a8
Submitter: Zuul
Branch: master

commit d02b173d170edb1002f36aa44fe507752a8c47a8
Author: Akihiro Motoki <email address hidden>
Date: Wed Feb 27 00:10:01 2019 +0900

    Consume networking-sfc stein b1

    As of now, due to bug 1817555, networking-sfc code cannot be pulled
    from the master branch even if we specify it in 'required-projects'.
    As a workaround, this commit specifies networking-sfc stein b1
    in requirements.txt (and lower-constraints.txt accordingly).

    Change-Id: I6142075b70231cef9a76d51f50d075081cb2d7de
    Related-Bug: #1816644

[Boden R (boden)]

Thanks for comment #10 and the PS.
I also pushed one to tricircle https://review.openstack.org/#/c/639662/

It seems we have a short term solution, but in practice it seems the approach in comment #10 would require us to publish releases of projects more frequently. In some cases this would mean requiring new releases before we can merge neutron-lib consumption patches; slowing the velocity.

IMHO it seems we still need a solution to use required-projects from source in devstack based jobs.
Not sure what others think about this topic??

[Thomas Morin (tmmorin-orange)]

I agree we need a long term solution to that.

Perhaps the following would work (possible not far from something somebody proposed earlier, perhaps you boden):
- move deps corresponding to required-projects in a different requirements file, let's say "os-required-projects.txt"
- in tox.ini, add -ros-required-projects.txt lines : the tox-siblings logic will make sure that the git version of these projects is used for these tox jobs
- in devstack, rely on PROJECTS to have the git version (by avoiding these projects in requiremnts.txt we avoid that the git version would be overwriten on a simple "pip install ." by devstack)

An alternative, perhaps, could be: cook devstack plugins so that they comment out the lines corresponding to required-projects in requirements.txt ?

[Boden R (boden)]

Thomas, if I understand comment #13 it's basically what I did for tricircle [1]; separating those dependencies we want from git into their own requirements file?

Based on [1] this does work, but it kind of goes against the purpose of a requirements.txt file that can be used to install all the reqs for a repo/project since in this case anything installing the project needs to know to install using both the requirements.txt and os-required-projects.txt.. Just my $0.02 and based on my (limited) understanding.

Another option may be to have the run playbook remove those required projects from requirements.txt so that they are only used from PROJECTS or LIBS_FROM_GIT... However I'm not sure when the playbooks and moreover this might be considered bad form.

[1] https://review.openstack.org/#/c/638099/10

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/638429
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.