Scons fails when building in parallel

[Filip Pytloun]

When building with eg `scons -j4`, build fails on dependency cycle:

scons: *** Found dependency cycle(s):
  Internal Error: no cycle found for node usr/lib/libthriftasio.so.0.0.0 (<SCons.Node.FS.File object at 0x2b76e1865250>) in state pending

Console log attached.

[Valentine Sinitsyn (valentine-sinitsyn)]

Anyone working on this?

[Krzysztof Klimonda (kklimonda)]

I've put some initial work into fixing it[1][2][3][4], and I can build current master with -j25. It's an ongoing effort, especially with Scons 2.5 being more robust in detecting circular dependencies, but I should be able to push those upstream soonish[tm].

[1] https://github.com/kklimonda/contrail-build/commit/c937af2af23697d13f6383b326bd98a25b1d76ca
[2] https://github.com/kklimonda/contrail-controller/commit/a6bffb30e210193cfb8a78a4567dea820f361feb
[3] https://github.com/kklimonda/contrail-controller/commit/481ae952f9dbfcfede95926ac91cf2494922ff2a
[4] https://github.com/kklimonda/contrail-controller/commit/bbf22cc5a4aaece17db3d8e65b669be98d1c65fb

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34917
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34916
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34915
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34918
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34919
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34920
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34915
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34916
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34924
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34986
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35098
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34986
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34916
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35773
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/34924
Committed: http://github.com/Juniper/contrail-controller/commit/1cf533dd590db7f74946849a44464cd170a53df2
Submitter: Zuul (<email address hidden>)
Branch: master

commit 1cf533dd590db7f74946849a44464cd170a53df2
Author: Krzysztof Klimonda <email address hidden>
Date: Fri Aug 25 18:41:57 2017 +0200

Add thrift/config.h to thrift build products

Some of our code depends indirectly on #build/include/thrift/config.h
and will not build (failing, or silently being ignored) when we don't
have that file on our list of products. Additionally, remove the
config.status target, as we don't care about that files.

Partial-Bug: #1476261
Change-Id: I8e741582ed9c8a58c23372a9257bb854660f16e6

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/34920
Committed: http://github.com/Juniper/contrail-controller/commit/cb69beb977e03847dd1e668eaa5d7a6f7bb70c5f
Submitter: Zuul (<email address hidden>)
Branch: master

commit cb69beb977e03847dd1e668eaa5d7a6f7bb70c5f
Author: Krzysztof Klimonda <email address hidden>
Date: Thu Aug 10 17:24:55 2017 +0200

Fix the mismatch between what's built and what we think is built.

GenerateBuildInfoPyCode() creates control_node/buildinfo.py, but
we add buildinfo.py to the SCons target list and other targets
can't correctly generate dependencies.

Partial-Bug: #1476261
Change-Id: Ibcf39a6c83de28fcfa1b77247e074164085bdffb

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/34919
Committed: http://github.com/Juniper/contrail-controller/commit/6192003ffc902a8a47146a5f529c9a3b27473751
Submitter: Zuul (<email address hidden>)
Branch: master

commit 6192003ffc902a8a47146a5f529c9a3b27473751
Author: Krzysztof Klimonda <email address hidden>
Date: Fri Aug 25 11:52:41 2017 +0200

Make contrail-vrouter-agent dependencies explicit

contrail-vrouter-agent should depend on required libraries and object files
explicitly, and not through buildinfo.cc - buildinfo.{cc,h} should not have
any dependencies of their own, as they don't require anything. That
dependency was also breaking multithreaded builds, probably by confusing
Scons dependency tracking.

Partial-Bug: #1476261
Change-Id: If5f220ca228e8bbcefe03b66c3ae1c5efd105381

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/34918
Committed: http://github.com/Juniper/contrail-controller/commit/8e4e461d5041ff2ee17d46aaabc25e7c5badeaf9
Submitter: Zuul (<email address hidden>)
Branch: master

commit 8e4e461d5041ff2ee17d46aaabc25e7c5badeaf9
Author: Krzysztof Klimonda <email address hidden>
Date: Thu Aug 10 17:20:13 2017 +0200

api-server generateDS.py depends on vnc_api python module.

generateDS imports ifmap_frontend.py, which imports
vnc_api.resource_common.py, which can fail if we are running
parallel build - create an explicit dependency on the generated
python sources from vnc_api, so that we build them in the correct
order.

Partial-Bug: #1476261
Change-Id: If1ee98877d2f751db11030cf2af49b8de90687fd

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35542
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35098
Committed: http://github.com/Juniper/contrail-controller/commit/5add62e307b9ed99879ac3d80e7b4414a1d9aaf9
Submitter: Zuul (<email address hidden>)
Branch: master

commit 5add62e307b9ed99879ac3d80e7b4414a1d9aaf9
Author: Krzysztof Klimonda <email address hidden>
Date: Mon Aug 28 10:38:37 2017 +0000

Explicitly define Bind9 build products

FIXME: This patch requires some more work, now some targets
(e.g. metadata_proxy.o) will rebuild on the consecutive scons runs.

All headers and libraries built during bind9 build process should be
declared so that other targets can pick them up and build the dependency
tree.

Additionally, remove #build/include/isc directory from the list of
products, which Scons does not support - instead make Scons work for us
and build the dependency tree when possible.

Partial-Bug: #1476261
Change-Id: I83082d1a998ec2d5d88f508c23383a25bae73d30

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/34916
Committed: http://github.com/Juniper/contrail-build/commit/ced5b8fed4dce10173cc45fb490d6c782eadf540
Submitter: Zuul (<email address hidden>)
Branch: master

commit ced5b8fed4dce10173cc45fb490d6c782eadf540
Author: Krzysztof Klimonda <email address hidden>
Date: Fri Aug 25 13:09:59 2017 +0200

Rebuild buildinfo.{cc,h} on every Scons run

We are creating circular dependencies (direct and indirect) by passing
source to the env.Command that's used to generate build information.

Given that buildinfo.{cc,h} don't depend on source files, we can just
wrap Command executed from GenerateBuildInfoCode in AlwaysBuild. Then
we can modify SConscript from other repositories and stop passing any
source files.

Partial-Bug: #1476261
Change-Id: I009204e9bc6f80197f16bde248bbef254d12ab9c

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34986
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34915
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34986
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34915
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34917
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34915
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34917
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35773
Committed: http://github.com/Juniper/contrail-controller/commit/86b32077bb06054c73f421489f9ea3ce28f9e611
Submitter: Zuul (<email address hidden>)
Branch: master

commit 86b32077bb06054c73f421489f9ea3ce28f9e611
Author: Krzysztof Klimonda <email address hidden>
Date: Wed Sep 20 14:30:53 2017 +0200

Add log4cplus headers to the list of products

On systems that build log4cplus, SCons has to be made aware
of the headers created by the build, so C++ Scanner can properly
create dependencies for source files.

Change-Id: I9ba5c89b967e50523e409ce210f9acf6d8dc1688
Partial-Bug: #1476261

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/34915
Committed: http://github.com/Juniper/contrail-build/commit/44188cc63d220a66c58e04ee7665983cf51bfee6
Submitter: Zuul (<email address hidden>)
Branch: master

commit 44188cc63d220a66c58e04ee7665983cf51bfee6
Author: Krzysztof Klimonda <email address hidden>
Date: Fri Aug 25 12:53:04 2017 +0200

Extend SandeshGenCpp to generate additional source files

Some targets (e.g. contrail-query-engine) depend on
_request_skeleton.cpp source files, but even though those files are
generated, SandeshGenCpp does not register them with Scons.
Additionally, other targets provide their own implementation of the
autogenerated stub methods, and for them we have to skip adding extra
dependencies.

Expand SandeshGenCpp so that it can be passed an optional argument,
specyfying a list of extra files to add to the Scons dependency system.

Partial-Bug: #1476261
Change-Id: I5f1515ac6a536d03f41449d04802ac7b3525d555

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34917
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35542
Committed: http://github.com/Juniper/contrail-controller/commit/2677174188124711c92ffd199c57c2424be0773b
Submitter: Zuul (<email address hidden>)
Branch: master

commit 2677174188124711c92ffd199c57c2424be0773b
Author: Krzysztof Klimonda <email address hidden>
Date: Tue Sep 12 21:17:01 2017 +0200

Only copy files after they've been generated

Partial-Bug: #1476261
Change-Id: Ib742d2a7a959b6ede79ec5470ae18143fe8b423e

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/34986
Committed: http://github.com/Juniper/contrail-controller/commit/6b2f0d96a1b8b24faf2f3ac0b864834f6b271725
Submitter: Zuul (<email address hidden>)
Branch: master

commit 6b2f0d96a1b8b24faf2f3ac0b864834f6b271725
Author: Krzysztof Klimonda <email address hidden>
Date: Tue Sep 19 13:31:45 2017 +0200

Explicitly define RapidJSON products

Define a product of RapidJSON build (just headers) to be consumed by
targets that depend on them.

Remove explicit dependency on '#build/include/rapidjson/', as depending
on directories is not officially supported by Scons and the
OpServerProxy.cc dependencies should be handled by Scons anyway now that
it knows where to look for them.

Partial-Bug: #1476261
Change-Id: Ic59aa3f378bf50e5f4c2d4f9388374a0d4c116a6

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/34917
Committed: http://github.com/Juniper/contrail-controller/commit/7096b888d526d04f70453c2f2f68c10b49a9cd3e
Submitter: Zuul (<email address hidden>)
Branch: master

commit 7096b888d526d04f70453c2f2f68c10b49a9cd3e
Author: Krzysztof Klimonda <email address hidden>
Date: Thu Aug 3 16:55:23 2017 +0200

Update SConscript to register sandesh-generated skeletons with SCons.

analytics_request_skeleton.cc is a build dependency of
contrail-query-engine, and so has to be registered with SCons. Call a
variant of SandeshGenCpp that registers those source files.

However, contrail-collector is already linking against another source
file with the same symbols and build it with analytics_request_skeleton.cc
fails, so we filter them from vizd and buildinfo.cc targets.

Partial-Bug: #1476261
Depends-On: I5f1515ac6a536d03f41449d04802ac7b3525d555
Change-Id: I97bac46e88b3a60bc5c396913df5f287d894c031

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35647
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35647
Committed: http://github.com/Juniper/contrail-controller/commit/d55cc3dcee8161a7c15674ea1f9bed314540a15b
Submitter: Zuul (<email address hidden>)
Branch: master

commit d55cc3dcee8161a7c15674ea1f9bed314540a15b
Author: Krzysztof Klimonda <email address hidden>
Date: Fri Sep 15 21:50:22 2017 +0200

Add contrail-tor-agent to the list of default targets

Partial-Bug: #1476261

Change-Id: Ie54c25bd932d41ef1b4cd8f681b249e9ecc3c8a0

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/37892
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/37893
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/37987
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/37988
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/37989
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/37990
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/37991
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/37992
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/37993
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/37995
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/37996
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/37997
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/37998
Submitter: Krzysztof Klimonda (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/37892
Committed: http://github.com/Juniper/contrail-build/commit/a66d27c439df28de6096a1233d6bef4f9b020718
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit a66d27c439df28de6096a1233d6bef4f9b020718
Author: Krzysztof Klimonda <email address hidden>
Date: Fri Aug 25 12:53:04 2017 +0200

Extend SandeshGenCpp to generate additional source files

Some targets (e.g. contrail-query-engine) depend on
_request_skeleton.cpp source files, but even though those files are
generated, SandeshGenCpp does not register them with Scons.
Additionally, other targets provide their own implementation of the
autogenerated stub methods, and for them we have to skip adding extra
dependencies.

Expand SandeshGenCpp so that it can be passed an optional argument,
specyfying a list of extra files to add to the Scons dependency system.

Partial-Bug: #1476261
Change-Id: I5f1515ac6a536d03f41449d04802ac7b3525d555
(cherry picked from commit 44188cc63d220a66c58e04ee7665983cf51bfee6)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/37997
Committed: http://github.com/Juniper/contrail-controller/commit/6cd13f8a1ed9766e073b85e9e4a7f2e388b72e7d
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 6cd13f8a1ed9766e073b85e9e4a7f2e388b72e7d
Author: Krzysztof Klimonda <email address hidden>
Date: Thu Aug 10 17:24:55 2017 +0200

Fix the mismatch between what's built and what we think is built.

GenerateBuildInfoPyCode() creates control_node/buildinfo.py, but
we add buildinfo.py to the SCons target list and other targets
can't correctly generate dependencies.

Partial-Bug: #1476261
Change-Id: Ibcf39a6c83de28fcfa1b77247e074164085bdffb
(cherry picked from commit cb69beb977e03847dd1e668eaa5d7a6f7bb70c5f)