checkbox-remote is SLOW when running tests
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Next Generation Checkbox (CLI) |
Fix Released
|
High
|
Sylvain Pineau |
Bug Description
Discussed briefly while in the Taipei sprint but when runnign tests via checkbox-remote Ive noticed that the tests are insanely slow.
For example, this is running the test-firmware launcher in the server provider manually, locally on a machine:
ubuntu@jellyplum:~$ time test-firmware
=======
There are 0 incomplete sessions that might be resumed
Preparing...
<SNIP>
real 2m33.180s
user 1m48.855s
sys 0m37.771s
And this is the same test run via checkbox-remote locally:
=======
file://
file://
file://
file://
Do you want to submit 'upload to certification' report?
y => yes
n => no
[yn]: n
real 24m50.807s
user 0m23.067s
sys 0m4.435s
this looked initially like it was due to the sosreport job, since that generates a large tarball that is then attached to the submission via the attachment plugin, which with remote, transmits this across whatever remote is using to transfer data from the slave to the master. So I retried this by removing the sosreport jobs.
That seems to have made a huge difference, getting times back down to locally non-remote run attempts:
real 2m52.436s
user 0m15.049s
sys 0m3.158s
So now one last verification, I run checkbox remote over the internet without the sosreport job, and the same test plan that just took 3 minutes is still running after over an hour.
Related branches
- Sylvain Pineau (community): Approve
- Maciej Kisielewski (community): Approve
-
Diff: 5072 lines (+1341/-944)29 files modifiedcheckbox_ng/launcher/master.py (+35/-11)
dev/null (+0/-247)
plainbox/impl/session/remote_assistant.py (+24/-14)
plainbox/vendor/rpyc/__init__.py (+5/-4)
plainbox/vendor/rpyc/core/__init__.py (+2/-2)
plainbox/vendor/rpyc/core/async.py (+1/-1)
plainbox/vendor/rpyc/core/async_.py (+6/-1)
plainbox/vendor/rpyc/core/brine.py (+140/-79)
plainbox/vendor/rpyc/core/channel.py (+22/-8)
plainbox/vendor/rpyc/core/consts.py (+28/-29)
plainbox/vendor/rpyc/core/netref.py (+143/-77)
plainbox/vendor/rpyc/core/protocol.py (+242/-199)
plainbox/vendor/rpyc/core/reactor.py (+7/-3)
plainbox/vendor/rpyc/core/service.py (+38/-16)
plainbox/vendor/rpyc/core/stream.py (+171/-45)
plainbox/vendor/rpyc/core/vinegar.py (+36/-11)
plainbox/vendor/rpyc/lib/__init__.py (+117/-6)
plainbox/vendor/rpyc/lib/colls.py (+28/-5)
plainbox/vendor/rpyc/lib/compat.py (+28/-10)
plainbox/vendor/rpyc/utils/__init__.py (+0/-1)
plainbox/vendor/rpyc/utils/authenticators.py (+6/-8)
plainbox/vendor/rpyc/utils/classic.py (+60/-34)
plainbox/vendor/rpyc/utils/factory.py (+27/-11)
plainbox/vendor/rpyc/utils/helpers.py (+24/-4)
plainbox/vendor/rpyc/utils/registry.py (+32/-27)
plainbox/vendor/rpyc/utils/server.py (+48/-55)
plainbox/vendor/rpyc/utils/teleportation.py (+46/-26)
plainbox/vendor/rpyc/utils/zerodeploy.py (+23/-8)
plainbox/vendor/rpyc/version.py (+2/-2)
Changed in checkbox-ng: | |
assignee: | nobody → Sylvain Pineau (sylvain-pineau) |
status: | New → In Progress |
Changed in checkbox-ng: | |
milestone: | none → 1.8.0 |
Changed in checkbox-ng: | |
status: | In Progress → Fix Committed |
Changed in checkbox-ng: | |
status: | Fix Committed → Fix Released |
After a lot of discussion in Taipei, joc thinks this could be related to just bottlenecks with RPyC and the amount of data being passed back and forth during testing with Checkbox Remote and testflinger connections over the internet.
Also, after talking with Andy, I bumped this to High for our sake because this is going to seriously affect regression testing for us and that's a huge focal point for 2020. We'll be expected quite soon to be testing servers remotely in earnest and this definitely blocks that if a test run takes orders of magnitude longer when run via Checkbox Remote.