sshuttle autopkgtest is rather lengthy

[Dan Bungert]

Recent test runs of sshuttle autopkgtest are approaching 5 hours.

Here's time stats for 2 example runs on amd64 impish.

1.0.5-1ubuntu1 sshuttle/1.0.5-1ubuntu1 2021-05-05 20:31:35 UTC 4h 59m 27s racb pass log   artifacts
1.0.5-1ubuntu1 sshuttle/1.0.5-1ubuntu1 2021-05-05 20:21:23 UTC 4h 49m 33s rikmills pass log   artifacts

This can be seen as well in local autopkgtest testing.

Can some of the testing be shortened or run in parallel? A quick look at the logs suggests that a lot of time is spent waiting on timeouts. Perhaps the timeouts could be configured to a shorter number in some cases?

[Dan Streetman (ddstreet)]

> Can some of the testing be shortened

Well the test really does nothing other than run 'ssh' from container A to container B before sshuttle is run and again while it's running. So (other than timeouts) the test can't be shortened much. I think most of the time comes from timeouts and the lxd container management.

> or run in parallel?

I wish...I tried, this was rejected:
https://code.launchpad.net/~ddstreet/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/401036

If you can change vorlon's mind that would be great; the 'big' tests get 4 vcpus so things could (theoretically) be parallelized up to 4 tests at once.

However, if the reduction of timeouts and any lxd image handling improvements are done, the test might not need more cores; and also if the main cause of the long test isn't being cpu-bound then parallelization may not help.

> A quick look at the logs suggests that a lot of time is spent waiting on timeouts. Perhaps the timeouts could be configured to a shorter number in some cases?

probably so, i uploaded a new version (to impish) that reduces the expected timeouts down to 20 seconds, and also the detection of when sshuttle had started seems to have not been working in all cases (since it seems the sshuttle output on success varies across releases). Hopefully this will speed it up some.

You might be able to get the length down more by messing with lxd to see if you can use an approach other than snapshotting to put each test instance back into a 'fresh' state; maybe even just re-launching instead of restoring from snapshot.

[Brian Murray (brian-murray)]

Looking at the impish test results it looks like the new tests should take about an hour in addition to fixing any flakiness.

[Brian Murray (brian-murray)]

Hello Dan, or anyone else affected,

Accepted sshuttle into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sshuttle/1.0.4-1ubuntu4.21.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Dan, or anyone else affected,

Accepted sshuttle into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sshuttle/1.0.4-1ubuntu4.20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Dan, or anyone else affected,

Accepted sshuttle into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sshuttle/0.78.5-1ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Dan Bungert (dbungert)]

Local runtime in qemu with the proposed version for hirsute is 27 minutes, versus 4 hours previously. Nice!

[Dan Streetman (ddstreet)]

ran multiple autopkgtests for all archs on all updated releases, only 2 test runs failed:

https://autopkgtest.ubuntu.com/results/autopkgtest-hirsute/hirsute/arm64/s/sshuttle/20210707_130842_cdcc9@/log.gz

https://autopkgtest.ubuntu.com/results/autopkgtest-focal/focal/s390x/s/sshuttle/20210707_130756_78731@/log.gz

those failed due to general networking problems with the autopkgtest infrastructure which is an unfortunate regular issue:

Exception: Could not launch reverse-remote-hirsute: Error: Failed instance creation: Failed getting image: Failed parsing stream: Get "https://cloud-images.ubuntu.com/daily/streams/v1/index.json": Unable to connect to: cloud-images.ubuntu.com:443

Exception: Could not launch reverse-remote-focal: Error: Failed instance creation: Failed getting image: Failed parsing stream: Get "https://cloud-images.ubuntu.com/daily/streams/v1/index.json": Unable to connect to: cloud-images.ubuntu.com:443

[Steve Langasek (vorlon)]

The version of sshuttle in the proposed pocket of Groovy that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion.

[Brian Murray (brian-murray)]

The Hirsute Hippo has reached End of Life, so this bug will not be fixed for that release.