Update_excuses is permanently outdated

[Balint Reczey]

There is several hours between updates when there are big ongoing transitions.

This is too long to work with the page when it is needed the most.

It should be regenerated under 10 minutes.

[Balint Reczey (rbalint)]

One idea to speed it up:

https://irclogs.ubuntu.com/2020/11/19/%23ubuntu-release.html#t09:39
...
doko let's wait for the next two britney runs, 5h... 09:39
doko Laney: did you try to run britney using pypy3? 09:41
doko hmm, python-apt as a C extension. but nothing else 09:48
sil2100 tjaalton: sweet! 09:51
Laney doko: no, but that's probably not the problem, if you read any of the logs it's spending almost all the time hitting a remote https server to look for new test results 10:04
Laney we should get those in bulk in advance somehow 10:04
...

[Balint Reczey (rbalint)]

Getting the results in bulk by dowloading autopkgtest.db and working from that seem to be viable.

autopkgtest.db seem to be refreshed in every ~15 minutes. Can we make it every 5 minutes maybe in per-release filtered scheme?

$ while curl -s -v -X HEAD http://autopkgtest.ubuntu.com/static/autopkgtest.db 2>&1 | grep -m 1 '^< Last-Modified:' ; do sleep 30; done
< Last-Modified: Thu, 28 Jan 2021 14:06:42 GMT
< Last-Modified: Thu, 28 Jan 2021 14:06:42 GMT
< Last-Modified: Thu, 28 Jan 2021 14:20:21 GMT

The data in the .db is a few minutes old:

$ sqlite3 autopkgtest.db
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> select run_id FROM result WHERE run_id > '20210127' ORDER BY run_id DESC LIMIT 1;
20210128_141721_baa3a@

A few minutes should be enough to do the rest of the processin, so if autopkgtest.db can be exported in every ~5 minutes the Britney run can finish in ~10 minutes after the last test affecting the run finished.

I'm proceeding with implementing using autopkgtest.db in Britney.

[Iain Lane (laney)]

On Thu, Jan 28, 2021 at 02:33:15PM -0000, Balint Reczey wrote:
> I'm proceeding with implementing using autopkgtest.db in Britney.

It's fine to do this, although I wish you would coordinate before
deciding on a course of action, but:

  1) This must be done in a way that can be upstreamed, and then
  actually upstreamed. We can't be in a position again where it's
  several weeks of someone's time to rebase britney.

  2) Eventually I think we should move to a proper database so that the
  website itself can be made faster. If you could keep that in mind then
  that migration could be made a lot easier by this work.

Cheers,

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]

[Balint Reczey (rbalint)]

ACK, I'll keep those in mind.

[Steve Langasek (vorlon)]

I've implemented support for reading results from the sqlite db instead of from swift at <https://code.launchpad.net/~vorlon/britney/+git/britney2-ubuntu/+ref/adt-results-from-swift>, and this shows no real improvement in the timing of a britney run. It turns out autopkgtest result collection is not a bottleneck.

Here are some results instead from cProfile:

         3560963156 function calls (3516702356 primitive calls) in 2408.282 seconds

   Ordered by: standard name

   ncalls tottime percall cumtime percall filename:lineno(function)
      140 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:119(release)
      136 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:159(__init__)
      136 0.000 0.000 0.002 0.000 <frozen importlib._bootstrap>:163(__enter__)
      136 0.000 0.000 0.001 0.000 <frozen importlib._bootstrap>:170(__exit__)
      140 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:176(_get_module_lock)
      136 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:190(cb)
        4 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:195(_lock_unlock_module)
   178/23 0.000 0.000 0.193 0.008 <frozen importlib._bootstrap>:214(_call_with_frames_removed)
      136 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:225(_verbose_message)
       13 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:235(_requires_builtin_wrapper)
      136 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:310(__init__)
      136 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:314(__enter__)
      136 0.000 0.000 0.001 0.000 <frozen importlib._bootstrap>:321(__exit__)
      544 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:324(<genexpr>)
      115 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:35(_new_module)
      137 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:372(__init__)
      236 0.000 0.000 0.006 0.000 <frozen importlib._bootstrap>:406(cached)
      136 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:419(parent)
      136 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:427(has_location)
       13 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:436(spec_from_loader)
      136 0.001 0.000 0.008 0.000 <frozen importlib._bootstrap>:510(_init_module_attrs)
  136/135 0.000 0.000 0.036 0.000 <frozen importlib._bootstrap>:570(module_from_spec)
   136/19 0.001 0.000 0.204 0.011 <frozen importlib._bootstrap>:659(_load_unlocked)
      136 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:716(find_spec)
       13 0.000 0.000 0.001 0.000 <frozen importlib._bootstrap>:737(create_module)
      136 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:74(__init__)
       13 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:745(exec_module)
       13 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:762(is_package)
      123 0.000 0.000 ...

[Steve Langasek (vorlon)]

sort -n -k2 -r shows these are the top calls:

2944024185 942.398 0.000 942.398 0.000 hints.py:88(type)
   256020 904.025 0.004 1869.080 0.007 hints.py:38(<listcomp>)
  1670374 94.349 0.000 142.543 0.000 tester.py:424(_check_loop)
     1561 49.672 0.032 49.672 0.032 {method 'read' of '_ssl._SSLSocket' objects}
   532381 22.663 0.000 22.680 0.000 autopkgtest.py:503(has_autodep8)
    21860 21.483 0.001 22.520 0.001 autopkgtest.py:1261(check_ever_passed_before)
 33388156 15.323 0.000 15.381 0.000 {method 'update' of 'set' objects}
 31577054 14.433 0.000 14.433 0.000 universe.py:118(relations_of)
    43720 13.408 0.000 14.854 0.000 autopkgtest.py:1110(result_in_baseline)
   230608 12.665 0.000 1654.613 0.007 excusefinder.py:74(_should_upgrade_srcarch)
45813036/15528599 12.656 0.000 23.659 0.000 encoder.py:332(_iterencode_dict)
10874803/53327 12.116 0.000 31.761 0.001 copy.py:137(deepcopy)

And here btw in the full invocation: python3 -m cProfile /home/ubuntu-archive/proposed-migration.test/code/b2/britney.py -c /home/ubuntu-archive/proposed-migration.test/code/b2/britney.conf.ubuntu.impish -v --distribution=ubuntu --series=impish

The top call is surprising, the call itself is trivial but it seems to have been called an extraordinary number of times. I'll dig to see what's going on there.

[Steve Langasek (vorlon)]

https://code.launchpad.net/~vorlon/britney/+git/britney2-ubuntu/+merge/409703

[Steve Langasek (vorlon)]

We have now landed four branches to address performance:
- https://code.launchpad.net/~vorlon/britney/+git/britney2-ubuntu/+merge/409703
- https://code.launchpad.net/~vorlon/britney/+git/britney2-ubuntu/+merge/409806
- https://code.launchpad.net/~vorlon/britney/+git/britney2-ubuntu/+merge/409810
- https://code.launchpad.net/~vorlon/britney/britney1-nuninst-cache/+merge/409871

These are effective at reducing the time of a "minimal" britney run from over an hour to under a half hour.

The remaining bottlenecks that I see are:
- processing badtest hints. This isn't something we can short-circuit because the vast majority of hints we have are badtest hints; either we need to optimize the hint handling or reduce the number of hints (the last is a side effect of implementing baseline testing).
- submitting autopkgtest requests. There is probably little we can do to optimize this.

So I'm considering this done.