majority of packages get into "unknown" instead of e.g. universe

[Yaroslav Halchenko]

If you visit https://popcon.ubuntu.com/ and download "universe" (https://popcon.ubuntu.com/universe/by_inst.gz) and "unknown" (https://popcon.ubuntu.com/unknown/by_inst) you will see that many packages which are in universe (e.g., git-annex) are not present in "universe" but rather in "unknown":

universe:

 $> zcat <(wget -q -O- https://popcon.ubuntu.com/universe/by_inst.gz) | nl | tail
 18208 18197 mythbuntu-diskless-client 1 0 1 0 0 (Michael Haas)
 18209 18198 roxen4-doc 1 0 0 0 1 (Turbo Fredriksson)
 18210 18199 zorroutils 1 0 1 0 0 (Aurélien GÉrÔme)
 18211 18200 fai-nfsroot 0 0 0 0 0 (Thomas Lange)
 18212 18201 gforge-theme-starterpack 0 0 0 0 0 (Christian Bayle)
 18213 18202 libslurm11-dev 0 0 0 0 0 (Gennaro Oliva)
 18214 18203 ppmtofb 0 0 0 0 0 (Chris Lawrence)
 18215 18204 rhkbf 0 0 0 0 0 (Dmitry E. Oboukhov)
 18216 ------------------------------------------------------------------
 18217 18204 Total 447833653 271213 332353797 22489 115186154
 $> zcat <(wget -q -O- https://popcon.ubuntu.com/universe/by_inst.gz) | grep git-annex

unknown:

 $> wget -q -O- https://popcon.ubuntu.com/unknown/by_inst | nl | tail
 296594 296583 zope-zaaplugins 0 0 0 0 0 (Unknown)
 296595 296584 zope-zattachmentattribute 0 0 0 0 0 (Unknown)
 296596 296585 zope-zpatterns 0 0 0 0 0 (Unknown)
 296597 296586 zope-zshell 0 0 0 0 0 (Unknown)
 296598 296587 zope2.11-sandbox 0 0 0 0 0 (Unknown)
 296599 296588 zope2.6-verbosesecurity 0 0 0 0 0 (Unknown)
 296600 296589 zope2.7-sandbox 0 0 0 0 0 (Unknown)
 296601 296590 zopex3-sandbox 0 0 0 0 0 (Unknown)
 296602 ------------------------------------------------------------------
 296603 296590 Total 1319985827 2097898 913982492 122381 403783056

 $> wget -q -O- https://popcon.ubuntu.com/unknown/by_inst | nl | grep git-annex
 23764 23753 git-annex 747 6 740 1 0 (Unknown)
 129581 129570 elpa-magit-annex 5 0 5 0 0 (Unknown)
 138155 138144 elpa-git-annex 4 0 4 0 0 (Unknown)
 168880 168869 git-annex-standalone 2 0 2 0 0 (Unknown)

whenever git-annex is in universe: https://packages.ubuntu.com/disco/git-annex

NB original motivation for inspecting ubuntu popcon is because in NeuroDebian we saw slow decrease (but not complete disappearance) of popcon submissions from Ubuntu systems: see http://neuro.debian.net/popularity.html#popularity-contest . We know that it doesn't reflect reality since the number of "subscriptions" (downloads of our APT configuration lines) is dominated by Ubuntu boxes: http://neuro.debian.net/popularity.html#repository-subscriptions . So there is yet another mystery to be resolved.

[Yaroslav Halchenko (yarikoptic)]

regarding the original issue + NB: additional point of information:

 $> zgrep neurodebian-popularity-contest by_inst*
 by_inst:14151 neurodebian-popularity-contest 726 3 41 1 681 (Neurodebian Team)
 by_inst_ubuntu_unknown.gz:18478 neurodebian-popularity-contest 1326 0 29 0 1297 (Unknown)

so there is ~2k popcon submitters on ubuntu, but for some submissions get into unknown, and we receive probably only 10% of them at our popcon server.