some changelogs are missing from the server used

[Sebastien Bacher]

g-s uses http://changelogs.ubuntu.com/changelogs/binary but that server seems to miss some updates

example
http://changelogs.ubuntu.com/changelogs/binary/f/firefox/45.0+build2-0ubuntu1

where that one works
http://changelogs.ubuntu.com/changelogs/pool/main/f/firefox/firefox_45.0+build2-0ubuntu1/changelog

update-manager is using the second url, we should either get the server fixed or change to use the working one (the url includes the component to use so we would need to get that information in get_changelog, also the version need to be stripped from its epoch if there is one)

[Robert Ancell (robert-ancell)]

Since it's harder to get the second URL I'll wait to see if the server can be fixed.

[Matthias Klumpp (ximion)]

FWIW, since APT 1.2 we have an automatic facility to download the correct changelog via APT, so there is actually no need to implement something manually again - APT will do the right thing.
Not sure how this is implemented in the GS APT plugin, but using APTs pkgAcqChangelog is always the correct way to get changelogs with nwer APT.

[Sebastien Bacher (seb128)]

Thanks Matthias, the current apt plugin loads directly the dpkg/apt db on disk and does the parsing directly reading those files (e.g not apt api is used) so unsure how practical it would be to do such change. In practice reading the url should work in all case if the server was correctly updating...

[Brian Murray (brian-murray)]

Looking at the logs on the server it is no longer regularly extracting changelogs.

INFO:root:skipped: 99, extracted: 0
INFO:root:last checked: 2016-03-10 07:38:01.930000
INFO:root:libotr 4.1.0-2ubuntu0.1 in PENDING state
INFO:root:libotr 4.0.0-2.2ubuntu1.1 in PENDING state
INFO:root:libotr 3.2.0-4ubuntu0.3 in PENDING state
INFO:root:nova 13.0.0~b3-0ubuntu1 in PENDING state
INFO:root:skipped: 99, extracted: 0
INFO:root:last checked: 2016-03-10 07:39:02.070000
INFO:root:extracting changelog for libotr 4.1.0-2ubuntu0.1
INFO:root:extracting changelog for libotr 4.0.0-2.2ubuntu1.1
INFO:root:extracting changelog for libotr 3.2.0-4ubuntu0.3
INFO:root:nova 13.0.0~b3-0ubuntu1 in PENDING state
INFO:root:skipped: 99, extracted: 3
INFO:root:last checked: 2016-03-10 07:40:02.480000
INFO:root:docker.io 1.10.2-0ubuntu2 in PENDING state
INFO:root:nova 13.0.0~b3-0ubuntu1 in PENDING state
INFO:root:skipped: 101, extracted: 0

I was seeing warnings regarding "another extractor is running" and will dig some more.

[Matthias Klumpp (ximion)]

@seb128: That was only a reply to Roberts "it's harder" statement, showing the easiest way (tm) to solve this.
The server issue itself should of course be resolved anyway - you never know what else is using the data...
Cheers :)

[Brian Murray (brian-murray)]

Its worth noting that the server had changes that weren't committed to a branch in Launchpad making things more "interesting".

[Brian Murray (brian-murray)]

I've shuffled things around in the code and I'm manually running the extractor (for the past 30 days) and it seems to be creating missing symlinks for multiple packages.

DEBUG:root:skipping already existing './changelogs/pool/universe/h/haskell-graphviz/haskell-graphviz_2999.18.0.2-1build1'
DEBUG:root:source package: 'haskell-gtk: 0.13.9-3build1 (2016-02-10 15:18:47.607495+00:00)'
DEBUG:root:skipping already existing './changelogs/pool/universe/h/haskell-gtk/haskell-gtk_0.13.9-3build1'
DEBUG:root:create compat symlink './changelogs/pool/universe/h/haskell-gtk/haskell-gtk_0.13.9-3build1' -> './changelogs/binary/libg/libghc-gtk-doc/0.13.9-3build1'
DEBUG:root:create compat symlink './changelogs/pool/universe/h/haskell-gtk/haskell-gtk_0.13.9-3build1' -> './changelogs/binary/libg/libghc-gtk-prof/0.13.9-3build1'
DEBUG:root:create compat symlink './changelogs/pool/universe/h/haskell-gtk/haskell-gtk_0.13.9-3build1' -> './changelogs/binary/libg/libghc-gtk-dev/0.13.9-3build1'
DEBUG:root:source package: 'haskell-gtk-traymanager: 0.1.5-5build1 (2016-02-10 15:18:47.607495+00:00)'

[Brian Murray (brian-murray)]

The extractor has now created symlinks that were missing within the past 30 days. If you think we need more let me know and I'll kick off another run over a longer time period.

[Brian Murray (brian-murray)]

I've also made the log file include some additional information which revealed that the "already running message" is normal because the cronjob is running continuously. For example:

2016-03-11 16:56:02,145:16886:last checked: 2016-03-11 08:53:03.120000
2016-03-11 16:56:09,308:16886:extracting changelog for backup2l 1.5-8
2016-03-11 16:56:13,172:16886:extracting changelog for csmith 2.2.0-2
2016-03-11 16:56:19,553:16886:extracting changelog for eterm 0.9.6-4
2016-03-11 16:56:23,640:16886:ksysguard 5.5.4-0ubuntu1 in PENDING state
2016-03-11 16:57:01,602:17180:another extractor is running, exiting
2016-03-11 16:57:37,987:16886:extracting changelog for mistral 2.0.0~b3-3
2016-03-11 16:58:01,831:17405:another extractor is running, exiting
2016-03-11 16:59:04,324:17635:another extractor is running, exiting
2016-03-11 16:59:14,534:16886:extracting changelog for netbeans 8.1+dfsg2-3

Notice the different process ids.