script hangs and never finishes

[Jason Grammenos]

The /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
job that runs weekly appears to hang and never complets, slowly building up the number of hanging processes

user@archive:~$ ps -ef | grep check-new
root 1618 1 0 Feb26 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 3489 1 0 Jul09 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 5951 1 0 Mar19 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 14385 1 0 Apr02 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 14490 1 0 Jul23 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 15157 1 0 Apr30 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 17853 1 0 Jun04 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 21845 1 0 Jun11 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 22240 1 0 May21 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 22607 1 0 Jul02 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 23452 1 0 Apr16 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 27171 1 0 Feb19 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 28447 1 0 Jun18 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 29531 1 0 Mar05 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 30531 1 0 Jun25 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 31763 1 0 May14 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 32360 1 0 May07 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q

user@archive:~$ sudo strace -p 32360
Process 32360 attached
futex(0x27b0fd0, FUTEX_WAIT_PRIVATE, 0, NULL

Running Trusty

user@archive:~$ lsb_release -d
Description: Ubuntu 14.04.5 LTS

user@archive:~$ dpkg -l | grep ubuntu-re
ii ubuntu-release-upgrader-core 1:0.220.9 all manage release upgrades

user@archive:~$ apt-cache policy ubuntu-release-upgrader-core
ubuntu-release-upgrader-core:
  Installed: 1:0.220.9
  Candidate: 1:0.220.9
  Version table:
 *** 1:0.220.9 0
        500 http://ca.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1:0.220.2 0
        500 http://ca.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

Any assistance in resolving these hanging processes that grow for no apparent reason.
I could simply disable the cron job, but i am trying to get to the root of the issues and resolve it.

Expected behaviour:
The script should terminate and close the process.

what happened:
the script hang and runs in background forever.

[Brian Murray (brian-murray)]

Thanks for reporting this issue, what does /etc/update-manager/release-upgrades contain?

[Jason Grammenos (jason.grammenos.agility)]

user@mm02archive:~# cat /etc/update-manager/release-upgrades
# Default behavior for the release upgrader.

[DEFAULT]
# Default prompting behavior, valid options:
#
# never - Never check for a new release.
# normal - Check to see if a new release is available. If more than one new
# release is found, the release upgrader will attempt to upgrade to
# the release that immediately succeeds the currently-running
# release.
# lts - Check to see if a new LTS release is available. The upgrader
# will attempt to upgrade to the first LTS release available after
# the currently-running one. Note that this option should not be
# used if the currently-running release is not itself an LTS
# release, since in that case the upgrader won't be able to
# determine if a newer release is available.
Prompt=lts

[Brian Murray (brian-murray)]

Can you connect to the changelog server?

(trusty-amd64)root@impulse:/tmp# wget http://changelogs.ubuntu.com/meta-release-lts
--2017-08-15 08:04:50-- http://changelogs.ubuntu.com/meta-release-lts
Resolving changelogs.ubuntu.com (changelogs.ubuntu.com)... 2001:67c:1560:8008::11, 91.189.95.15
Connecting to changelogs.ubuntu.com (changelogs.ubuntu.com)|2001:67c:1560:8008::11|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3683 (3.6K)
Saving to: 'meta-release-lts'

100%[============================================================================================================================>] 3,683 --.-K/s in 0s

2017-08-15 08:04:51 (728 MB/s) - 'meta-release-lts' saved [3683/3683]

[Jason Grammenos (jason.grammenos.agility)]

Yes i can connect to the changelog server.

user@archive:~$ wget http://changelogs.ubuntu.com/meta-release-lts
--2017-08-15 11:12:37-- http://changelogs.ubuntu.com/meta-release-lts
Resolving changelogs.ubuntu.com (changelogs.ubuntu.com)... 91.189.95.15, 2001:67c:1560:8008::11
Connecting to changelogs.ubuntu.com (changelogs.ubuntu.com)|91.189.95.15|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3683 (3.6K)
Saving to: ‘meta-release-lts’

100%[===============================================================================================================================================>] 3,683 --.-K/s in 0s

2017-08-15 11:12:37 (278 MB/s) - ‘meta-release-lts’ saved [3683/3683]

[Jason Grammenos (jason.grammenos.agility)]

Hello, the ticket is still marked as incomplete, what additional information do you need from me.

[Jason Grammenos (jason.grammenos.agility)]

user@somenode:~$ ps -ef | grep check-new
root 5102 1 0 Jul30 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 9228 1 0 Jul16 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 16718 1 0 Jul02 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 18286 1 0 Aug20 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 40711 1 0 Jun25 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 42309 1 0 Jul23 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 44891 1 0 Jun18 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 49048 1 0 Jun11 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 56328 1 0 Aug27 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 67336 1 0 Aug06 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 89995 1 0 Jul09 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 130476 1 0 Aug13 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
user@somenode:~$ lsb_release -d
Description: Ubuntu 16.04.2 LTS

user@somenode:~$ dpkg -l | grep ubuntu-re
ii ubuntu-release-upgrader-core 1:16.04.21 all manage release upgrades

user@somenode:~$ sudo strace -p 130476
strace: Process 130476 attached
futex(0x115c510, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff^Cstrace: Process 130476 detached
 <detached ...>

I can now confirm that this also occurs on 16.04

[Jason Grammenos (jason.grammenos.agility)]

This is becoming a persistent problem on numerous servers that I monitoring/maintain.
I am seriously considering disabling this cron.
Is there anyway I could get some traction on this issue?

[Jason Grammenos (jason.grammenos.agility)]

this issue persist

jason@server03:~$ pgrep check-new-re
765
1925
1952
3852
4072
5238
5467
5646
6464
6753
9409
12877
14336
15288
16599
17057
18756
20018
20029
20174
20255
20726
22315
22523
22855
24292
24876
25964
26093
27981
28489
29387
30350
32556
jason@server03:~$ pgrep check-new-re | wc -l
34

jason@server03:~$ ps -ef | grep check-new-re
root 765 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 1925 1 0 Feb04 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 1952 1 0 Mar11 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 3852 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 4072 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 5238 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 5467 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 5646 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 6464 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 6753 1 0 Jan21 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 9409 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 12877 1 0 Feb25 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 14336 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 15288 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 16599 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 17057 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 18756 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 20018 1 0 Mar18 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 20029 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 20174 1 0 Jan28 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 20255 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 20726 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 22315 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
root 22523 1 0 2017 ? 00:00:00 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
...