Kernel panic when used with upstart after 0.11-4ubuntu2.1 update

Bug #1878723 reported by Duncan Barclay on 2020-05-15
130
This bug affects 19 people
Affects Status Importance Assigned to Milestone
json-c (Ubuntu)
Critical
Unassigned

Bug Description

Installing the 0.11-4ubuntu2.1 security update on a Xenial system with upstart installed, the system crashes with a kernel panic.

The error message is:

[ 99.992278] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[ 99.992278]
[ 99.996057] CPU: 0 PID: 1 Comm: init Not tainted 4.4.0-1105-aws #116-Ubuntu
[ 99.996057] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[ 99.996057] 0000000000000086 0f10ff6977efbf32 ffff88003d45fe10 ffffffff8140926b
[ 99.996057] ffffffff81caddf8 ffff88003d45fea8 ffff88003d45fe98 ffffffff81195a84
[ 99.996057] ffff880000000010 ffff88003d45fea8 ffff88003d45fe40 0f10ff6977efbf32
[ 99.996057] Call Trace:
[ 99.996057] [<ffffffff8140926b>] dump_stack+0x6d/0x92
[ 99.996057] [<ffffffff81195a84>] panic+0xd3/0x227
[ 99.996057] [<ffffffff81088ded>] do_exit+0xb9d/0xba0
[ 99.996057] [<ffffffff81088e77>] do_group_exit+0x47/0xb0
[ 99.996057] [<ffffffff81088ef4>] SyS_exit_group+0x14/0x20
[ 99.996057] [<ffffffff818449db>] entry_SYSCALL_64_fastpath+0x22/0xcb
[ 99.996057] Kernel Offset: disabled

Downgrading to libjson-c2_0.11-4ubuntu2 resolves the issue.

Steps to reproduce:
* Create a system with Xenial installed (I'm using an AWS instance with AMI ami-0f2ed58082cb08a4d)
* Install upstart: apt-get install upstart-sysv
* Reboot
* Update apt and upgrade the packages: apt-get update && apt-get upgrade . This causes the kernel panic.
* To repeat the kernel panic, run dpkg --configure -a

CVE References

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in json-c (Ubuntu):
status: New → Confirmed

I also experienced this bug today. What seems to have happened is that upstart init filled all of the memory on my machine while libjson-c2 was updating, leading to oom-killer killing everything. The kernel panic here would happen if oom-killer tries to act on init. After a reboot my machine managed to upgrade fully before crashing due to running out of memory again.

Peter Sanford (psanford) wrote :

Also ran into this. It looks like this bug got backported https://github.com/json-c/json-c/issues/599. The upstream fix is this: https://github.com/besser82/json-c/commit/7a4807fe0cdb1d9e20273c79762cbf54833aaae4

a1bert (a1bert) wrote :

today's unattended upgrade killed a lot of our hosts running xenial + upstart, please mark as critical

Paride Legovini (paride) on 2020-05-15
Changed in json-c (Ubuntu):
importance: Undecided → Critical
Andy Whitcroft (apw) wrote :

Removed the broken version from the archive, and copied back the previous version.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package json-c - 0.13.1+dfsg-7ubuntu0.2

---------------
json-c (0.13.1+dfsg-7ubuntu0.2) focal-security; urgency=medium

  * Revert the security fixes and rebuild the old version (LP: #1878723)

 -- Chris Coulson <email address hidden> Fri, 15 May 2020 12:31:48 +0100

Changed in json-c (Ubuntu):
status: Confirmed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package json-c - 0.13.1+dfsg-4ubuntu0.2

---------------
json-c (0.13.1+dfsg-4ubuntu0.2) eoan-security; urgency=medium

  * Revert the security fixes and rebuild the old version (LP: #1878723)

 -- Chris Coulson <email address hidden> Fri, 15 May 2020 12:33:18 +0100

Changed in json-c (Ubuntu):
status: Confirmed → Fix Released
Chris Coulson (chrisccoulson) wrote :

The updated packages just revert the security fixes which resolves this immediate issue. Keeping the bug open though because we still want to resolve the underlying issue so that we can republish an update with the security fixes in it

Changed in json-c (Ubuntu):
status: Fix Released → Triaged
tags: added: id-5ebe6d95d5ae066b83cac917
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package json-c - 0.11-4ubuntu2.2

---------------
json-c (0.11-4ubuntu2.2) xenial-security; urgency=medium

  * Revert the security fixes and rebuild the old version (LP: #1878723)

 -- Chris Coulson <email address hidden> Fri, 15 May 2020 12:11:57 +0100

Changed in json-c (Ubuntu):
status: Triaged → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package json-c - 0.12.1-1.3ubuntu0.2

---------------
json-c (0.12.1-1.3ubuntu0.2) bionic-security; urgency=medium

  * Revert the security fixes and rebuild the old version (LP: #1878723)

 -- Chris Coulson <email address hidden> Fri, 15 May 2020 12:33:56 +0100

Changed in json-c (Ubuntu):
status: Triaged → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package json-c - 0.13.1+dfsg-4ubuntu0.2

---------------
json-c (0.13.1+dfsg-4ubuntu0.2) eoan-security; urgency=medium

  * Revert the security fixes and rebuild the old version (LP: #1878723)

 -- Chris Coulson <email address hidden> Fri, 15 May 2020 12:33:18 +0100

Changed in json-c (Ubuntu):
status: Triaged → Fix Released
Eric Desrochers (slashd) on 2020-05-15
Changed in json-c (Ubuntu):
status: Fix Released → Triaged
Leonard Boyce (leonardb4356) wrote :

This is not fixed with the updated package.

Attempts to install libjson-c2_0.11-4ubuntu2.2_amd64.deb result in the same lockup which can only be recovered with a hard reboot.

root@HMPBgZ:~# dpkg -i libjson-c2_0.11-4ubuntu2.2_amd64.deb
(Reading database ... 154353 files and directories currently installed.)
Preparing to unpack libjson-c2_0.11-4ubuntu2.2_amd64.deb ...
Unpacking libjson-c2:amd64 (0.11-4ubuntu2.2) over (0.11-4ubuntu2.1) ...
Setting up libjson-c2:amd64 (0.11-4ubuntu2.2) ...
packet_write_wait: Connection to xx.xx.xx.xx port 22: Broken pipe

Leonard Boyce (leonardb4356) wrote :

Sorry, I posted too soon.

Post reboot:

root@HMPBgZ:~# apt-get update
....
Fetched 51.5 kB in 1s (46.4 kB/s)
Reading package lists... Done
root@HMPBgZ:~# apt-get upgrade
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.
root@HMPBgZ:~# dpkg -s libjson-c2
Package: libjson-c2
Status: install ok half-configured
Priority: extra
Section: libs
Installed-Size: 67
Maintainer: Ubuntu Developers <email address hidden>
Architecture: amd64
Multi-Arch: same
Source: json-c
Version: 0.11-4ubuntu2.2
Config-Version: 0.11-4ubuntu2
Depends: libc6 (>= 2.14)
Description: JSON manipulation library - shared library
 This library allows you to easily construct JSON objects in C,
 output them as JSON formatted strings and parse JSON formatted
 strings back into the C representation of JSON objects.
Homepage: https://github.com/json-c/json-c/wiki
Original-Maintainer: fabien boucher <email address hidden>

Dan Streetman (ddstreet) wrote :

> Attempts to install libjson-c2_0.11-4ubuntu2.2_amd64.deb result in the same lockup which can only be recovered with a hard reboot.

The 2.2 version includes only a revert of the patch, so for any system affected by this, the currently-running init still contains the faulty code, and the fixed package's post-install script calls telinit on the currently-running init, which triggers the problem. Work is in progress to produce a new version that also adjusts the package's post-install script to detect the problematic version and avoid calling telinit.

For impacted systems, you can follow this process to upgrade to the 0.11-4ubuntu2.2_amd64.deb package version:

- Obtain the latest package:
```
$ curl -LO https://launchpad.net/ubuntu/+archive/primary/+files/libjson-c2_0.11-4ubuntu2.2_amd64.deb
```

- Unpack (install without configuring) the package:
```
$ dpkg --unpack libjson-c2_0.11-4ubuntu2.2_amd64.deb
```

- Reboot; this is mandatory in order to proceed:
```
$ sudo reboot
```

- Configure the package:
```
$ sudo apt install -f
```

Damian harouff (cekkent) wrote :

Just want to give a heads up that manual unpack followed by apt install -f still crashes, however the VM automatically rebooted this time (instead of remaining dead) and I was able to proceed with usual apt update afterwards with no further issues.

root@web1:~# curl -LO https://launchpad.net/ubuntu/+archive/primary/+files/libjson-c2_0.11-4ubuntu2.2_amd64.deb
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 21858 100 21858 0 0 10273 0 0:00:02 0:00:02 --:--:-- 164k
root@web1:~# dpkg --unpack libjson-c2_0.11-4ubuntu2.2_amd64.deb
(Reading database ... 39096 files and directories currently installed.)
Preparing to unpack libjson-c2_0.11-4ubuntu2.2_amd64.deb ...
Unpacking libjson-c2:amd64 (0.11-4ubuntu2.2) over (0.11-4ubuntu2.2) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
root@web1:~# reboot
root@web1:~# logout
[root@redacted ~]# vzctl enter 040199a0-e959-4428-87e6-76dd3b26262e
entered into CT
root@web1:/# apt install -f
Reading package lists... Done
Building dependency tree... 50%
[1]+ Stopped apt install -f
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 15 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up libjson-c2:amd64 (0.11-4ubuntu2.2) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Got signal 9
            [root@redacted ~]#

Dan Streetman (ddstreet) wrote :

> Just want to give a heads up that manual unpack followed by apt install -f still crashes

as stated in comment 15, you MUST reboot between dpkg --unpack and apt install -f.

Chris Newcomer (cnewcomer) wrote :

FYI, version 0.11-4ubuntu2.5 was recently released. If you are still experiencing this issue, the following steps will correct it:

1. Remove the postinst file from the affected package (version 0.11-4ubuntu2.1):
sudo rm /var/lib/dpkg/info/libjson-c2\:amd64.postinst

2. Re-run the package configure step to correct its half-installed status:
sudo dpkg --configure -a

3. Update the package list from the repository:
sudo apt update

4. Upgrade the affected package to the latest version:
sudo apt install libjson-c2

5. Reboot to activate the new package:
sudo reboot

Any user running a version of the libjson-c2 package not equal to 0.11-4ubuntu2.1 will not have to go through these steps and updating the package to the new version will be done the normal way; either through unattended-updates or manual apt upgrade.

Hiroyuki YAMAMORI (h-yamamo) wrote :

I have a Xenial server with upstart.
I've made patches for CVE-2020-12762.

See:
https://gist.github.com/h-yamamo/61161ea78f3bb7761e508ca531c7ffff

Thank you.

Hiroyuki YAMAMORI (h-yamamo) wrote :

Official site released the fix about bionic and xenial:
https://github.com/json-c/json-c/commit/f2b7d0b5cbd0eccf4fb3c1851ec0864952be1057

But above has a problem in lh_table_insert():

  if t->count > INT_MAX * LH_LOAD_FACTOR then error return

Thank you.

iGadget (igadget) wrote :

So if I'm on 20.04, not running Upstart and not experiencing kernel panics, why would I want to install this 'update', re-introducing known security issues? Why is it even pushed to systems running 20.04 without Upstart?

John Firebaugh (jfirebaugh) wrote :

@cnewcomer, can you clarify whether or not 0.11-4ubuntu2.5 contains the (fixed) security update?

Zach Marano (zmarano) wrote :

This regression also broke other packages that depend on libjson-c including the google-compute-engine-oslogin package in both 16.04 and 18.04 (at least). The rollback worked fine for the affected users. However what we are trying to determine is if the update to the update works or not. Is there a proposed libjson-c package that includes the upstream fixes referenced in #21 (https://github.com/json-c/json-c/pull/608)?

vinod (pandeyvinod.india) wrote :

This package update is still braking few instances hosted on AWS. it is the same issue what we have seen in version "libjson-c2 0.11-4ubuntu2.1" and impacted a lot of users.

same issue we are facing in the version libjson-c2/now 0.11-4ubuntu2.6 which caused few aws instances to go in kernel panic mode.

do ubuntu have through testing before pushing update and what about revert back. can ubuntu revert back to previous package if unattended upgrade sees any error?

Chris Newcomer (cnewcomer) wrote :

@pandeyvinod.india: Can you help us try to understand why your instances are still panicing?
- Do you have a copy of the panic string?
- How did you update to the 0.11-4ubuntu2.6 package?
- Did you reboot the instance after the package was installed?
- Have any of the instances that previously paniced had a second panic?

Stephane Bakhos (sbakhos) wrote :

I have one instance that kernel panicked when it updated to 0.11-4ubuntu2.6 using auto updates.

However I cannot confirm that I actually rebooted that instance after installing 0.11-4ubuntu2.5
So far none of my other instances panicked.

Can we close this bug?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.