-json output not valid when -class filter used

[HuckleBerry]

When lshw is invoked with the -json option, a single JSON object describing the entire system is produced. Hardware components are nested within the JSON object structure. When the -class filter option is used with -json, one would expect an array of JSON objects, each representing one of the matching hardware components. Instead, the output appears to simply exclude the non-relevant parts of the original JSON object, keeping the parts of the structure that match the filter. The result of which is not a valid JSON object, which is sometimes missing required commas and is always missing the enclosing array brackets.

Examples from two machines with varying hardware layouts are attached (with additional -sanitze option used). Machine 1 with the 'disk' class is the most incorrect.

Machine 1
sudo lshw -version = B.02.16
sudo dpkg -s lshw | grep Version = Version: 02.16-2
lsb_release -rd = Ubuntu 13.10
sudo lshw | grep '*-'
  *-core
     *-firmware
     *-memory:0 UNCLAIMED
        *-bank UNCLAIMED
     *-memory:1
        *-bank
     *-cache:0
     *-cache:1
     *-cache:2
     *-cpu
     *-memory:2 UNCLAIMED
     *-memory:3 UNCLAIMED
     *-pci
        *-pci:0
        *-display
        *-multimedia:0
        *-usb:0
        *-communication
        *-network
        *-usb:1
        *-multimedia:1
        *-pci:1
        *-pci:2
           *-network
        *-pci:3
           *-usb
        *-usb:2
        *-isa
        *-storage
        *-serial UNCLAIMED
     *-scsi:0
        *-disk
     *-scsi:1
        *-disk
           *-volume:0
           *-volume:1
           *-volume:2
     *-scsi:2
        *-disk
           *-volume:0
           *-volume:1
           *-volume:2
     *-scsi:3
        *-disk
     *-scsi:4
        *-disk
     *-scsi:5
        *-disk
           *-volume:0
           *-volume:1
  *-power UNCLAIMED

Machine 2
sudo lshw -version = B.02.16
sudo dpkg -s lshw | grep Version = Version: 02.16-2ubuntu1.2
lsb_release -rd = Ubuntu 14.04.1 LTS
 sudo lshw | grep '*-'
  *-core
     *-firmware:0
     *-cpu:0
     *-memory:0
        *-bank:0
        *-bank:1
     *-firmware:1
     *-cpu:1
     *-memory:1
     *-memory:2 UNCLAIMED
     *-memory:3 UNCLAIMED
     *-pci
        *-isa
        *-ide
        *-bridge UNCLAIMED
        *-display UNCLAIMED
        *-generic
  *-network

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[James Saint-Rossy (james-saintrossy)]

Seeing the same bugs (Missing array brackets and missing commas in between objects)

lsb_release -rd
Description: Ubuntu 14.04.2 LTS
Release: 14.04

lshw -version
B.02.16

lshw -json -C storage -C drive

[Richard Wall (richardw)]

Broken upstream. There are a bunch of related issues:
 * http://www.ezix.org/project/query?status=assigned&status=new&status=reopened&component=lshw&description=~json&order=priority

[Richard Wall (richardw)]

I'm going to give up on the `lshw -class -json` combination.
 * https://github.com/ClusterHQ/flocker/pull/2769

[Murat Uenalan (muenalan)]

Can confirm, still not fixed in B.02.17 (Xenial). Example:

$ sudo lshw -json -c display

  {
[snip .. 3 graphics cards]
  },

which is not JSON. I think the listing of multiple devices it the unsupported case. Should have been

[
  {
[snip .. 3 graphics cards]
  }
]

which has outer brackets, and no trailing comma.

[Bryan Seitz (seitz-a)]

hmm is this still broken in the latest git?

[Bryan Seitz (seitz-a)]

Confirmed json output is still broken in various ways.

[Jeff Silverman (jeffsilverm)]

Still broken:

$ sudo lshw -json -C network | jq .
{
  "id": "network",
  "class": "network",
...
    "100bt-fd": "100Mbit/s (full duplex)",
    "1000bt-fd": "1Gbit/s (full duplex)",
    "autonegotiation": "Auto-negotiation"
  }
}
parse error: Expected value before ',' at line 48, column 4
$

Note that there is no leading [

$ sudo lshw -json -C network | head -5
                                                                                                                                              {
    "id"; : "network",
    "class"; : "network",
    "claimed" : true,
    "handle" : "PCI:0000:00:19.0",

But there is a comma at line 48. Why is jq failing there? Because there is no [. If I insert one, then the problem disappears.

$ sudo lshw -json -C network | head -52 | tail -6
    }
  },
                                                            {
    "id"; : "network",
    "class" : "network",
    "claimed" : true,

Note that there is no ] at the end of the file, which also causes jq to fail.

$ sudo lshw -json -C network | tail -6
    },
    "capabilities" : {
    "ethernet" : true,
    "physical" : "Physical interface"
    }
  }
$

[Vern Hart (vern)]

I can confirm, this is still broken.

In my output, I'm missing the leading and trailing square bracket. Also, between some stanzas there are close and open curly braces (on the same line) without a comma in between.

Since those are the only two syntactical errors in the output I'm working with, this simple one-liner works around it:

(echo "["; lshw -json -C network | sed 's/} .*{/}, {/'; echo "]") | jq .

[Russell Jones (russell-jones-oxphys)]

this has actually got worse, as it now (as of 19.10 / 02.18.85-0.3 at least) breaks when no class is used
e.g.

$ lshw -json -quiet | python3 -m json.tool
Expecting ',' delimiter: line 26, column 8 (char 785)

lshw -xml | xmllint -

seems to indicate xml output works ok, but the format is annoying :(

[Russell Jones (russell-jones-oxphys)]

https://github.com/lyonel/lshw/tree/B.02.18 outputs valid json with

lshw -json -quiet

though it does have the problem described in https://bugs.launchpad.net/ubuntu/+source/lshw/+bug/1593233 i.e. can only run as root, and crashes otherwise.

Using git bisect, the change that broke output was https://github.com/lyonel/lshw/pull/28
See also https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=929206

[Russell Jones (russell-jones-oxphys)]

One can see the full extent of the brokenness of the command (in all versions I've seen) using

#!/usr/bin/env python3

from itertools import combinations, chain, repeat
from subprocess import check_output as sco
import json

cmd = ['lshw', '-json', '-quiet']

classes = 'system bus memory processor bridge display generic multimedia storage network'.split()

if __name__ == '__main__':
    for flags in chain(*(combinations(classes, n) for n in range(len(classes)))):
        flags = chain(*zip(repeat('-c'), flags))
        flags = list(flags)
        if True:
            try:
                json.loads(sco(cmd + flags))
                print(" ".join(flags), "was OK")
            except json.decoder.JSONDecodeError as e:
                print(' '.join(flags))
                print(e)

[Nilson Lopes (noslin005)]

There are no fixes yet for Ubuntu 19.10 (Eoan Ermine).

sudo lshw -json -quiet | python3 -m json.tool
Expecting ',' delimiter: line 26 column 8 (char 792)

sudo dpkg -s lshw | grep Version
Version: 02.18.85-0.3

[Vern Hart (vern)]

This no longer appears to be a problem on bionic or lunar (only two systems I had handy).

Both jq and python3 -m json.tool show no errors. I also ran Russell's script from comment #13 (that's a LOT of combinations!) and had no exceptions.

[Waldek M. (weakcamel)]

Personally, I'm still seeing an issue related to commas in json output on Bionic.

$ apt show lshw 2>/dev/null | head -n2
Package: lshw
Version: 02.18-0.1ubuntu6.18.04.2

$ sudo lshw -C display -json | jq
{
  "id": "display",
  "class": "display",
  "claimed": true,
  "handle": "PCI:0000:00:0f.0",
  "description": "VGA compatible controller",
  "product": "SVGA II Adapter",
  "vendor": "VMware",
  "physid": "f",
  "businfo": "pci@0000:00:0f.0",
  "version": "00",
  "width": 32,
  "clock": 33000000,
  "configuration": {
    "driver": "vmwgfx",
    "latency": "64"
  },
  "capabilities": {
    "vga_controller": true,
    "bus_master": "bus mastering",
    "cap_list": "PCI capabilities listing",
    "rom": "extension ROM"
  }
}
parse error: Expected value before ',' at line 24, column 4

===

It's basically a spurious comma at the end ouf the output

$ sudo lshw -C display -json
# ...[several spurious newlines] {
    "id" : "display",
    "class" : "display",
    "claimed" : true,
    "handle" : "PCI:0000:00:0f.0",
    "description" : "VGA compatible controller",
    "product" : "SVGA II Adapter",
    "vendor" : "VMware",
    "physid" : "f",
    "businfo" : "pci@0000:00:0f.0",
    "version" : "00",
    "width" : 32,
    "clock" : 33000000,
    "configuration" : {
      "driver" : "vmwgfx",
      "latency" : "64"
    },
    "capabilities" : {
      "vga_controller" : true,
      "bus_master" : "bus mastering",
      "cap_list" : "PCI capabilities listing",
      "rom" : "extension ROM"
    }
  },