Hardware Database not browseable

[Tero Karvinen]

Cannot browse hardware database. Only very generic data without any details is available on http://hwdb.ubuntu.com/ .

Hardware database ids are sometimes refered to in Ubuntu bugs, but I could not find any way to see the actual records. Collecting the data is not very usefull if it is not accessible. Who owns the data now, are they property of some company?

Hardware database related pages in Ubuntu wiki have not been modified for a while
https://wiki.ubuntu.com/UbuntuDownUnder/BOFs/HardwareDatabaseRoadmap?highlight=%28HardwareDatabase%29
https://wiki.ubuntu.com/HardwareDatabase?highlight=%28HardwareDatabase%29

[Oliver Grawert (ogra)]

you can wget the dataset and inspect it locally:
wget http://hwdb.ubuntu.com/2c4003b0aa8f0403128d43ca208dbc00.xml.bz2

sorry, i havent gotten around to implement a SQL backend yet, if somenone is eager to do it, please contact me.

[Dennis Kaarsemaker (dennis)]

Ah, why not. If you can give me the specs of the data structures (and
the way hwdb-client wants to interact with the server) I will try and
make a nice django model of it, after that first step we can think about
searching.

[Tero Karvinen (karvinen+launchpad)]

This bug has been marked "in progress" for five months. Is someone still working on the browsable hardware database?

Looking forward to see the hardware data...

[Tero Karvinen (karvinen+launchpad)]

It has been over half a year since last comment about any progress on this.

[Robert Collins (lifeless)]

Hardware database browsing is not likely in the short term : if someone wants to enhance the hwdb-client and submit a patch, so that the 'show online record' pulls the bz2 file and shows data from it, that might be nice. Our immediate focus with the client is to start making it useful for driving tests from it. This requires backend work on the system.

[Tero Karvinen (karvinen+launchpad)]

Who has access to all hardware database data?

That data would be really usefull when deciding what hardware to buy or when developing software.

[Shirish Agarwal (shirishag75)]

hi all,
       I am guessing that http://hwdb.ubuntu.com/2c4003b0aa8f0403128d43ca208dbc00.xml.bz2 in that the numbers before the datasheet are the person who has his own datasheet. I tried both wget http://hwdb.ubuntu.com/2c4003b0aa8f0403128d43ca208dbc00.xml.bz2 as well as inserting mine with wget http://hwdb.ubuntu.com/bb5847b52512fe16a507a8df0886486d.xml.bz2 and I get the same error.

Resolving hwdb.ubuntu.com... 82.211.81.183
Connecting to hwdb.ubuntu.com|82.211.81.183|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
19:13:41 ERROR 403: Forbidden.

According to wikipedia http://en.wikipedia.org/wiki/HTTP_403 :-

The 403 Forbidden error message is an HTTP status code indicating that the client was able to communicate with the server, but the server doesn't let the user see what was requested.

An example of descriptive English text that can often accompany such a response is You don't have permission to access /foo on this server.

    Can somebody tell me why are we forbidden to see our own submissions, or for that matter being able to see some sample submission. The bug was filed about a yr. and a month back.

[Greg Olyo (gregolyo)]

Browsable would be indeed be nice. BUT...

I think it is even more important to make it clear to those who are contributing their hardware
data that *not being able to retrieve it* does NOT (necessisarily) mean that their submission failed.

The wording currently returned for (aparently all) queries from
   http://hwdb.ubuntu.com/?xml=00000000000000000000000000000000
to
   http://hwdb.ubuntu.com/?xml=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
includes the words "If you dont see any cpu or memory data below, you might have sent a broken file,".

This definitely works against the goal of "Increase[ing] participation in the Ubuntu Hardware Database".
   see: https://blueprints.launchpad.net/ubuntu/+spec/increase-hwdb-participation

May I suggest that the text of the "interim page" (shown below) be changed ASAP to indicate clearly that
the retreval feature currently does NOT work. It might also be good to include a link to this page:
https://bugs.launchpad.net/ubuntu/+source/hwdb-client/+bug/30910

Also, if there is any quick fix that could positively indicate that the submitted data was recieved and
appears to be valid, that would be very helpful.

Thanks for listening,

-Greg

================================================================
Ubuntu hardware database
Submissions Total: 619467 Today: 413241

Page autorefresh: every 10min
Ubuntu hardware database entry : 835a2d0b2d075ce4051a553f1211537a

This is an interim page, that does not show more then some basic data from the dataset.
If you dont see any cpu or memory data below, you might have sent a broken file,
this is most likely the case if hal is not running or has the wrong (non ubuntu hoary or breezy) version.

Couldnt detect laptop
(assuming Desktop)
=================================================================

[Roland Wolters (wolters-liste)]

As long as the hwdb is not browseable there should be a static page stating exactly that.
In the current state it just looks broken and since it is a major Ubuntu page this doesn't look good at all.

A text which might be better than the current broken interface would be:

The Ubuntu Hardware Database is currently not browseable. However, the data are aggregated and can be retrieved in a xml package from this address:
http....(path to xml package).

[Tommy Trussell (tommy-trussell)]

adding my note of agreement that the current arrangement is a problem -- I have tried using the hwdb several times since the nice GUI was added, and the vague adminishment that "you might have sent a broken file" is very frustrating, especially when submitting data for a malfunctioning machine.

[Kapis (capiscuas)]

That is indeed very frustating. I'm doing laptop ubuntu testing and sometimes I would like to ask retrieve the info from the Ubuntu HW Database and apparently doesn't work.
Pls fix this ASAP.

[Andrew Fenn (andrewfenn)]

Is anyone working on this bug?

Since no one has the code apart from the guys at hwdb.ubuntu.com what does anyway expect us hackers to do?

[Adam Niedling (krychek)]

Setting this to "In Progress" based on:
http://brainstorm.ubuntu.com/idea/40/

[Tero Karvinen (karvinen+launchpad)]

After two years, hwdb is still not browsable. Hardy beta seems to encourage users to send data.

Many questions are still open two years after opening the bug:
- Who owns the data in Ubuntu Hardware Database? Would Canonical like to own it?
- When will everyone get access to HwDb?
- Why doesn't hardware database client clearly state that information is sent to a closed database?

[Adam Niedling (krychek)]

http://ubuntuhcl.org/

I think this is satisfying enough.

[Tero Karvinen (karvinen+launchpad)]

Thanks for the link to the forum. However, it does not seem to be the Ubuntu hardware database.

I only found short, written comments about computers and hardware. None of this information seemed to come from hardware database client. Comments did not seem to contain exact technical details collected by automatic software.

Could you point out where this site has information from Ubuntu hardware database, collected by the hardware database client? Thanks already.

[Adam Niedling (krychek)]

The data there wasn't collected by the hardware database client but it was sent by users. I think it's more useful to read human words rather than automated computer language. The database client doesn't even work for me it freezes all the time and even when I have sound the client doesn't play any sound.
Don't you agree it's better this way?

[Roland Wolters (wolters-liste)]

The current hwdb is broken. Pointing to some other homepage is no fix at all: the hwdb is still there, not browseable and data are still generated by the users. Stating that this bug is fixed doesn't help anyone because it is simply wrong.

So either you close down the hwdb or you fix it. If you close it down, then you might want to rely on some homepage with user generated content - but that's not the question here.

[Oliver Grawert (ogra)]

assigning to the appropriate package so the right people get aware of it

[Tero Karvinen (karvinen+launchpad)]

Adam: As Roland already pointed out, a web forum is not a solution to this bug. And no, I don't think a web forum is better than a working hardware database.

Hard, detailed technical data about a setup that's verified by human to work would be valuable.

[jogwi (jogwi)]

Sad that this missing feature is still pending to be implemented. Where is the above mentioned web interface to read the hwdb-entries?

[Pablo Castellano (pablocastellano)]

It would be very useful. Don't let this discussion to be forgotten.

[Jim Fisher (jedijf)]

Thanks for taking the time to make Ubuntu better. This has been addressed in Ubuntu Brainstorm (See Above).

Requesting that this be set to Wishlist.

Thanks for taking the time to share your opinion!

[Stéphane Maniaci (stephh)]

What's up with this ? It's been (apparently) 3 years since http://hwdb.ubuntu.com is down, and I think it would be really a great improvement to fix it, so that users can check their hardware before installing Ubuntu, get in touch with others users if they encounter any problem, etc. Any news ?

[David Murphy (schwuk)]

Marking as invalid for hwtest (which has since been renamed to checkbox). Checkbox/hwtest replaced hwdb-client in 8.04, and is not configured to submit data to http://hwdb.ubuntu.com. Instead, data is submitted to Launchpad where it is accessible via the API, and in the future the site itself. There are currently no plans to start submitting data to hwdb.ubuntu.com again.

[Tero Karvinen (karvinen+launchpad)]

Dave: so how can I view the whole hardware database then?

We still don't know who owns the data. We still can't browse the data. Names and URLs change, but the problem is still the same as it was years ago, in 2006.

Many see this bug as a problem: this bug has over 20 subscribers and two duplicates.

[daverpr (daverpr-bellsouth)]

Tried to get me name off hwdb-client list years ago with no luck. Sorry
have nothing to do with Ubuntu.

----- Original Message -----
From: "Tero Karvinen" <email address hidden>
To: <email address hidden>
Sent: Wednesday, May 13, 2009 9:21 AM
Subject: [Bug 30910] Re: Hardware Database not browseable

Dave: so how can I view the whole hardware database then?

We still don't know who owns the data. We still can't browse the data.
Names and URLs change, but the problem is still the same as it was years
ago, in 2006.

Many see this bug as a problem: this bug has over 20 subscribers and two
duplicates.

--
Hardware Database not browseable
https://bugs.launchpad.net/bugs/30910
You received this bug notification because you are subscribed to hwdb-
client in ubuntu.

Status in Checkbox System Testing: New
Status in “hwdb-client” source package in Ubuntu: Invalid
Status in “hwtest” source package in Ubuntu: Invalid

Bug description:
Cannot browse hardware database. Only very generic data without any
details is available on http://C.ubuntu.com/ .

Hardware database ids are sometimes refered to in Ubuntu bugs, but I
could not find any way to see the actual records. Collecting the data is
not very usefull if it is not accessible. Who owns the data now, are
they property of some company?

Hardware database related pages in Ubuntu wiki have not been modified
for a while
https://wiki.ubuntu.com/UbuntuDownUnder/BOFs/HardwareDatabaseRoadmap?highlight=%28HardwareDatabase%29
https://wiki.ubuntu.com/HardwareDatabase?highlight=%28HardwareDatabase%29

[David Murphy (schwuk)]

I'm marking this as invalid for Checkbox (since it is only responsible for collecting and sending the data, and it already provides local access to the data it collects), and marking it as affecting Launchpad Bugs which is where the new hardware database is being developed.

Regarding the old data (and URL), I've started making enquiries as to what we should do with this.

[Paul Sladen (sladen)]

Dave, thanks for that. IIRC, the whole dataset 2GB+ dataset was available if you knew the URL; so it should be possible to provide that again with a more stable (and publically published) URL, if for no reason more than historical interest.

[Tero Karvinen (karvinen+launchpad)]

Happy to hear that data collected by many volunteers is not lost.

So could we see the database then? What's the URL?

[Stuart Bishop (stub)]

I've opened Bug #377285 on obtaining the new hardware database submissions. This is a separate issue to being able to browse them

[Tommy Trussell (tommy-trussell)]

The database still isn't exactly browseable, but you can see a little information about what has been submitted. On the launchpad page for checkbox https://launchpad.net/checkbox there are some answers, including https://answers.launchpad.net/checkbox/+question/26749 which points out

> For now, the user interface simply provides a list of the information
> each user has submitted using the hwtest application in order to
> make it accessible to the community. Here is the URL where [username]
> should be replaced with your own username:
>
> https://launchpad.net/~[username]/+hwdb-submissions

SO you can look at the list of hwdb submissions and satisfy yourself that at least they haven't been thrown away all these years.

[Kip Warner (kip)]

As a game developer, knowing the typical hardware available of potential end users is critical. The hardware database would allow me a rough survey of the average minimal capabilities I could expect. But as others have mentioned, it is not really usable in its present state.

[karaluh (karaluh)]

The bug is now over seven years old. I know that nowadays Ubuntu hardware support is way better, but I still have many devices which work bad, or doesn't work at all.

[Kip Warner (kip)]

This bug is definitely getting on age, and as I mentioned in comment #33, it is very important to me.

[Robert Collins (lifeless)]

If it is so important to you, you might consider implementing a fix :)

[Kip Warner (kip)]

We can't implement a fix if we don't have access to the database. What use is a new and wonderfully written and exposed API if the backend it relies on is inaccessible?

[Robert Collins (lifeless)]

Your fix would run on the Canonical instance of the codebase, and have
access to the database.

[Kip Warner (kip)]

In an ideal world, that would suffice, but in reality you need to test it as you write it. Otherwise we could write device drivers for hardware we don't have.

[Jeff Lane  (bladernr)]

This provides a component level browsable interface with info on systems that are officially certified
http://www.ubuntu.com/certification/catalog/

And this, while not browseable on the component level, provides info on machines that have been tested by the community, including a listing of hardware contained in each one.
https://friendly.ubuntu.com/

At least does that help to some degree?

[Kip Warner (kip)]

Hey Jeff. It might help some people, but unfortunately not in my case. I don't need a hardware database of Ubuntu compatible hardware at present. What I need as an Ubuntu game developer is to see statistics on the most common hardware that is being used (e.g. GPU, CPU, etc.).

[Jeff Lane  (bladernr)]

Hrmmm... So two things...

First, you should get up with balloons and charlie_tca and others in the Ubuntu QA team:

https://wiki.ubuntu.com/QATeam

The first link I provided, while not in a necessarily perfect UI and not accessible via API can give you an idea of the most common items, if you do some digging... for example:
http://www.ubuntu.com/certification/catalog/component/pci/10de:0a6c/

Shows the nVidia GT218M certified on several Lattitude and Thinkpad models.

For a more broad view, however, as the Certification list only provides data on systems requested for certification from the OEM, the Ubuntu Friendly stuff will probably give a much better view of what "the real world" has Ubuntu running on.

The QA team was recently talking about overhauling the Friendly site, and what you need would be best, IMO, as a discussion with them. I could see, for example, adding a view to the Friendly site that displays GPU by number of systems containing it, or perhaps a list of the 10 most commonly submitted GPUs.

or other components like CPU, sound card, etc.

I do NOT know about access to the Ubuntu Friendly database, that's something balloons and others on that team could speak about though.

So, while I don't know of any way to get that data right now, at least that's a way you could get improvements made to the site(s) to give you that data down the road.

[Kip Warner (kip)]

Hey Jeff. Great response and thanks. As I mentioned, my main concern now, and I can't speak for everyone else with issues with the hw database, is not knowing what is the most common hardware used by Ubuntu users. I'm less concerned with whether this motherboard or that video card is certified to work with it.

If I had more time and wasn't working on this new Ubuntu game title full time, I would definitely take your recommendation and follow up with the suggested people.

[Tero Karvinen (karvinen+launchpad)]

Dear Jeff,

You just changed an 11 year old, confirmed bug to "Invalid". The bug also has two duplicates and many subscribers and comments, so it seems it's considered important by many.

Would you care to elaborate what reasons were behind this change?

Thank you.

[Adam Niedling (krychek)]

I guess it's no longer an issue that the hardware database is not browseable since there is no hardware database.

http://hwdb.ubuntu.com/ is not available as I can see it.

[Jeff Lane  (bladernr)]

@Tero

As Adam indicated, hwdb no longer exists and hasn't for some time. The bug can not and will not ever be resolved as none of the component parts (especially the projects the duplicates were filed against) exist any longer. They have either been abandoned, or transitioned to newer projects.

For completeness, anyone looking for an official list of hardware supported by Ubuntu, they should look to https://certification.ubuntu.com

[Tommy Trussell (tommy-trussell)]

I know folks are desperate to close this bug for good, but I'd like to point out I just tried the link I mentioned in comment #32 above

https://launchpad.net/~[username]/+hwdb-submissions

and I can STILL see the hwdb entries I contributed years ago. SO the data still seems to be around in some form.

The Checkbox project is still around, but I haven't used it lately, so I don't know if it still inserts data into launchpad / hwdb anymore.

[Chris Wayne (cwayne)]

This isn't a case of folks 'being desperate to close this bug for good', this is a case of 'this project is no longer viable or supported', and therefore it is correct to mark this bug invalid. Yes checkbox still exists, but we don't use hwdb, nor do we plan to resurrect it. For checking on hardware marked as working with Ubuntu, the canonical place to check is certification.ubuntu.com as Jeff pointed out in comment #47. We will be leaving this bug as invalid.

[Kip Warner (kip)]

I agree with the concerns of some of those above. If the hardware database is no longer maintained, closing the bug as a wishlist is less than ideal because it doesn't actually address the OP's original concern.

It would be better if there was a hardware database Canonical maintained. This can't be rocket science to build:

* Ubiquity installer makes some queries to the system with the user's permission.
* Installer makes some JSON POST requests to remote server.
* Remote server stores anonymous hardware description in a MySQL or PostgreSQL database.
* Database has a web based front end that shows some pies, tables, a field for queries, etc.

This can probably be put together in a few days.

[Jeff Lane  (bladernr)]

On Tue, Nov 28, 2017 at 9:54 AM, Kip Warner <email address hidden> wrote:
> I agree with the concerns of some of those above. If the hardware
> database is no longer maintained, closing the bug as a wishlist is less
> than ideal because it doesn't actually address the OP's original
> concern.

But it does in the only meaningful way it will be addressed. Lets
re-examine the original bug:

>> Cannot browse hardware database. Only very generic data without any details is available on http://hwdb.ubuntu.com/ .

This is resolved, maybe not satisfactorily, but resolved in that
hwdb.ubuntu.com was sunset several years ago, it is not coming back.

>> Hardware database ids are sometimes refered to in Ubuntu bugs, but I could not find any way to see the actual records. Collecting the data is not very usefull if it is
>> not accessible. Who owns the data now, are they property of some company?

Ubuntu bugs no longer refer to hardware database IDs as there is no
longer ANY method in Ubuntu so submit them, no endpoint on LP to
receive them, and no database front end to publicly display them. The
data submitted way back then remains tied to the user's LP account and
that is as far as that data will go.

If you would like to recover that data somehow, then a more
appropriate venue would be file a NEW, and more apropos Launchpad bug
that covers the need to recover that old hwdb data via some sort of
API mechanism.

>> Hardware database related pages in Ubuntu wiki have not been modified for a while
>> https://wiki.ubuntu.com/UbuntuDownUnder/BOFs/HardwareDatabaseRoadmap?highlight=%28HardwareDatabase%29
>> https://wiki.ubuntu.com/HardwareDatabase?highlight=%28HardwareDatabase%29

These will not be modified again, because that project is dead.

> It would be better if there was a hardware database Canonical
> maintained. This can't be rocket science to build:

There IS a hardware database Canonical maintains, and it has been
linked several times in the bug comments. That database is found at:

https://certification.ubuntu.com

And maintains a list of all the Officially Supported hardware from IoT
to Desktops/laptops to SoCs and Servers. There is also a catalog
available that shows you every single device that appears in every
system certified.

> * Ubiquity installer makes some queries to the system with the user's permission.
> * Installer makes some JSON POST requests to remote server.
> * Remote server stores anonymous hardware description in a MySQL or PostgreSQL database.
> * Database has a web based front end that shows some pies, tables, a field for queries, etc.
>
> This can probably be put together in a few days.

At this point, trying to resurrect a badly designed 11 year old
database of hardware no one owns or can purchase any longer is not the
right way to get this done. I would suggest going to the community
and getting some Ubuntu members together who are willing to develop,
build and host a new database with a better, cleaner design, using
better, more modern tools and encourage people to participate by
submitting data.

[Kip Warner (kip)]

On Tue, 2017-11-28 at 02:36 +0000, Jeff Lane wrote:
> There IS a hardware database Canonical maintains, and it has been
> linked several times in the bug comments. That database is found at:
>
> https://certification.ubuntu.com
>
> And maintains a list of all the Officially Supported hardware from
> IoT to Desktops/laptops to SoCs and Servers. There is also a catalog
> available that shows you every single device that appears in every
> system certified.

This isn't really what they're getting at, or maybe I am only speaking
for myself. That hardware database isn't a collection of hardware
statistics of Ubuntu users which is useful for developers making design
choices.

> At this point, trying to resurrect a badly designed 11 year old
> database of hardware no one owns or can purchase any longer is not
> the right way to get this done. I would suggest going to the
> community and getting some Ubuntu members together who are willing to
> develop, build and host a new database with a better, cleaner design,
> using better, more modern tools and encourage people to participate
> by submitting data.

I didn't suggest resurrecting the old one. I suggested how to go about
building a new one.

--
Kip Warner | Senior Software Engineer
OpenPGP signed/encrypted mail preferred
http://www.thevertigo.com