Review Request: fedora-app-install - Fedora application data

[Jeff Johnson]

tracker

[Richard (richard-redhat-bugs)]

Spec URL: http://people.freedesktop.org/~hughsient/temp/fedora-app-install.spec
SRPM URL: http://people.freedesktop.org/~hughsient/temp/fedora-app-install-20090306-1.src.rpm

Description: Fedora application data such as icons and translations for applications not yet installed. This package is designed to be updated every few months as new applications are added and translations are updated.

This only includes data from the fedora rawhide repository, as other repositories will be packaged and merged using the app-install framework.

This data will be used in application browsers such as gnome-app-install and
gpk-app-install in future versions of gnome-packagekit.

See http://blogs.gnome.org/hughsie/2009/03/05/application-installing/ and http://blogs.gnome.org/hughsie/2009/03/06/application-installing-ii/ for background.

rpmlint fedora-app-install-20090306-1.src.rpm fedora-app-install-20090306-1.noarch.rpm
fedora-app-install.noarch: W: no-documentation

I'm not quite sure what licence to use in this spec file, as the icons in the packages will be marked with different licences, and I'm not sure if you can mark an icon (the binary) or a translation as GPLv2+. Advice welcome.

This review depends on the review for app-install, https://bugzilla.redhat.com/show_bug.cgi?id=488962

[Richard (richard-redhat-bugs)]

For reference, I've got packages here for rpmfusion-free-app-install and rpmfusion-nonfree-app-install also, but I want to get the fedora one reviewed first.

[James (james-redhat-bugs)]

This should not be approved for Fedora. We have a mechanism for shipping repository metadata, and bundling it in packages is not it.

[seth (seth-redhat-bugs)]

+1 to Comment #2. We've shipped package metadata in packages in the past. Comps and specspo come to mind. They were both abject failures at being kept current and maintained. We're much better off doing this repo-side only.

[Richard (richard-redhat-bugs)]

(In reply to comment #2)
> This should not be approved for Fedora. We have a mechanism for shipping
> repository metadata, and bundling it in packages is not it.

This isn't repository meta data nor is it package metadata - this is application data. This is nothing like comps or specspo at all. This is also for functionality (the client side database) that is shared between distributions, and other distributions don't share our repomd or the same packaging tools.

Please guys, don't just protest against this because it's not doing things the yum way. Putting icons and all translations in the yum repomd is not suitable unless you want to download large amounts of extra metadata every time a few packages change, and would add a large chunks of functionality to yum. I've got no intention of adding huge amounts of code to yum, as it already difficult to interface with yum using PackageKit and I really don't think yum should understand the concept of applications, rather than packages. yum is meant to be a package manager, not an application manager.

I've been updating hal-info quite a bit in the last few years, and this has worked well for this sort of data. We cannot do this repo-side and use the mechanism in a cross-distro way. I've talked in depth with suse, ubuntu and foresight developers about this, and using yum in this way is not the right thing to do.

Richard.

[James (james-redhat-bugs)]

> This isn't repository meta data nor is it package metadata

 How is it not package metadata ... the data comes _directly_ from the packages in the repo.

> Please guys, don't just protest against this because it's not doing things
> the yum way.

 This isn't "the yum way" it's "the Fedora way" and "the RHEL way" and "the CentOS way" ... each has a process for getting repo. MD out to clients, and putting it in packages isn't it.
 Obviously for Fedora you can just ask FESCO to say that stuff random bits of package data inside a single package is fine ... at which point it doesn't matter if upstream yum developers like it or not, that'll be the new Fedora way.

> Putting icons and all translations in the yum repomd is not suitable
> unless you want to download large amounts of extra metadata every time a few
> packages change

 Translations for all .desktop files is less than primar or filelists, icons may be more and so you may want to find a different method than just putting everything in one file and linking it to the repomd.xml.
 Install from CD MD will never change, so only new updates in the updates repo. will generate new MD there ... as against the "dump package data in another package" way which will have to update everything, anytime one package hits updates with this data changed. The other option being to intentionally have the package metadata by out of sync. ... and I hope that you aren't planning on that as an option, because that's a huge failure.

> I've got no intention of adding huge amounts of code to yum

 Indeed, and I would say that's the biggest problem I have with PK ... not that it's that relevant here, as the infrastructure you _need_ in yum is 0 lines of code ... the sane amount to change would be a little more.
 This is all about how you distribute things in Fedora/RHEL/whatever ... so you _need_ to change something around mash time, I think ... but I'm not 100% sure, speak to Jesse etc.

> yum is meant to be a package manager, not an application manager.

 I don't see why the distinction applies to yum and not PK. But again this isn't a yum issue, I would have the same objections for the same reasons if we shipped pkcon as the cmd line interface.

> I've been updating hal-info quite a bit in the last few years, and this has
> worked well for this sort of data.

 hal-info is not generated directly from package data, so it is not analogous.

[Jesse (jesse-redhat-bugs)]

We already have one package attempting to gather things out of packages, specspo. It's horribly behind, often wrong, and generally useless. I'd rather not see Fedora repeat this mistake.

If you want to be able to get information about available packages, we have repodata for that reason. If you think content will be too big, generate a file for each package that is "extra info". Clients can fetch just the extra info they care about. It'll require some engineering on the repodata creation side, and a lot of work to make sure it doesn't slow composes down, but that's the right place to put it, right along side all the other data about available packages.

I will not support this package.

[Bill (bill-redhat-bugs)]

Feh. Someone asked me for my opinion. Well, you're all right, and you're all wrong. :P

I agree with Richard - there's value in having this be a distribution neutral standard, and you're not going to get there by just creating another metadata file in createrepo. Moreover, referring to 'comps' and 'specspo' in the past is a little odd - comps-extras still exists, as does the specspo package. Furthermore, given how the data is defined, there's no good way to have it in the package metadata and have it update with sane, low bandwidth, semantics.

However, I also agree with Jesse - having this in package format this way isn't good either. Building this out of band against the repository is a big old hack, and doesn't scale or translate well. You want to automate these sorts of tasks so they're always up to date. Moreover, having a new package each time doesn't gain you any sorts of caching benefits.

What you really want is an incremental protocol that can deliver the new icons & entries w/translations since the last time you updated. Neither a package, nor the yum metadata, fits this well. Honestly, an online service with local caching seems more appropriate.

(As a side point, legally, you'd have to have the license be "A & B & C & D.."... where that's the license of all the icons in total. And don't forget trademarks. This gets messy fast.)

[Jeremy (jeremy-redhat-bugs)]

(In reply to comment #4)
> (In reply to comment #2)
> > This should not be approved for Fedora. We have a mechanism for shipping
> > repository metadata, and bundling it in packages is not it.
>
> This isn't repository meta data nor is it package metadata - this is
> application data. This is nothing like comps or specspo at all. This is also
> for functionality (the client side database) that is shared between
> distributions, and other distributions don't share our repomd or the same
> packaging tools.

Richard -- how is the functionality shared by distributions? If I've followed what you've posted on your blog, then each repository has to have its own metadata (errr, "database") defining what applications are available within it. This then is told to the app-installer by registering it and saying "the database for this repo is over here". But if that's the case, why wouldn't it make sense to have the database registered by the repo itself? The main reason I can think of is that we're actually better off in allowing additional arbitrary metadata via yum than other package managers which are very hard-coded and can't do this sort of on-the-fly additional metadata :-)

> Please guys, don't just protest against this because it's not doing things the
> yum way. Putting icons and all translations in the yum repomd is not suitable
> unless you want to download large amounts of extra metadata every time a few
> packages change, and would add a large chunks of functionality to yum. I've got
> no intention of adding huge amounts of code to yum, as it already difficult to
> interface with yum using PackageKit and I really don't think yum should
> understand the concept of applications, rather than packages. yum is meant to
> be a package manager, not an application manager.

*grin* Isn't PackageKit a package manager by its very name? ;-)

Really, the line between "application" and "package" is as much a historical accident as anything. They aren't one to one mappings, sucks... but so it goes. It gives us some nice things too. I fully agree that people in general want to be installing something to actually do something (an application) vs some arbitrary package/set of packages. But that doesn't mean that we have to go to a whole new level of abstraction and metadata generation/distribution to get there.

> I've been updating hal-info quite a bit in the last few years, and this has
> worked well for this sort of data. We cannot do this repo-side and use the
> mechanism in a cross-distro way. I've talked in depth with suse, ubuntu and
> foresight developers about this, and using yum in this way is not the right
> thing to do.

As someone else mentioned, the difference is that hal-info isn't generated out of information about packages being shipped and rather is generated based on hardware. So it's more like hwdata than something like comps or specspo. Also, it seems somewhat a shame here that you've talked in depth with developers with every distribution *other* than Fedora :(

[Jeremy (jeremy-redhat-bugs)]

(In reply to comment #7)
> What you really want is an incremental protocol that can deliver the new icons
> & entries w/translations since the last time you updated. Neither a package,
> nor the yum metadata, fits this well. Honestly, an online service with local
> caching seems more appropriate.

In addition, you probably want to be smart about how you grab things (there's no need to grab the icons for everything at once, for example, you can be intelligent about only grabbing what you can see on the screen plus the next N for some value of N -- if someone scrolls way down in the list and has to wait for a few icons to load, as long as they lazily do so, that's okay!)

[Richard (richard-redhat-bugs)]

(In reply to comment #6)
> I will not support this package.

Does this mean this package review will be blocked? What about packages like smart and apt-rpm that do things differently to yum and the core distro? I'm not arguing this package should be installed by default, just be available to install.

[Richard (richard-redhat-bugs)]

(In reply to comment #7)
> What you really want is an incremental protocol that can deliver the new icons
> & entries w/translations since the last time you updated. Neither a package,
> nor the yum metadata, fits this well. Honestly, an online service with local
> caching seems more appropriate.

I prototyped this, but it wouldn't have scaled well, and there were inconsolable differences with debian, e.g versions, icons and translations for iceweasel.

> (As a side point, legally, you'd have to have the license be "A & B & C &
> D.."... where that's the license of all the icons in total. And don't forget
> trademarks. This gets messy fast.)

I'll get the generate script to generate the licence string and update the spec. Thanks.

[Richard (richard-redhat-bugs)]

(In reply to comment #9)
> In addition, you probably want to be smart about how you grab things (there's
> no need to grab the icons for everything at once, for example, you can be
> intelligent about only grabbing what you can see on the screen plus the next N
> for some value of N -- if someone scrolls way down in the list and has to wait
> for a few icons to load, as long as they lazily do so, that's okay!)

Sure, but we search in the locale, and hence we need that data upfront. Having icons download once per-user also seems painful (as the tools run as the user session, not root).

[James (james-redhat-bugs)]

> Does this mean this package review will be blocked?

 I certainly hope so, it would be a significant mistake to ship this.

> What about packages like smart and apt-rpm that do things differently to yum
> and the core distro?

 Noone is saying you can't ship PK. I'm pretty sure noone cares about the review for the tools that generate/consume the metadata in this package.
 But if smart decided that the md5sums in our repomd.xml was hard to deal with, so they'd just ship a smart-pkg-data ... then, yeh, I'd say that was a bad idea and would try and block it.

> Sure, but we search in the locale, and hence we need that data upfront.

 Again, the textual part of the data is _tiny_ ... even more so for updates (the bit that changes). And I'd be surprised if it changed anywhere near as often as primary/etc.
 The fact you have tied icons with the textual data in this implementation doesn't mean you have to continue to do that in another one.

[James (james-redhat-bugs)]

 To respond to Bill:

> I agree with Richard - there's value in having this be a distribution neutral
> standard,

 I somewhat agree, to the extent that PK can move forward it'll be to "standardize" higher level things that what it does currently. And I do think that's a much brighter future than what we have currently.

> and you're not going to get there by just creating another metadata
> file in createrepo.

 However this I disagree with, to have "pkg install" and "pkg remove" in PK it didn't mandate how the backend functioned and I see no reason that Ubuntu can't drop metadata packages if they are insane enough to while we just use repo metadata.
 The "std." is in that everyone calls PK_lookup_application_data() or whatever ... not that each distribution has to merge to the same lowest common denominator.
 Yes, it's more work ... and it sucks that PK is writing everything 4 times, but then I've always said that.

> Moreover, referring to 'comps' and 'specspo' in the past is
> a little odd - comps-extras still exists, as does the specspo package.

 Indeed, but the state of specspo is a reason to reject this package not allow it through. No doubt specspo went live much faster due to it's implementation and I'm sure it will be more work for PK to have this feature implemented properly ... but look at the historical data on specspo.
 Yes it's still used as initially released, but it's still _barely_ working. "LANG=de_DE.utf-8 yum info kernel" has a translated description, but an en_US summary. "yum search 'Ein Betrachter und Editor'" still finds nothing.

[Matthias (matthias-redhat-bugs)]

The comparison with specspo is misleading. The translations in specspo are something that needs to be maintained separately, and that is where specspo failed.

The application metadata in this package is taken from desktop files where it is successfully maintained and updated.

[Richard (richard-redhat-bugs)]

(In reply to comment #14)
> Indeed, but the state of specspo is a reason to reject this package not allow
> it through.

I don't understand why you keep referring to specspo. It's separate data about packages that doesn't really have a way of being tied to upstream. If the spec files were translated in CVS, and then specspo extracted the translations from the spec files (which are actively translated) and packaged it separately, then I would agree. But it's a separate project with no infrastructure tie in.

[Richard (richard-redhat-bugs)]

(In reply to comment #14)
> Yes, it's more work ... and it sucks that PK is writing everything 4 times,
> but then I've always said that.

I agree we're rewriting some parts of what yum used to do, but we're doing it for a good reason. I do think it's important to work with other distros and look at the big picture rather than look at just what's possible to do with Fedora.

PackageKit isn't just a front end to yum, and it's never going to be. I really don't think what your disagreements with PackageKit have to do with this merge review. Perhaps you should send mail about that.

This data will be used by gnome-app-install also, and that's got nothing to do with the PackageKit project at all.

[Richard (richard-redhat-bugs)]

(In reply to comment #11)
> I'll get the generate script to generate the licence string and update the
> spec. Thanks.

This is the license:

GPLv2+ and (LGPLv2+ and BSD) and EPL and LGPLv2 and (LGPLv2 with exceptions) and MIT and (GPLv2+ and LGPLv2+) and OFL and LGPLv2+ and (ASL 1.1) and (ASL 2.0) and GPLv2 and BSD and (Artistic 2.0 and GPLv2 and GPLv2+ and LGPLv2+ and LPPL and MIT and Public Domain and UCD and Utopia) and (GPLv2 and LGPLv2+) and (GPL+ or Artistic) and (MIT and GPL+ and TCL) and GPLv3 and (GPLv2+ and MIT) and (ASL 2.0 and MIT and BSD and CC-BY and LGPLv2+ and (AFL or LGPLv2+)) and GPLv3+ and (GPL+ with exceptions) and (LGPLv3 and LGPLv2+ and MPLv1.1 and BSD) and (GPLv2+ or OSL 2.1) and GPL+ and (OSL 2.0) and (QPL or GPLv2 or GPLv3) and (GPLv2+ with exceptions) and LGPLv3+ and CPL and WTFPL and (BSD and GPLv2+) and (SCRIP License) and (ASL 2.0 and MIT and BSD) and (BSD and MIT) and (Public Domain and MIT) and LPPL and (FTL or GPLv2+) and (GPLv2 with exceptions and BSD and CPL and Public Domain) and GPL and zlib and (GPLv2 with exceptions) and (GPLv2+ or Ruby) and (ASL 2.0 and W3C and Public Domain) and (MPLv1.1 or GPLv2+ or LGPLv2+) and (GPLv2 and LGPLv2) and (MIT or LGPLv2+ or BSD) and (GPLv2+ and GFDL) and GFDL and (Artistic 2.0) and (Public Domain) and (GPL+ and LGPLv2+) and (LGPLv2+ and GPLv2+ and GFDL) and libtiff and (QPL and (LGPLv2+ with exceptions)) and PHP and (GPLv2+ and GPLv2 and MIT) and ((CPL or GPLv2+ or LGPLv2+) and ASL 1.1 and MIT and Ruby) and OpenLDAP and (GPLv2 and Redistributable, no modification permitted) and (LGPLv2+ with exceptions and BSD) and (BSD and LGPLv2+ and (BSD or LGPLv2)) and ISC and (MIT with advertising) and (BSD and ImageMagick) and Artistic and (MIT and Lucida and Public Domain) and ERPL and (GPLv2+ and LGPLv2+ and MIT) and (LGPLv2+ or ASL 2.0) and (GPLv3+ and GPLv2+ with exceptions) and MPLv1.0 and CeCILL and MPLv1.1 and OpenPBS and OpenSSL and (ASL 2.0 and BSD) and Ruby and (LGPLv2 or Artistic clarified) and (Freely redistributable without restriction) and (GPLv2 and zlib) and (LGPLv2+ and CC-BY-SA) and Vim and AGPLv1 and (MIT and LGPLv2) and (MIT and GPLv2+) and (Redistributable, no modification permitted) and (Open Publication) and (CC-BY and Free Art and GPL+) and (LGPLv2+ and (BSD)) and (BSD and (GPLv2+ or Artistic or LGPLv2+) and LGPLv2) and (GPL+ and GFDL and CC-BY-SA and Public Domain) and (GPL+ or LGPLv2+ or MIT or MPLv1.1) and NetCDF and (LGPLv2 or MPLv1.1) and CC-BY-SA and CC-BY and (GPLv2+ and LGPLv2+ with exceptions) and (BSD with advertising) and (GPLv2+ or LGPLv2+) and (GPLv2 and GPLv3+ and MIT) and (Lucida and MIT and Public Domain) and (LGPLv2+ and GPLv2+) and (EPL and GPLv2 and LGPLv2) and (LGPLv2 and (LGPLv2+ or CPL)) and (GPLv2 with exceptions or CDDL) and (TMate License and ASL 1.1) and IJG and (Bitstream Vera and Public Domain) and (GPL+ or MPLv1.0) and IBM and TCL and (GPLv2+ or AFL) and ((GPLv2 or GPLv3) and GPLv2+ and LGPLv2+ and LGPLv2) and (BSD and LGPLv2+ and GPLv2+ and Public Domain) and (LGPLv2+ and LGPLv2+ with exceptions and GPLv2+) and (GPL+ and BSD and MIT and Public Domain) and (GPLv2+ or Artistic 2.0) and (Artistic clarified) and (ASL 1.1 and ASL 2....

[Richard (richard-redhat-bugs)]

Tell a lie, that was a script error. The actual licence is much shorter:

GPLv2+ and BSD and GPLv2 and GPLv3+ and GPL+ and (QPL or GPLv2 or GPLv3) and (Public Domain) and LGPLv2+ and (GPLv2+ and GFDL) and MIT and (GPLv2+ and LGPLv2+ and MIT) and (GPL+ or Artistic) and LGPLv2 and MPLv1.1 and GPLv3 and (GPLv2 with exceptions) and (GPLv2+ and LGPLv2+) and (ASL 2.0) and (GPLv2+ and LGPLv2+ and GFDL) and (GPLv2+ and Python) and (LGPLv3 and LGPLv2+ and MPLv1.1 and BSD) and (GPLv2+ and MIT) and (GPLv2 with exceptions or CDDL) and NGPL and BitTorrent and (Crystal Stacker) and (LGPLv2+ and GPLv2+) and GFDL and zlib and (Freely redistributable without restriction) and CeCILL and (GPLv2+ or Artistic) and (SPL or LGPLv2+) and (GPLv2+ and CC-BY-SA) and (MIT and GPLv2) and Teeworlds and (GPLv2 and GPLv3+) and QPL and (GPLv2+ with exceptions) and (GPLv2+ and GPLv2) and (GPL+ and GPLv2 and GPLv2+ and LGPLv2+) and (GPLv2 or GPLv3) and (GPLv2+ and GPLv2 and (GPLv2+ or MIT)) and (Redistributable, no modification permitted) and Vim and (GPLv2 and GPLv2+) and (BSD and GPLv2+ and Python) and (Open Publication) and (GPLv2 and BSD) and (GPLv2+ and Redistributable, no modification permitted) and WTFPL and (GPLv3 and Public Domain) and (GPL+ and LGPLv2+) and (GPLv2+ and MPLv1.1 and LGPLv2+ with exceptions) and (MPLv1.1 or GPLv2+ or LGPLv2+) and (GPLv2+ and GPLv2 and MIT) and (LGPLv2+ with exceptions) and (GPLv2+ and GFDL and MIT) and (GPLv2 with exceptions and BSD and CPL and Public Domain) and EPL and (MIT and BSD and ZPLv2.0 and Bitstream Vera and OFL) and (GPLv3+ and MIT) and (GPLv2+ and CC-BY and CC-BY-SA) and (GPLv2+ and Adobe) and OpenPBS and (GPLv2+ with exceptions and GFDL) and (GPLv2+ and LGPLv2+ and CPL and MIT) and (BSD and BSD with advertising and LGPLv2+ and GPLv2+) and IBM and (LGPLv2 with exceptions or GPLv3 with exceptions) and (zlib and GPLv2+) and (CC-BY and CC-BY-SA and Public Domain) and TCL and (GPLv2+ and Free Art) and (ASL 2.0 and MIT and BSD and CC-BY and LGPLv2+ and (AFL or LGPLv2+)) and (ASL 2.0 and SPL) and (GPLv2+ and LGPLv3+) and (MIT with advertising) and (LGPLv3 and CC-BY and CC-BY-SA and Public Domain) and (ASL 2.0 and MIT) and (GPLv2+ and Freely redistributable without restriction)

[Bill (bill-redhat-bugs)]

Realistically, the license of the package would be the union of the licenses of the *icons* in the dependent packages, which is different than the union of the licenses of the packages themselves (and hopefully shorter). But that can't be sanely automated.

(The fact that the license of the package ends up being this ludicrous should give some idea that this isn't really the best way to go about creating and deploying this...)

(In reply to comment #11)
> I prototyped this, but it wouldn't have scaled well, and there were
> inconsolable differences with debian, e.g versions, icons and translations for
> iceweasel.

How so?

The problem is, essentially, that you want this delivered and updated in an incrementable format, as the data's going to be (generally) only additive, and in small chunks. Doing that as packages is pretty wasteful.

Ideally, you'd have a server-side program that does this for you - you ask it for the differences between what you have, and what is the latest, and it sends it to you. But since all we have is yum, which is defined to not have any server components aside from raw transfer of files, any sort of metadata that wants to be deployed gets shoehorned into either a static metadata chunk, or a package. Neither of which is really appropriate here.

[seth (seth-redhat-bugs)]

Bill,
 There's another problem here. We cannot depend on specific infrastructure living in some location for some good reasons:

1. it scales for crap
2. it won't work for people wanting to respin/derive work from fedora
3. did I mention it scales for crap?

[Richard (richard-redhat-bugs)]

(In reply to comment #21)
> 1. it scales for crap

Totally agreed. We would need some serious bandwidth and hardware for 1000s of concurrent users. The mirrors wouldn't also like thousands of tiny 40Kb files.

[seth (seth-redhat-bugs)]

Mirrors already have thousands of small files. Look at the size of most rpms.
If push came to shove we could:
1. generate sets of app/icon metadata for each package
2. generate sets of app/icon metadata for each package of each group from comps (so you get all the apps from that group at once)
3. Do both.

We're talking about 80-150MB in roughly 2000 files, iirc. The mirrors can soak that up. And having an index file you grab from the repodata is just like things we already have.

[Alexey (alexey-redhat-bugs)]

I like the idea keeping app-install database and icons not in package but as a kind of "extra info" that will could have benefits of caching and incremental updating (comment #6 and comment #7). I'm interested in implementing it as GSoC project. The exact way of generating and storing metadata should perhaps be first discussed with infrastructure people.

[Richard (richard-redhat-bugs)]

(In reply to comment #23)
> We're talking about 80-150MB in roughly 2000 files, iirc. The mirrors can soak
> that up. And having an index file you grab from the repodata is just like
> things we already have.

So, the user launches add/remove software, and searches for "office".

1. The desktop metadata gets downloaded (few Mb)
2. The results get shown with icon-missing (14)
3. PackageKit instructs yum to download icon data for 14 packages
4. The icons get downloaded by yum
5. Add / remove software updates the icons with the new themed icons

Now, compare that to the Ubuntu add/remove experience:

1. The results are shown with the correct icons, near instantly

Now we need the desktop metadata in one file so we can perform searching on the file (like searching for 'office' in Hungarian) and because we want to get results instantaneously. I would argue we need the icons included in the metadata file as we want to show the icons with the search results as they appear.

The fact that Suse and Ubuntu want to share a common spec on this really makes integrating it so deeply with the Fedora repo metadata and yum core a bitter pill to swallow.

[Jesse (jesse-redhat-bugs)]

There is a terrible answer here, which is that you make users of PackageKit swallow a scheduled job that keeps all metadata fresh. Every few hours it just pulls down every single bit of metadata out there (think apt-get --update). That way whenever you go to use PackageKit, 9 times out of 10 you have the latest metadata and there is no need to go download anything new. And if there is a repo that is out of date (and we already have ways of discovering this very quickly/easily) the amount of new stuff to download will be quite small as compared to downloading for every repo.

[seth (seth-redhat-bugs)]

Putting a package in the distro that will be a giant bag of icons and translations that will need to updated whenever a pkg changes or gets added that adds/removes/changes that set is a bitter pill to swallow, too.

It's the WRONG way to do things, furthermore and unlike Suse and Ubuntu we have evidence of it being a bad idea in the form of two pkgs:

comps and specspo - both of which used to be packages trundled along in fedora/rhl/rhel.

I was talking to James about this problem and generating the metadata at createrepo time isn't terribly difficult. And the users benefit b/c instead of downloading a package containing all this content each time it is updated they can just download the fedora-updates metadata for this content.

Making this information be per-repo means that 3rd party and private repos can take advantage of it, too.

So, you want this metadata available to yum and PK, great, we can do that - but the info must live in the repository metadata - not in some random pkg in the distro.

[Rakesh (rakesh-redhat-bugs)]

*** Bug 488962 has been marked as a duplicate of this bug. ***

[Matthias (matthias-redhat-bugs)]

> comps and specspo - both of which used to be packages trundled along in
> fedora/rhl/rhel.

You keep brining specspo into this, without addressing earlier comments on why specspo is a different situation altogether. Not a great way to have a discussion.

[James (james-redhat-bugs)]

> You keep brining specspo into this, without addressing earlier comments on why
> specspo is a different situation altogether

How is it different?

specspo:

. Metadata generated from packages.
. Have to download everything at once.
. Package updates change some of the metadata, often enough, thus. OOD.
. Problems with multiple repos. (specspo only has one package).
. Major problems with turning repos. on/off.

"this BZ":

. Metadata generated from packages.
. Have to download everything at once.
. Package updates change some of the metadata, often enough, thus. OOD.
. Major problems with multiple repos. (specspo only has one package).
. Major problems with turning repos. on/off.

[Matthias (matthias-redhat-bugs)]

You got it wrong

specspo:
 - metadata that has to be maintained separately

appdata:
 - metadata that is automatically derived from packages

[seth (seth-redhat-bugs)]

specspo _was_ pulled from the translated fields in the pkg spec files.

it wasn't maintained separately.

[Matthias (matthias-redhat-bugs)]

And who put the translations there, the package maintainer ? I think not.
Very much looks like separately maintained to me, even if it is in the same file.

[seth (seth-redhat-bugs)]

I'm confused - what does that have to do with anything?

The point is - running another program to generate data then stuffing it into an rpm which we then ship out is a bad plan. Just as it was with specspo. Putting this data into the repodata is a better plan.
;

[James (james-redhat-bugs)]

> You got it wrong
>
> specspo:
> - metadata that has to be maintained separately
>
> appdata:
> - metadata that is automatically derived from packages

 Appdata wants rantings etc. ... which is outside data.

 But to expand on what Seth said, even if we assume the above is a difference and that there are other differences (like appdata want non-text for example) ... those differences don't matter from the point of view of distributing via. packages or repodata.

 On the other side the fact that the data the user needs will change based on which repos. are enabled and change as the packages change in their enabled repos. _in both cases_ is relevant.

[Richard (richard-redhat-bugs)]

(In reply to comment #34)
> The point is - running another program to generate data then stuffing it into
> an rpm which we then ship out is a bad plan. Just as it was with specspo.
> Putting this data into the repodata is a better plan.

This is the process in generating the data:

* explode every rpm containing a desktop file (including selected deps where appropriate)
* extract lots of data from the desktop file
* copy the icons from the exploded root, converting and *resizing* where required
* tarring up the icon tree
* pushing the translations and desktop metadata into a sqlite database

This takes 4 hours on my laptop to do for fedora, or 30 minutes for rpmfusion. This is not something we want to do every day, for data that probably should only be updated once or twice a release (Ubuntu .

So, looking at positives and negatives, for putting it in a package:

Package:
 * Allows us to ship the data on the media, so the user doesn't have to "download metadata" every time they open the application tool
 * Allows us to query the database directly, rather than convincing PackageKit [to download us the single chunk of metadata and then import the metadata into the system database, as root]
 * Means we share the *same tools* as Ubuntu, Suse and Foresight.
 * Means we don't have to patch KPackageKit which already depends on the system database.
 * Means we can merge rating stats from other external sources before we ship the data, e.g. the gnome3 application usage stats
 * It already exists, and someone is willing to maintain the package

Metadata:
 * Allows us to ignore the fact that there are a few different licenses
 * Means we have to dedicate 4 hours per compose (possibly quicker if you have the rpms locally, but as we've discussed, the builders don't)
 * We have data changing daily that differs by only a few superficial entries
 * Means we can't use external data sources, for instance application usage stats
 * The app-install hooks in yum or PackageKit do not exist, and no-one is willing to write code for them. I've heard lots and lots of stop energy from the yum guys, but nobody has offered to write any code.

Also bear in mind: Ubuntu has been shipping app-install-data FOR FIVE YEARS. It works for them as a package. Why do you think they shipped it as a package back then? And still now? They have a very similar infrastructure to us.

The biggest point, and the most important from a user experience point of view is that we need to query the application database directly. If we make every query go through PackageKit, then through yum then we've killed our performance. Instead of getting responses back in the required 25ms for search-as-you-type we're talking about latencies of hundreds of milliseconds, or if yum has to check and download new metadata, *seconds*. This sucks and is simply not good enough.

[Matthias (matthias-redhat-bugs)]

We should probably take this to fesco.

'I don't like it, therefore I'll block the package review' is not the right way to decide this.

[seth (seth-redhat-bugs)]

>
> This is the process in generating the data:
>
> * explode every rpm containing a desktop file (including selected deps where
> appropriate)
> * extract lots of data from the desktop file
> * copy the icons from the exploded root, converting and *resizing* where
> required
> * tarring up the icon tree
> * pushing the translations and desktop metadata into a sqlite database
>
> This takes 4 hours on my laptop to do for fedora, or 30 minutes for rpmfusion.
> This is not something we want to do every day, for data that probably should
> only be updated once or twice a release

1. the above is only true if you run the generator everytime and you don't cache anything. Outrageously naive, I think.

2. I'm going to talk to Florian about his implementation of the same - I seem to recall it taking substantially shorter amount of tim.

3. There is no requirement that we have to do it everytime the repo is changed. Only if it is changed substantively.

> So, looking at positives and negatives, for putting it in a package:
>
> Package:
> * Allows us to ship the data on the media, so the user doesn't have to
> "download metadata" every time they open the application tool

Yah - they'll just have stale data.

> * Allows us to query the database directly, rather than convincing PackageKit
> [to download us the single chunk of metadata and then import the metadata into
> the system database, as root]

We don't have to import that metadata as root, now. Yum hasn't had to do that for quite some time. Moreover it's just a sqlite db.

> * Means we share the *same tools* as Ubuntu, Suse and Foresight.

this definitely falls into the IBIWISI category considering we do not seem to be sharing ANY tools with any of these distros to speak of.

> Metadata:
> * Allows us to ignore the fact that there are a few different licenses

A few? Really?

> * Means we have to dedicate 4 hours per compose (possibly quicker if you have
> the rpms locally, but as we've discussed, the builders don't)

And you don't need to do this at all if you extract the data from the pkgdb.

> * We have data changing daily that differs by only a few superficial entries

And this is why you cache entries and update the db.

> * Means we can't use external data sources, for instance application usage
> stats

We can use whatever data sources we want, in fact.

> * The app-install hooks in yum or PackageKit do not exist, and no-one is
> willing to write code for them. I've heard lots and lots of stop energy from
> the yum guys, but nobody has offered to write any code.

I think what you've heard from us is years of experience cautioning you from chasing down a blind alley.

[Tom (tom-redhat-bugs)]

IMHO, this sort of disagreement is what FESCo is intended to resolve.

[Richard (richard-redhat-bugs)]

(In reply to comment #39)
> IMHO, this sort of disagreement is what FESCo is intended to resolve.

Yes, this would be good for FESCo to discuss. Seth (and the other yum guys) do not agree with the implementation, but as far as I am aware, this isn't reason enough to fail a review request.

[seth (seth-redhat-bugs)]

the whole argument here is that there should not need to be a review request b/c the pkg in question

1. should not BE a package at all
2. should not be in the distro, AT ALL.

So - by saying you think it shouldn't fail a review request you are, inherently begging the question.

[Bill (bill-redhat-bugs)]

However, given that it's not obviously afoul of the packaging guidelines, this would appear to be a FESCo concern. (We ship many many things I think are bad ideas.)

[seth (seth-redhat-bugs)]

I've got no problem with it going to fesco - I just wanted to make sure it was clear that whether or not this goes in as a package in the distro is the point of the argument and not a side issue.

[Richard (richard-redhat-bugs)]

Also, I think it makes a lot of sense to whitelist or blacklist desktop files -- Ubuntu ask maintainers to submit applications manually so that the list is small and high quality, and that we can ensure apps are correctly categorized, and have things like reviews and screenshots. I don't think putting every application (package with desktop files) shipped in Fedora into an application installer is a good idea.

i.e. you can't just run a script against pkgdb without some sort of manual control and merging different 'wads' of data. We probably might want different data for each spin. I think starting with the distro exploded tree is a good start, but the KDE spin is going to need different "ratings" on each application compared to the GNOME spin. And maybe a different definition of "application" is required for the server spin; that's fine with the app-install schema I'm suggesting.

Also, this code exists now. It's ready to go. We've got upstream support from kpackagekit. We've got downstream support from the other distros that are watching us and certainly Ubuntu and Suse are willing to follow suit.

This is probably very relevant to FESCo.

[Tom (tom-redhat-bugs)]

The absurdity of the license string and the nature of this package having an aggressively changing license with every new release makes it functionally impossible to audit.

While an argument could be made that the .desktop files are not copyrightable, the icons clearly are.

I'm blocking this against FE-Legal. I would strongly encourage you to work to find a way to not need to bundle all of these icons within a single package.

[Richard (richard-redhat-bugs)]

(In reply to comment #45)
> I would strongly encourage you to work to find a way to not need to bundle
> all of these icons within a single package.

Such as...?

Richard.

[Tom (tom-redhat-bugs)]

Maybe they could live in the pkgdb? Maybe there is a way to accomplish your goal without using the icons?

[seth (seth-redhat-bugs)]

Spot:
 Let's say we stuff the icons into a sqlite db in the pkgdb. And then we get that downloaded as metadata, do we need to have a special license for that since it is just a compilation and not a derivative work?

 Option two: only show icons of the apps the user has installed - since the icons would be on the local disk already.

[Richard (richard-redhat-bugs)]

(In reply to comment #48)
> Spot:
> Let's say we stuff the icons into a sqlite db in the pkgdb. And then we get
> that downloaded as metadata, do we need to have a special license for that
> since it is just a compilation and not a derivative work?

I don't see how collecting icons and pushing them into a tar.gz archive is any different to pushing them into a sqlite db, from a legal point of view.

> Option two: only show icons of the apps the user has installed - since the
> icons would be on the local disk already.

No, we want application icons in the application browser.

Richard.

[Tom (tom-redhat-bugs)]

I think shoving the icons into the repodata and distributing them in a single bundle has the same problems as distributing them in a single RPM.

Just thinking out loud here, but how about a situation where the icon is only shown when the user clicks on an item from a list to get more information. Then, that icon (and only that icon) is downloaded from something like the pkgdb if the network is up, and a category placeholder icon is used if it is not.

[seth (seth-redhat-bugs)]

(In reply to comment #50)
> I think shoving the icons into the repodata and distributing them in a single
> bundle has the same problems as distributing them in a single RPM.

Thanks, that's why I asked - I didn't know if we were going to be better off or not.

> Just thinking out loud here, but how about a situation where the icon is only
> shown when the user clicks on an item from a list to get more information.
> Then, that icon (and only that icon) is downloaded from something like the
> pkgdb if the network is up, and a category placeholder icon is used if it is
> not.

I'm a little worried about the cost on the pkgdb servers/proxies - but it's not necessarily insurmountable.

[Adel (adel-redhat-bugs)]

(In reply to comment #50)

> Just thinking out loud here, but how about a situation where the icon is only
> shown when the user clicks on an item from a list to get more information.
> Then, that icon (and only that icon) is downloaded from something like the
> pkgdb if the network is up, and a category placeholder icon is used if it is
> not.

This (and the "only show for installed apps") suggestion both suck IMO. (from a user experience POV).

Regarding auditing, as the license tag is generated by a script you'd have to audit the script (i.e check whether it works correctly) rather than checking every license by hand.

[Richard (richard-redhat-bugs)]

(In reply to comment #52)
> Regarding auditing, as the license tag is generated by a script you'd have to
> audit the script (i.e check whether it works correctly) rather than checking
> every license by hand.

See http://github.com/hughsie/app-install/raw/master/contrib/app-install-generate-yum.py -- it's a trivial script.

[Matthias (matthias-redhat-bugs)]

> This (and the "only show for installed apps") suggestion both suck IMO. (from a
> user experience POV).

I agree. We can't have user experience dictated by length of license field...

[Michael (michael-redhat-bugs)]

Richard, this looks obsolete, can it be closed?