Wishlist: Install NEWS in builds

[Uday Reddy]

The VM NEWS file gets lost when distros install VM. Change the build process so that it gets installed in emacs/etc.

Ulrich thinks all VM-related files should go into a subdirectory $(prefix)/share/emacs/etc/vm. Other files that could go into this subdirectory are

CHANGES
README
TODO

Perhaps the images files should go here too?

[Tim Cross (tcross)]

I agree that we need to ensure the NEWS file is not lost and is available to users when they install a packaged version of VM.

However, not sure if <prefix>/share/emacs/etc is the correct location. Not all distros have a <prefix>/share/emacs/etc directory. Ubuntu and Debian certainly don't.

I also notice that many packages do include a NEWS file in <prefix>/share/doc/<package> i.e. /usr/share/doc/vm. This directory also often contains a distro specific README file i.e. README.debian and other related files. The debian and ubuntu packages include such directories and files.

Another problem with using <prefix>/share/emacs/etc is that I don't think many users would actually look there for documentation. Much more likely to look in /usr/share/doc/vm.

Perhaps a better approach would be to put a note in the README file asking distro package maintainers to include the NEWS file and to log bugs with distros if they forget to do this.

[Uday Reddy (reddyuday)]

Tim Cross writes:

> However, not sure if <prefix>/share/emacs/etc is the correct location.
> Not all distros have a <prefix>/share/emacs/etc directory. Ubuntu and
> Debian certainly don't.

The distros decide their own directory structure. What we put in our
build is only indicative.

The problem we face is that Gnu Emacs sets up a site-lisp directory
for elisp files, but it doesn't set up site-bin, site-info and
site-etc directories for the other kinds of files. The rationale for
site-xxx naming is so that they can be preserved when you upgrade
Emacs.

> Another problem with using <prefix>/share/emacs/etc is that I don't
> think many users would actually look there for documentation. Much more
> likely to look in /usr/share/doc/vm.

But they go to emacs/etc to find the GNUS-NEWS?

Cheers,
Uday

[Tim Cross (tcross)]

Uday Reddy writes:
 > Tim Cross writes:
 >
 > > However, not sure if <prefix>/share/emacs/etc is the correct
 > > location. Not all distros have a <prefix>/share/emacs/etc
 > > directory. Ubuntu and Debian certainly don't.
 >
 > The distros decide their own directory structure. What we put in
 > our build is only indicative.
 >

Correct.

 > The problem we face is that Gnu Emacs sets up a site-lisp directory
 > for elisp files, but it doesn't set up site-bin, site-info and
 > site-etc directories for the other kinds of files. The rationale
 > for site-xxx naming is so that they can be preserved when you
 > upgrade Emacs.
 >
 > > Another problem with using <prefix>/share/emacs/etc is that I
 > > don't think many users would actually look there for
 > > documentation. Much more likely to look in /usr/share/doc/vm.
 >
 > But they go to emacs/etc to find the GNUS-NEWS?
 >

Not quite. Under Debian/Ubuntu, the contents of the emacs etc directory are
put in /usr/share/emacs/<emacs-version>/etc. Note that distros derived from
Debian are setup in such a way to allow multiple emacs versions to be
installed at the same time. This means potentially multiple emacs etc
directories.

The more I think about it, the less I think this is worth doing. Given that

* Distro package maintainers will put the files whereever is customzry for
  their system

* We cannot know or satisfy the various different distro policies on file
  locations.

* Distro package maintainers like to avoid making changes to upstream sources
  if possible

* The current proposal would involve creation of a directory and installation
  of a single file which none of the distro maintainers are likely to want,
  requiring them to modify more of the upstream sources. It is also unlikely
  anyone who is installing from sources rather than a package will look in
  <prefix./share/emacs/etc (or even know the directory exists).

* Putting the file in the emacs etc directory i.e.
  <prefix>share/emacs/<emacs-version>/etc is a bad idea as it ties that NEWS
  file to a specific emacs version. If the user updates their emacs, the VM
  news file is in the wrong etc directory (or is lost when the user removes
  that directory). I also don't think we should mix up emacs and non-core
  elisp files.

* The proposal is of little benefit to anyone installing by hand as they will
  have the NEWS file in the root directory of the tar ball or bzr branch
  anyway.

* It is common on many systems (Debian, Ubuntu, RedHat, Fedora, CentOS) for
  README, NEWS, ChangeLog, configuraiton examples, additional documentation
  etc to go in /usr/share/doc/<package>. It is also common to see bug reports
  logged against distro packages that have failed to include a file in this
  directory which the upstream maintainers feel is important for end users.

* The use of an etc/ directory to put the NEWS file doesn't really fit with
  the Filesystem Hierarchy Standard.

My suggestions would be

1. Modify the install to put the NEWS, README and any other files that a
relevant in <prefix>/share/doc/vm. While this will require the creation of
both a share/doc and share/doc/vm directory on many systems that don't have a
...

[Uday Reddy (reddyuday)]

Tim, We decide the structure of the *upstream* distribution and leave
downstream decisions to downstream. Jonathan (Fedora) and Manoj
(Debian) will hopefully tell us the distros' point of view.

But, from an average Emacs user's point of view, I think the
documentation of all Emacs packages should be in one place, modulo
version specializations etc. That means that in the upstream
distribution it should be in emacs/etc/vm or emacs/site-etc/vm.

Uday

[Tim Cross (tcross)]

Uday Reddy writes:
 > Tim, We decide the structure of the *upstream* distribution and
 > leave downstream decisions to downstream. Jonathan (Fedora) and
 > Manoj (Debian) will hopefully tell us the distros' point of view.
 >

I have not disagreed with either of those points. My point is that we should
try to make it as easy as possible for the downstream package maintainers to
do what we would like i.e. include the NEWS file. The current proposal does
the opposite as it will create additional work as they will need to
modify the upstream sources more. Isn't the objective here to get the NEWS
file included in packaged versions?

 > But, from an average Emacs user's point of view, I think the
 > documentation of all Emacs packages should be in one place, modulo
 > version specializations etc. That means that in the upstream
 > distribution it should be in emacs/etc/vm or emacs/site-etc/vm.

From this average Emacs user's point of view, I disagree with creation of
additional directories in the 'public' filesystem hierarchy which are
used by no other package other than VM, particulary when it is for only one or
two files. I also disagree with using etc or site-etc as this use of 'etc' is
not in line with the FSH which specifies etc as a location for configuration
files. The file(s) under discussion are documentation files and should go
under a 'doc' directory.

Whether all emacs package documentation should go in one place or not is a
separate issue to that of getting distro packages to include the NEWS file. I
don't disagree with the idea, but making an ad hoc decision to create
shre/emacs/etc or share/emace/site-etc will do little towards making a
difference.

Have you considered looking into how EPA handles this issue (especially as it
is now part of emacs 24)?

Tim

--
Tim Cross
<email address hidden>

There are two types of people in IT - those who do not manage what they
understand and those who do not understand what they manage.

[Ulrich Müller (ulm)]

> I also disagree with using etc or site-etc as this use of 'etc' is
> not in line with the FSH which specifies etc as a location for configuration
> files.

The FHS says that /etc (directly under the root directory) is for
system configuration files. It doesn't say anything about etc
_sub_directories elsewhere in the filesystem. /usr/share/emacs/etc
is just the analogue of /usr/share/emacs/<version>/etc used by GNU
Emacs. Other Elisp packages use the /usr/share/emacs/etc/<package>
location too.

I hope that everyone agrees that the files should go somewhere under
/usr/share ("read-only architecture independent data files" according
to FHS)?

> The file(s) under discussion are documentation files and should go
> under a 'doc' directory.

In fact, I had already suggested (back in April) to use
/usr/share/doc/vm for the NEWS and README files (but Uday had
objections that it was too Unix specific). But in any case, we should
make the location configurable, so that distros can change it
according to their needs.

[Ulrich Müller (ulm)]

And since you've asked for the distros' point of view, here is the one from Gentoo:

- README and NEWS should go to /usr/share/doc/vm
- Pixmap files (*.xpm) should go under
     /usr/share/emacs/etc/vm (first choice) or
     /usr/share/vm (second choice)

[Uday Reddy (reddyuday)]

Ulrich Müller writes:

> In fact, I had already suggested (back in April) to use
> /usr/share/doc/vm for the NEWS and README files (but Uday had
> objections that it was too Unix specific). But in any case, we should
> make the location configurable, so that distros can change it
> according to their needs.

It hasn't been so much a question of Unix vs. non-Unix. But rather a
question of where would users look for the documentation of emacs
packages? If /usr/share/doc is where they would, then that is
perfectly fine.

I have checked on our department's CentOS installation. The
Gnu-packaged documentation is in /usr/share/emacs/<version>/etc, but
the CentOS people seem to have also put the emacs README file in
/usr/share/doc/emacs-<version>. And, this README file starts, funnily
enough, with "This directory tree holds version 21.4 of GNU Emacs..."
Most other contents of emacs/<version>/etc haven't been copied here.

There were also the documentations of other non-gnu packages in
/usr/share/doc, such as AucTeX, psgml etc. So, it looks like gnu and
non-gnu packages are treated differently.

Well, such is life. So, let us go with /usr/share/doc.

Cheers,
Uday

[Andreas Gustafsson (gson)]

As for the pkgsrc packaging system used by NetBSD among others, most of the packages that install NEWS files seem to put them in $prefix/share/doc/<packagename>/NEWS.

[Jonathan Underwood (jonathan-underwood)]

OK, Uday asked me to comment on this since I package VM for Fedora.

Here's how things work for Fedora packaging (and by extension RHEL and CentOS):

Files such as NEWS, COPYING, README and other documentation are denoted as %doc in the package spec file. What this does is say to the packaging software (i.e. rpmbuild) "These files are documentation for the package". rpmbuild then installs them in /usr/share/doc/%{packagename}-%{version}.

In other words, we don't actually make use of the way that a package installs documents natively. We really don't want to install them under some directory such as /usr/share/emacs/etc. If the upstream vm release does that we'd end up moving them anyway (this is no big deal though).

The above discussion though seems to have broadened in scope by considering where images etc should be installed. My feeling on this is that it's a problem with upstream emacs. Currently, for almost all add-on packages, everything (lisp, image files etc) is dumped into /usr/share/emacs/site-lisp/package. It would be great if there was a better directory structure for add-on packages such as:

/usr./share/emacs/site-packages
/usr./share/emacs/site-packages/lisp
/usr./share/emacs/site-packages/images
/usr./share/emacs/site-packages/etc

What would be great is if upstream promoted a directory structure for add-ons, and then all packages adopted use of that. This should also be integrated with Tom Tromey's package.el (see recent discussions on emacs-devel).

However, it would be a really bad move for VM to autonomously adopt some directory structure of its own.

S, in summary, I feel:

1/ Don't worry about installing NEWS etc - it's up to distributions to do that according to their packaging policies.

2/ Discuss with upstream emacs developers a better strategy for subdirectories for add-on packages.

[Uday Reddy (reddyuday)]

Ulrich Müller writes:

> And since you've asked for the distros' point of view, here is the one
> from Gentoo:
>
> - README and NEWS should go to /usr/share/doc/vm
> - Pixmap files (*.xpm) should go under
> /usr/share/emacs/etc/vm (first choice) or
> /usr/share/vm (second choice)

Does gentoo use /usr/share/emacs/etc at present? Do any other
distro's use it?

Any objections to putting README and NEWS in both the locations?

Cheers,
Uday

[Uday Reddy (reddyuday)]

Uday S Reddy writes:

> There were also the documentations of other non-gnu packages in
> /usr/share/doc, such as AucTeX, psgml etc. So, it looks like gnu and
> non-gnu packages are treated differently.

Our CentOS installation has a directory /usr/share/emacs/site-lisp,
and the various packages like Emacsspeak, AucTeX and psgml use this
for everything, including images, data, documentation etc.

Does anybody else use this structure?

[Ulrich Müller (ulm)]

> Does gentoo use /usr/share/emacs/etc at present? Do any other distro's
> use it?

Gentoo currently uses:
   /usr/share/emacs/site-lisp/<package>/ exclusively for lisp files
   /usr/share/emacs/etc/<package>/ for anything that isn't lisp
                                           (images, sounds, XML schemas, ...)

We made some major effort to move all non-lisp files out of the site-lisp directory.
For some upstream packages it was easy since they have a configure option or variable for this. However, there is no unique convention how the option should be called. Some examples:

   --with-pixmapdir (VM)
   --with-icondir (navi2ch)
   --with-etcdir (Gnus)
   --with-packagedatadir (AUCTeX)
   PIXMAPDIR (Wanderlust)
   ICONDIR (emacs-w3m)

> Any objections to putting README and NEWS in both the locations?

Well, the "distro" and the "upstream" point of view may be different here. From a distro point of view I have to agree with Jonathan, don't bother about the NEWS file. We will tweak the installation to our needs (which means that the file will be installed in /usr/share/doc/<package>-<version>/).

But from an upstream point of view it might be desirable to install the NEWS file with the default "make install".

Why not add a configure option like --with-docdir? Then users (and distro maintainers) could choose their preferred location, or even turn off installation of these files.

Ulrich

[Jonathan Underwood (jonathan-underwood)]

On 11 August 2010 14:38, Uday Reddy <email address hidden> wrote:
> Uday S Reddy writes:
>
>> There were also the documentations of other non-gnu packages in
>> /usr/share/doc, such as AucTeX, psgml etc.  So, it looks like gnu and
>> non-gnu packages are treated differently.
>
> Our CentOS installation has a directory /usr/share/emacs/site-lisp,
> and the various packages like Emacsspeak, AucTeX and psgml use this
> for everything, including images, data, documentation etc.
>
> Does anybody else use this structure?

This is the structure we use for emacs add-on packages for Fedora etc
simply because that's how things have been for a long time, and this
is simply because upstream emacs only creates
/usr/share/emacs/site-lisp by default, and hasn't given any
consideration to how packages should structure themselves. Over time,
packages have become more complex, adding images etc as well as lisp.
To reiterate: this situation (i.e. dumping everything under site-lisp)
is sub-optimal, and needs reconsidering. But it needs to be done in
consultation with upstream emacs devs, so that it becomes a standard
adopted by all add-on packages, not just something VM decides upon.
And it needs to be standardized across distributions too. FWIW, xemacs
has a much better structure for packages:

$ ls /usr/share/xemacs/xemacs-packages/
etc/ lib-src/ lisp/ man/ pkginfo/

[That's for the packages which are packaged up by xemacs devs]

[Tim Cross (tcross)]

From: Jonathan Underwood <email address hidden>
Subject: Re: [Vm] [Bug 612222] Re: Wishlist: Install NEWS in builds
Date: Wed, 11 Aug 2010 14:00:36 -0000

> On 11 August 2010 14:38, Uday Reddy <email address hidden> wrote:
>> Uday S Reddy writes:
>>
>>> There were also the documentations of other non-gnu packages in
>>> /usr/share/doc, such as AucTeX, psgml etc.  So, it looks like gnu and
>>> non-gnu packages are treated differently.
>>
>> Our CentOS installation has a directory /usr/share/emacs/site-lisp,
>> and the various packages like Emacsspeak, AucTeX and psgml use this
>> for everything, including images, data, documentation etc.
>>
>> Does anybody else use this structure?
>
> This is the structure we use for emacs add-on packages for Fedora etc
> simply because that's how things have been for a long time, and this
> is simply because upstream emacs only creates
> /usr/share/emacs/site-lisp by default, and hasn't given any
> consideration to how packages should structure themselves. Over time,
> packages have become more complex, adding images etc as well as lisp.
> To reiterate: this situation (i.e. dumping everything under site-lisp)
> is sub-optimal, and needs reconsidering. But it needs to be done in
> consultation with upstream emacs devs, so that it becomes a standard
> adopted by all add-on packages, not just something VM decides upon.
> And it needs to be standardized across distributions too. FWIW, xemacs
> has a much better structure for packages:
>

I agree.

My suggestion is to hold off on making a decision here until after I've talked
to the elpa guys to find out what their plans are and whether we can make VM
elpa compatible. I'm hoping that ELPA can be the instrument that might get some
package stnadards for emacs add-ons.

Tim

[Uday Reddy (reddyuday)]

Ulrich Müller writes:

> Why not add a configure option like --with-docdir? Then users (and
> distro maintainers) could choose their preferred location, or even turn
> off installation of these files.

Hi Ulrich, I have added a --with-docdir option in the
lp:~vm/vm/build-issues branch, but I am not sure that it is working
right. docdir has a predefined setting which might be interfering
with the configure file. Please take a look.

Cheers,
Uday

[Uday Reddy (reddyuday)]

Ulrich Mueller writes (viewmail-info, 2011-12-29)

I've had some problems with path definitions:
- Variable vm-configure-docdir and vm-configure-infodir are both empty
  strings. Looks like their definition in lisp/Makefile.in is missing.
- The data files (NEWS etc.) are installed into etcdir, therefore
  vm-configure-datadir should be set to etcdir too. (One could think
  about renaming the lisp variable to vm-configure-etcdir. However, in
  the patch below I've chosen not to do this.)
The patch included below addresses these path issues.

[Uday Reddy (reddyuday)]

Thanks, Ulrich. I have applied the patch.

What about the Makefiles in other subdirectories? Don't they need to be
updated too?

Do we still need datadir and datarootdir in the Makefiles now?

Also, the INSTALL file claims that the documentation will be installed in
${prefix}/share/doc/vm-X.Y.Z. But, I don't see the -X.Y.Z part being
generated at the moment.

The reason for putting NEWS etc. files in two places is to satisfy the users
that follow Emacs conversions and those that follow Linux conventions. We
have had plenty of problems in the past with users not knowing what has
changed because they had no access to the the NEWS file.

Cheers,
Uday

[Uday Reddy (reddyuday)]

Ulrich responds:

> What about the Makefiles in other subdirectories? Don't they need
> to be updated too?

So far, I haven't noticed any other problems with installation paths.

One thing that I just stumbled upon: lisp/Makefile.in uses infodir but
all other Makefiles use info_dir. The former is defined by autoconf,
the latter is explicitly assigned in configure.ac. Probably it should
be changed to info_dir everywhere.

> Do we still need datadir and datarootdir in the Makefiles now?

Right, they are not needed any more. Also it looks like top_srcdir can
be removed everywhere, except for the top level Makefile. I can
prepare a patch tomorrow.

> Also, the INSTALL file claims that the documentation will be
> installed in ${prefix}/share/doc/vm-X.Y.Z. But, I don't see the
> -X.Y.Z part being generated at the moment.

So the INSTALL file should be updated?

> The reason for putting NEWS etc. files in two places is to satisfy
> the users that follow Emacs conversions and those that follow Linux
> conventions. We have had plenty of problems in the past with users
> not knowing what has changed because they had no access to the the
> NEWS file.

I'm aware that this is a long standing issue. However, 8.1.1 didn't
install these files at all, and now 8.2.0b installs them twice. ;-)

[Uday Reddy (reddyuday)]

Ulrich Mueller writes:

> > What about the Makefiles in other subdirectories? Don't they need
> > to be updated too?
>
> So far, I haven't noticed any other problems with installation paths.

Well, for example, etcdir doesn't appear in the pixmaps directory's
Makefile.

> >> What do you think about installing them in docdir only, but with
> >> docdir being equal to etcdir by default (as it's the case for
> >> XEmacs already)? That way, distros could change the default as
> >> needed (according to their policy), without installing these files
> >> twice.

Ok, I think I can live with that. If you can include that in your patch
too, that would be great!

Cheers,
Uday

[Ulrich Müller (ulm)]

>>>>> On Fri, 30 Dec 2011, Uday Reddy wrote:

> Well, for example, etcdir doesn't appear in the pixmaps directory's
> Makefile.

>> >> What do you think about installing them in docdir only, but with
>> >> docdir being equal to etcdir by default (as it's the case for
>> >> XEmacs already)? That way, distros could change the default as
>> >> needed (according to their policy), without installing these
>> >> files twice.

> Ok, I think I can live with that. If you can include that in your
> patch too, that would be great!

Hi Uday,
You can find my changes in lp:~ulm/vm/experimental . I've split them
into several commits, so you can pick the ones that you like:

1316 Build system: Use infodir instead of info_dir throughout.

1317 Install ancillary documentation files in docdir only.
      Set default for docdir to be equal to etcdir.

1318 Remove unused and duplicate variable definitions from Makefiles.

1319 Add example.vm to list of documentation files.

1320 Don't install COPYING. The license is included in the Texinfo
      documentation already, which is also used be vm-license.el.
      (Also it's against the policy of most distros to install this file.)

Cheers,
Ulrich

[Ulrich Müller (ulm)]

>>>>> On Fri, 30 Dec 2011, Ulrich Mueller wrote:

> You can find my changes in lp:~ulm/vm/experimental . I've split them
> into several commits, so you can pick the ones that you like:

> [...]

> 1317 Install ancillary documentation files in docdir only.
> Set default for docdir to be equal to etcdir.

This misses a change to vm-view-news, which needs no longer look in
datadir:

1321 Don't look for NEWS file in datadir, it is now installed in docdir only.

[Uday Reddy (reddyuday)]

Hi Ulrich, it all seems to work well. Thanks very much for fixing them. It
would have taken me ages to do it!

The only remaining problem is that of these warnings generated by
configure.

----
config.status: creating Makefile
config.status: WARNING: 'Makefile.in' seems to ignore the --datarootdir setting
config.status: creating lisp/Makefile
config.status: WARNING: 'lisp/Makefile.in' seems to ignore the --datarootdir setting
config.status: creating info/Makefile
config.status: WARNING: 'info/Makefile.in' seems to ignore the --datarootdir setting
----

Won't it be ok to include datarootdir just satisfy configure? Naive users
are likely to get scared by such warnings.

Cheers,
Uday

[Ulrich Müller (ulm)]

>>>>> On Fri, 30 Dec 2011, Uday Reddy wrote:

> The only remaining problem is that of these warnings generated by
> configure.

> ----
> config.status: creating Makefile
> config.status: WARNING: 'Makefile.in' seems to ignore the --datarootdir setting
> config.status: creating lisp/Makefile
> config.status: WARNING: 'lisp/Makefile.in' seems to ignore the --datarootdir setting
> config.status: creating info/Makefile
> config.status: WARNING: 'info/Makefile.in' seems to ignore the --datarootdir setting
> ----

Oops, sorry, I've missed these. Not a real problem though, because
datarootdir isn't used in the generated Makefiles.

> Won't it be ok to include datarootdir just satisfy configure? Naive
> users are likely to get scared by such warnings.

Indeed, that may be better.
Fixed in lp:~ulm/vm/experimental rev.1322.

Cheers,
Ulrich

P.S.: Another thing that I've noticed, the top-level Makefile.in
contains the following lines for the xemacs-package target:

        cd info; make infodir=$(PKGDIR)/info install-pkg
        cd src; make infodir=$(PKGDIR)/bin install-pkg

The second line looks wrong to me. It's like this since 2007 [1]
though, and I'm not sure what's intended. Probably we need an XEmacs
person to check this.

Should I file a new bug, so that the issue won't be forgotten?

[1] <http://bazaar.launchpad.net/~vm/vm/8.2.x/revision/289>

[Uday Reddy (reddyuday)]

Ulrich Mueller writes:

> Should I file a new bug, so that the issue won't be forgotten?

I copied it to Bug 616679. (Will need to check later.)

Cheers,
Uday