Hyphens in macros are translated to underscore

[Tobias Gerschner]

I've noticed an error when attempting to use -Rsourcedir dynamically from the command line to access cached source files

The argument supplied to rpmbuild is here:
54 ++ rpmbuild -v -bs --nodeps --define _Rsourcedir/var/source-cache/p/pps-tools --define _topdir/tmp/tmp.5s8oZpzXVo --define _patchdir/tmp/tmp.5s8oZpzXVo --define _sourcedir/tmp/tmp.5s8oZpzXVo --define _srcrpmdir/home/tobias/SRPMS /home/data/tobias.old/private/yoper/packages/rocketfuel/pps-tools/pps-tools.spec

55 + sources_list='pps-tools_2012-02-11.tar.xz
56 /var/source_cache/p/pps_tools/pps-tools_2012-02-11.tar.xz'

As you can see on line 56 the output of the sources provided by rpm is wrong on 2 accounts:

The hyphen in source-cache has been converted to source_cache as well as the one in the path for pps_tools

I had a look in rpmGenpath hoping I could find a string conversion somewhere that I could tweak, but failed. I believe though it might be the side effect of trying to create safe remote URLs.

A strace -ff -eopen confirmed that rpmbuild is looking in the wrong place

open("/var/source_cache/p/pps_tools/pps-tools_2012-02-11.tar.xz", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)

[Tobias Gerschner (tobias-gerschner)]

This behaviour was observed on rpm 5.2.2

[Jeff Johnson (n3npq)]

Oooh nice!

Let me write up the blueprint and attempt to resurrect
the automated downloading.

For starters, its likely easier to create a my.macros file
and use like
    rpmbuild --macros=my.macros

The 1st line should be
    %{load:/usr/lib/rpm/macros}
and add other lines as needed.

Additional lines like
    %_Rsourcedir /var/source-cache/p/pps-tools
    %_topdir /tmp/tmp.5s8oZpzXVo
could then be added.

You can also use -D,--define as needed: CLI defines will be processed
after --macros (which will also process %{load:...} files. Note that the
nesting is 1-level only: i.e. my.macros can contain a %{load:...} but the file
that is loaded cannot then do a %{load:...}. The restriction is entirely
arbitrary to prevent infinite recursion if a file tries to load itself.

There are ways to use %{lua:...} to process strings, but its better to identify
why the conversion from hyphen to underscore is happening.

There is nothing with rpm macros that attempts hyphen -> underscore intentionally.

Any chance you might be ready to upgrade to rpm-5.3 or rpm-5.4?

[Tobias Gerschner (tobias-gerschner)]

Actually, the _WANT_ is to use the location provided through _Rsourcedir <first> and only if that location does not provide the needed sources attempt to fetch the sources. However we have practices in place that we do not rely on rpmbuild to actually fetch the sources. Actually, we even hack the source of rpm to specifically disable the automatic fetching of sources via rpmbuild.

However the output of rpmbuild of all referenced sources of a specfile with proper macro expansion is very useful to us.

The caching of sources happens based on a convention similar to the one used in debian, live example here: http://ftp.yoper-linux.org/pub/devel/source-cache/ .

File based macros are not appropriate, since these locations are really transient and used on the fly : http://powerplant.yoper-linux.org/projects/software-packages/repository/entry/rocketfuel/rrebuild/rrebuild.sh

From what I've seen so far we should be ready to upgrade to rpm-5.3, I have not yet read up of the implications of using 5.4. There's only the usual hesitation with respect to the noise and knock on effects when bumping the rpm5 version.

[Jeff Johnson (n3npq)]

The Rsourcedir -> Lsourcedir is unidirectional and driven by logic
of non-existence in Lsourcedir.

If you are disabling fetching, then you are expecting something different
from Rsourcedir than what is implemented.

Whether a macro is file-based like
   %{load:/path/to/somewhere/file}
or configuration like
    %oad:%{_somemacro}/file}
is entirely contained in how one chooses what macros to use. I
chose to illustrate with file paths.

At the end of the expansion -- and in expanded *.spec which you seem to want --
it ends up as a file path, so that I/O can be performed on the build machine.

ALl "caching" schemes are store-and-forward and involve rewrites of remote
paths to local paths, usually directories, with file names remaining constant.
That is what is implemented with %_Rsourcedir -> %_Lsourcedir and if
you have disabled fetching, then the store-and-forward mechanism
implemented in rpmbuild is useless, and you are likely better off designing
in your own macro configuration, and re-configuring rpm build to use your
\configuration rather than expecting %_Rsourcedir to do what isn't implemented.

Good: rpm-5.2.2 is mostly end-of-life these days.

rpm-5.3 is "production", rpm-5.4 is "development" and differs hardly at all from rpm-5.3
atm. I can neither here nor see your "noise and knock", bumping an rpm version is
a rather quiet operation.

[Jeff Johnson (n3npq)]

Re: http://powerplant.yoper-linux.org/projects/software-packages/repository/entry/rocketfuel/rrebuild/rrebuild.sh

This is a shell script which uses envvar's. rpmbuild is a program using macros.
You are attempting to impedance match the two name spaces by expanding
macros in a *.spec.

You might make more progress if you write an import (macros -> envvar)
and the equivalent export (envvar -> macros) for whatever configuration
you wish to use.

If the chosen path values are clearly/cleanly mapped, then it doesn't matter
how the *.spec uses macros conventionally, or whether wget in shell instead
of rpmbuild fetches files.

And a clean mapping is easier to debug than reading increasingly complex scripts
and templates and configuration driving a build process.