OpenERP v5.0.0-2: bdist_rpm files missing

[Gerry Reno]

The following files and directories are needed to make a complete server RPM:

/etc/logrotate.d/openerp-server
/etc/openerp-server.conf
/etc/rc.d/init.d/openerp-server
/var/log/openerp
/var/spool/openerp

UPDATE 2008-02-15: BUG IS NOW SOLVED. PATCHES ARE INCLUDED BELOW IN COMMENTS. - Gerry

[Gerry Reno (greno-verizon)]

# /etc/logrotate.d/openerp-server
/var/log/openerp/*.log {
    copytruncate
    missingok
    notifempty
}

[Gerry Reno (greno-verizon)]

#!/bin/bash
# /etc/rc.d/init.d/openerp-server This shell script takes care of starting and stopping
# openerp server
#
# chkconfig: - 95 05
# description: Starts openerp v4.x Server
#
# pidfile: /var/run/openerp-server$svr.pid
# config: /etc/openerp-server$svr.conf

PATH=/usr/bin:/sbin:/bin:/usr/sbin
export PATH

# Source function library.
. /etc/rc.d/init.d/functions

svr="${2:+-$2}"
PIDFILE=/var/spool/openerp/openerp-server$svr.pid
LOCKFILE=/var/lock/subsys/openerp-server$svr
LOGFILE=/var/log/openerp/openerp-server$svr.log

OPTS="--pidfile=$PIDFILE --logfile=$LOGFILE"

prog="openerp-server"

# check if the openerp-server conf file is present, then use it
if [ -f /etc/openerp-server$svr.conf ]; then
    OPTS="$OPTS -c /etc/openerp-server$svr.conf"
fi

# check the existence of the openerp-server script
[ -z "/usr/bin/openerp-server" ] && exit 0

RETVAL=0

start() {
    echo -n $"Starting $prog: "
    daemon --user openerp --check openerp-server "/usr/bin/setsid /usr/bin/openerp-server $OPTS &"

    RETVAL=$?
    [ $RETVAL -eq 0 ] && touch $LOCKFILE
    echo
    return $RETVAL
}

stop() {
    echo -n $"Stopping $prog: "
    kill -TERM `cat $PIDFILE` > /dev/null 2>&1
    RETVAL=$?
    if [ $RETVAL -eq 0 ] ; then
        rm -f $LOCKFILE

        echo_success
        echo
    else
        echo_failure
        echo
    fi
    return $RETVAL
}

restart() {
    stop
    start
}

condrestart() {
    [ -e $LOCKFILE ] && restart || :
}

rhstatus() {
    if [ -f $PIDFILE ] ; then
        checkpid `cat $PIDFILE`
        RETVAL=$?
        if [ $RETVAL -eq 0 ] ; then
            echo $"$prog is running..."
        else
            echo $"$prog is stopped"
        fi
    else
        echo $"$prog is stopped"
    fi
    return $RETVAL
}

case "$1" in
start)
    start
    ;;

stop)
    stop
    ;;

restart)
    restart
    ;;

condrestart)
    condrestart
    ;;

status)
    rhstatus
    ;;

probe)
    exit 0
    ;;

*)
    echo $"Usage: $0 {start|stop|status|restart|condrestart} [servername]"
    echo $"Example: $0 start # default"
    echo $"Example: $0 start devel # start named server from /etc/openerp-server-devel.conf"
    exit 1
esac

exit $?

[Gerry Reno (greno-verizon)]

Stephane,
  Just a note about CONFIG files:

  In order for config files to not get overwritten during an RPM update, they have to be located in a special macro in the SPEC file:
  %config(noreplace)

  If you don't put the config files under this macro then whatever is in the RPM will just replace what is in the filesystem.

  Google "RPM %config" for more info.

Regards,
Gerry

[Gerry Reno (greno-verizon)]

# openerp-server.conf
[options]
without_demo = False
upgrade = False
verbose = False
xmlrpc = True
db_user = oerp
db_password = abcdef
root_path = None
soap = False
translate_modules = ['all']
db_name = template1
netrpc = True
demo = {}
interface =
db_host = False
db_port = False
port = 8069
addons_path = None
reportgz = False

[Gerry Reno (greno-verizon)]

I have the RPM %config macro working for OpenERP configuration files, so now when a user upgrades their RPM, their config files will not be overwritten. The new config file will be written as CONFIGFILE.rpmnew and the user can then merge their existing config file with the new one.

=================================================
add the following files to your tree (contents are posted above):
etc/openerp-server.conf
etc/rc.d/init.d/openerp-server
etc/logrotate.d/openerp-server

add the following directories to your tree:
var/log/openerp
var/spool/openerp

=================================================
add the following lines to MANIFEST.in:
recursive-include etc *
recursive-include var *

=================================================
add the following lines to the end of rpminstall_sh.txt:
CONFIGFILES="\
%config(noreplace) /etc/openerp-server.conf
%config(noreplace) /etc/logrotate.d/openerp-server
%config(noreplace) /etc/rc.d/init.d/openerp-server
"

echo "$CONFIGFILES" | cat INSTALLED_FILES - > INSTALLED_FILES.new
mv INSTALLED_FILES.new INSTALLED_FILES

=================================================
add the following lines to setup.py in data_files() at about line 103:
        config_directory = opj('/etc')
        files.append((config_directory, [f for f in glob.glob('etc/*') if os.path.isfile(f)]))

        config_directory = opj('/etc', 'logrotate.d')
        files.append((config_directory, [f for f in glob.glob('etc/logrotate.d/*') if os.path.isfile(f)]))

        config_directory = opj('/etc', 'rc.d', 'init.d')
        files.append((config_directory, [f for f in glob.glob('etc/rc.d/init.d/*') if os.path.isfile(f)]))

        work_directory = opj('/var/log/openerp')
        files.append((work_directory, [f for f in glob.glob('var/log/openerp/*') if os.path.isfile(f)]))

        work_directory = opj('/var/spool/openerp')
        files.append((work_directory, [f for f in glob.glob('var/spool/openerp/*') if os.path.isfile(f)]))

=================================================
end

[Gerry Reno (greno-verizon)]

Ok, one small change to accommodate debian file locations:

In all above:
Change: /etc/rc.d/init.d/
To: /etc/init.d

RedHat and derivatives make a symlink between these two directories so you can use either but I don't think debian does that so this change is needed so that files are in correct location for debian as well.

[Gerry Reno (greno-verizon)]

Correction to rpminstall_sh.txt:

Better way:
Instead of adding the previous lines listed above,
just add this one line after the existing 'sed' line:
sed "s!^\(/etc/.*\)!%config(noreplace) \1!" -i INSTALLED_FILES

[Gerry Reno (greno-verizon)]

INCORPORATING THE CORRECTIONS, THE FILES SHOULD LOOK LIKE THIS:

=================================================
add the following files to your tree (contents are posted above):
etc/openerp-server.conf
etc/init.d/openerp-server
etc/logrotate.d/openerp-server

add the following directories to your tree:
var/log/openerp
var/spool/openerp

=================================================
add the following lines to MANIFEST.in:
recursive-include etc *
recursive-include var *

=================================================
add the following line after the existing 'sed' line in rpminstall_sh.txt:
sed "s!^\(/etc/.*\)!%config(noreplace) \1!" -i INSTALLED_FILES

=================================================
add the following lines to setup.py in data_files() at about line 103:
        config_directory = opj('/etc')
        files.append((config_directory, [f for f in glob.glob('etc/*') if os.path.isfile(f)]))

        config_directory = opj('/etc', 'logrotate.d')
        files.append((config_directory, [f for f in glob.glob('etc/logrotate.d/*') if os.path.isfile(f)]))

        config_directory = opj('/etc', 'init.d')
        files.append((config_directory, [f for f in glob.glob('etc/init.d/*') if os.path.isfile(f)]))

        work_directory = opj('/var/log/openerp')
        files.append((work_directory, [f for f in glob.glob('var/log/openerp/*') if os.path.isfile(f)]))

        work_directory = opj('/var/spool/openerp')
        files.append((work_directory, [f for f in glob.glob('var/spool/openerp/*') if os.path.isfile(f)]))

=================================================
end

[Christophe Simonis (OpenERP) (kangol)]

can you provide a patch file ?

[Gerry Reno (greno-verizon)]

Here is the patch file. You still need to create the 'etc' and 'var' directory structures and populate with files.

-Gerry

[Gerry Reno (greno-verizon)]

The patch is against revno 1728.

[Gerry Reno (greno-verizon)]

======================================================================================
MODIFIED SOLUTION: (needed because 'sdist' still cannot create empty directories after 8 years of bugs on this!!!)
This affected the 'var' empty directories and so what we do now is just check/create them in a %post section in SPEC file.
NEW SET OF FILES AND PATCHES FOLLOW.
======================================================================================

=================================================
add the following files to your tree (contents are posted above):
etc/openerp-server.conf
etc/init.d/openerp-server
etc/logrotate.d/openerp-server

=================================================
add the following lines to MANIFEST.in:
recursive-include etc *

=================================================
add the following line after the existing 'sed' line in rpminstall_sh.txt:
sed "s!^\(/etc/.*\)!%config(noreplace) \1!" -i INSTALLED_FILES

%post
[ ! -e /var/log/openerp ] && mkdir -p /var/log/openerp
[ ! -e /var/spool/openerp ] && mkdir -p /var/spool/openerp

=================================================
add the following lines to setup.py in data_files() at about line 103:
        config_directory = opj('/etc')
        files.append((config_directory, [f for f in glob.glob('etc/*') if os.path.isfile(f)]))

        config_directory = opj('/etc', 'logrotate.d')
        files.append((config_directory, [f for f in glob.glob('etc/logrotate.d/*') if os.path.isfile(f)]))

        config_directory = opj('/etc', 'init.d')
        files.append((config_directory, [f for f in glob.glob('etc/init.d/*') if os.path.isfile(f)]))

=================================================
end

PS. retested and upgraded RPM and everything worked fine.

[Gerry Reno (greno-verizon)]

NEW patch is against revno 1728.

[Gerry Reno (greno-verizon)]

======================================================================================
TWEAKED SOLUTION: (needed because certain file permissions were set generically which caused issues so I had to specifically set them)
PLUS: I added creation of 'openerp' user and group, and added install and run output showing location of important files.

NEW SET OF FILES AND PATCHES (vers 3) FOLLOW.
======================================================================================

=================================================
add the following files to your tree (contents are posted above):
etc/openerp-server.conf
etc/init.d/openerp-server
etc/logrotate.d/openerp-server

=================================================
add the following lines to MANIFEST.in:
recursive-include etc *

=================================================
add the following lines after the existing 'sed' line in rpminstall_sh.txt:
sed "s!^\(/etc/openerp-server.conf\)!%attr(-,root,root) %config(noreplace) \1!" -i INSTALLED_FILES
sed "s!^\(/etc/logrotate.d/openerp-server\)!%attr(-,root,root) %config(noreplace) \1!" -i INSTALLED_FILES
sed "s!^\(/etc/init.d/openerp-server\)!%attr(755,root,root) %config(noreplace) \1!" -i INSTALLED_FILES

%post
echo "Checking 'openerp' user and group ..."
id openerp 2>/dev/null || ((groupadd -r openerp && useradd -r -g openerp openerp -c "OpenERP Server" -d /var/spool/openerp -s /sbin/nologin) && id openerp)

[ ! -e /var/log/openerp ] && mkdir -p /var/log/openerp
[ ! -e /var/spool/openerp ] && mkdir -p /var/spool/openerp
chown openerp:openerp /var/log/openerp /var/spool/openerp

echo "Config file: /etc/openerp-server.conf"
echo "Log files under: /var/log/openerp"
echo "openerp home: /var/spool/openerp"
echo "If first install: Create a 'oerp' user and database in Postgresql; put database entries in config file."
echo "Control server: /etc/init.d/openerp-server help"

=================================================
add the following lines to setup.py in data_files() at about line 103:
        config_directory = opj('/etc')
        files.append((config_directory, [f for f in glob.glob('etc/*') if os.path.isfile(f)]))

        config_directory = opj('/etc', 'logrotate.d')
        files.append((config_directory, [f for f in glob.glob('etc/logrotate.d/*') if os.path.isfile(f)]))

        config_directory = opj('/etc', 'init.d')
        files.append((config_directory, [f for f in glob.glob('etc/init.d/*') if os.path.isfile(f)]))

=================================================
end

PS. retested and upgraded RPM and everything worked fine.

[Gerry Reno (greno-verizon)]

TWEAKED Patch is against revno 1728.

[Gerry Reno (greno-verizon)]

Where it says above: add the following files to your tree (contents are posted above):

PLEASE TAKE THE CONTENTS FROM THE TWEAKED gzipped-tar ATTACHMENT.

[Gerry Reno (greno-verizon)]

After you untar the files into your build tree and patch the files:

TO BUILD THE RPM:
In your patched tree:
$ python setup.py sdist
$ cp dist/openerp-server-5.0.0-3.tar.gz ~/build/python-dist/
$ cd ~/build/python-dist
$ tar xvf openerp-server-5.0.0-3.tar.gz
$ cd openerp-server-5.0.0-3
$ python setup.py bdist_rpm
...

TO INSTALL THE RPM:
$ su -
# yum localinstall /home/greno/build/python-dist/openerp-server-5.0.0-3/dist/openerp-server-5.0.0-3.noarch.rpm
Loading "fastestmirror" plugin
Loading "installonlyn" plugin
Setting up Local Package Process
Examining /home/greno/build/python-dist/openerp-server-5.0.0-3/dist/openerp-server-5.0.0-3.noarch.rpm: openerp-server - 5.0.0-3.noarch
Marking /home/greno/build/python-dist/openerp-server-5.0.0-3/dist/openerp-server-5.0.0-3.noarch.rpm to be installed
Loading mirror speeds from cached hostfile
 * fedora: download.fedora.redhat.com
 * updates: download.fedora.redhat.com
Resolving Dependencies
--> Running transaction check
---> Package openerp-server.noarch 0:5.0.0-3 set to be updated

Dependencies Resolved

=============================================================================
 Package Arch Version Repository Size
=============================================================================
Installing:
 openerp-server noarch 5.0.0-3 /home/greno/build/python-dist/openerp-server-5.0.0-3/dist/openerp-server-5.0.0-3.noarch.rpm 7.6 M

Transaction Summary
=============================================================================
Install 1 Package(s)
Update 0 Package(s)
Remove 0 Package(s)

Total download size: 7.6 M
Is this ok [y/N]: y
Downloading Packages:
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing: openerp-server ######################### [1/1]
Checking 'openerp' user and group ...
uid=497(openerp) gid=496(openerp) groups=496(openerp) context=user_u:system_r:rpm_script_t
Config file: /etc/openerp-server.conf
Log files under: /var/log/openerp
openerp home: /var/spool/openerp
If first install: Create a 'oerp' user and database in Postgresql; put database entries in config file.
Control server: /etc/init.d/openerp-server help

Installed: openerp-server.noarch 0:5.0.0-3
Complete!
==============================================================

AND TO START THE SERVER:
# /etc/init.d/openerp-server start
Starting openerp-server: [ OK ]
Config file: /etc/openerp-server.conf
Log file: /var/log/openerp/openerp-server.log
PID file: /var/spool/openerp/openerp-server.pid
Lock file: /var/lock/subsys/openerp-server

==============================================================

And that's what you should see if everything is working.

[Gerry Reno (greno-verizon)]

Where it says above: add the following files to your tree (contents are posted above):

PLEASE TAKE THE CONTENTS FROM THE TWEAKED-2 gzipped-tar ATTACHMENT.

[Gerry Reno (greno-verizon)]

TWEAKED-2 Patch is against revno 1759.

[Gerry Reno (greno-verizon)]

TWEAKED-3 SOLUTION:
Found a small 'release' parse error where we need to use rsplit rather than split alone. So I ran a new diff patch and file set.

Where it says above: add the following files to your tree (contents are posted above):

PLEASE TAKE THE CONTENTS FROM THE TWEAKED-3 gzipped-tar ATTACHMENT.

[Gerry Reno (greno-verizon)]

TWEAKED-3 Patch is against revno 1759.

[Gerry Reno (greno-verizon)]

TWEAKED-4 SOLUTION:
Sorry. Forgot to set the 'release' back to '3' after testing since we don't want to change it.

Where it says above: add the following files to your tree (contents are posted above):

PLEASE TAKE THE CONTENTS FROM THE TWEAKED-4 gzipped-tar ATTACHMENT.

[Gerry Reno (greno-verizon)]

TWEAKED-4 Patch is against revno 1759.

[Stephane Wirtel (OpenERP) (stephane-openerp)]

Ok, I have created a new branch with the trunk version and your patch. Can you test this branch because I have got an error when I try to create the rpm package. In fact the "-ba" option is not recognized by the rpm command.

https://code.launchpad.net/~stephane-openerp/openobject-server/rpm-config-files

Thanks

[Gerry Reno (greno-verizon)]

Stephane,
  Ok, I got the branch but all the files from TWEAKED-4 are not there, only the patches are there. We also need the files which will create a 'etc' directory structure. Right now there is no 'etc' directory just 'bin', 'doc' and 'man', 'pixmaps', 'win32'.

Regards,
Gerry

[Stephane Wirtel (OpenERP) (stephane-openerp)]

greno wrote:
> Stephane,
> Ok, I got the branch but all the files from TWEAKED-4 are not there, only the patches are there. We also need the files which will create a 'etc' directory structure. Right now there is no 'etc' directory just 'bin', 'doc' and 'man', 'pixmaps', 'win32'.
>
> Regards,
> Gerry
>
I have applied your tar ball in this branch, can you check that ?

Sorry for the delay

Regards,
Stephane

--
Stephane Wirtel - "As OpenERP is OpenSource, please feel free to contribute."
Developper - Technical Lecturer OpenERP
OpenERP - Tiny SPRL
Chaussee de Namur, 40
B-1367 Gerompont
Tel: [phone]
Web: http://www.tiny.be
Web: http://www.openerp.com
Planet: http://www.openerp.com/planet/
Blog: http://stephane-wirtel-at-tiny.blogspot.com

[Gerry Reno (greno-verizon)]

Stephane,
  I pulled the server branch and the files are now there. I was able to run 'setrelease', then 'sdist', copy tarball to work area, run 'bdist_rpm' but the RPM build is still reporting "Installed but unpackaged files". I used my workaround for the /usr/lib/rpm/redhat/macros file to get by this issue and then the RPMS built successfully. I installed the binary RPM and OpenERP server configuration occurred during the installation. I tried starting the server with '/etc/init.d/openerp-server start' and the server started first time without error. I was able to connect from the client and create a new database. So I think the only issue is that of the "Installed but unpackaged files".

Regards,
Gerry

[Stephane Wirtel (OpenERP) (stephane-openerp)]

greno wrote:
> Stephane,
> I pulled the server branch and the files are now there. I was able to run 'setrelease', then 'sdist', copy tarball to work area, run 'bdist_rpm' but the RPM build is still reporting "Installed but unpackaged files". I used my workaround for the /usr/lib/rpm/redhat/macros file to get by this issue and then the RPMS built successfully. I installed the binary RPM and OpenERP server configuration occurred during the installation. I tried starting the server with '/etc/init.d/openerp-server start' and the server started first time without error. I was able to connect from the client and create a new database. So I think the only issue is that of the "Installed but unpackaged files".
>
> Regards,
> Gerry
>
Gerry,

I don't have the time to solve this bug, can you provide a small patch and I
will apply to this branch ?

Regards,
Stephane

--
Stephane Wirtel - "As OpenERP is OpenSource, please feel free to contribute."
Developper - Technical Lecturer OpenERP
OpenERP - Tiny SPRL
Chaussee de Namur, 40
B-1367 Gerompont
Tel: [phone]
Web: http://www.tiny.be
Web: http://www.openerp.com
Planet: http://www.openerp.com/planet/
Blog: http://stephane-wirtel-at-tiny.blogspot.com

[Gerry Reno (greno-verizon)]

Stephane,
  The "Installed but unpackaged files" error was introduced again recently after another server bug was fixed.

  After the server is fully working without error, we need to do the following for the client:
  add 'setrelease' class and it's install cmd in 'setup()'

Regards,
Gerry

[Gerry Reno (greno-verizon)]

Stephane,
  Here was the change introduced in revno 1685 that caused the "Installed but unpackaged files" error to occur for all .pyc/.pyo files under:
 /usr/lib/python2.5/site-packages/openerp-server/addons/base

- innerfiles = filter(lambda file: os.path.splitext(file)[1] not in ('.pyc', '.py', '.pyd', '.pyo'), innerfiles)
+ innerfiles = filter(lambda file: os.path.splitext(file)[1] not in ('.pyc', '.pyd', '.pyo'), innerfiles)

  I have forgotten what this 1685 fixed but I know that people were complaining about files missing in the forum so I don't think this fix should be rolled back. Do you remember what problem 1685 actually fixed? Is there another way to fix that problem without changing the innerfiles?

What has happened is this fix has changed the files[] list and that is the cause of the IBUF error. I've attached the generated INSTALLED_FILES file. In it you can see at the end that all the 'addons/base' files are listed and the .py files have no corresponding .pyc/.pyo as the other .py files do. So that is why the error occurs. I don't see how to fix this because I'm not familiar with the innerfiles logic in setup.py.

Regards,
Gerry

[Gerry Reno (greno-verizon)]

Stephane,
  In watching the rpm generation process I can see where all the byte-compiling is occurring and then AFTER that the openerp-server/bin/addons .py files are copied to openerp-server/addons. So how to make the copy occur before the byte-compile or how to get the .pyc and .pyo files copied over to openerp-server/addons? Please take a look at this. I've gone about as far as I know how with this.

Regards,
Gerry

[Gerry Reno (greno-verizon)]

Stephane,
  I had a little time today so I investigated further the 'Installed by unpackaged files' problem introduced by bugfix 1685 and I found a solution. Patch is attached.
  Everything is working now. Please test on your Fedora as well.

Regards,
Gerry

[Gerry Reno (greno-verizon)]

Stephane,
  After you apply the latest patch, please build on your Fedora as well. I see you were having a problem with building the RPM with no -ba option. This means that you do not have 'rpmbuild' installed. You cannot build an RPM with rpm. You have to use rpmbuild and setup.py will use rpmbuild if it is present.
You need to install the following on your Fedora:
yum groupinstall "Development Tools"

  Also, we need to do something on the client as well: add 'setrelease' class and it's install cmd in 'setup()'.

Regards,
Gerry

[Todd Johnson (todd-toddejohnson)]

It works great with openerp-server-5.0.3. This also needs done with openerp-web. Any plans on doing this for openerp-web?

[pheller (pheller)]

Any status on this? Likely to see it merged into the trunk anytime soon?