Debian
kopanocore package

new kopanocore fails in autopkgtest in i386 (fails to start at all actually)

Bug #1733591 reported by Christian Ehrhardt  on 2017-11-21

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	kopanocore (Debian)	Fix Released	Unknown	debbugs #882496
	kopanocore (Ubuntu)	Fix Released	Undecided	Christian Ehrhardt 

Bug Description

Issue in autopkgtest on migration of recent kopanocore:
=> https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic/bionic/i386/k/kopanocore/20171121_085527_b723d@/log.gz

TL;DR - issue when installing:
Setting up kopano-server (8.3.4-4ubuntu3) ...
Determining localhost credentials from /etc/mysql/debian.cnf: succeeded.
dbconfig-common: writing config to /etc/dbconfig-common/kopano-server.conf

Creating config file /etc/dbconfig-common/kopano-server.conf with new version

Creating config file /etc/kopano/debian-db.cfg with new version
checking privileges on database kopanoserver for kopano-server@localhost: user creation needed.
granting access to database kopanoserver for kopano-server@localhost: success.
verifying access for kopano-server@localhost: success.
creating database kopanoserver: success.
verifying database kopanoserver exists: success.
dbconfig-common: flushing administrative password
Processing triggers for libc-bin (2.26-0ubuntu2) ...
Processing triggers for systemd (235-2ubuntu3) ...
done.
Check that we have a running server and can create users...
Unable to open Admin session: network error (0x80040115)
The server is not running, or not accessible through "default:".
Using the -v option (possibly multiple times) may give more hints.

I can reproduce the same in autopkgtest via:
$ autopkgtest --apt-upgrade --shell-fail --apt-upgrade --apt-pocket=proposed --no-built-binaries kopanocore_8.3.4-4ubuntu3.dsc -- qemu ~/work/autopkgtest-bionic-amd64.img
Affects i386 AND amd64 when ran in my autopkgtest

In a container the following should be the same:
# enable proposed
cat <<EOF | debconf-set-selections
kopano kopano-server/mysql/method select unix socket
kopano kopano-server/missing-db-package-error select abort
kopano kopano-server/internal/reconfiguring boolean true
kopano kopano-server/internal/skip-preseed boolean false
kopano kopano-server/dbconfig-upgrade boolean true
kopano kopano-server/dbconfig-install boolean true
kopano kopano-server/dbconfig-reinstall boolean true
kopano kopano-server/db/app-user string kopano-server
kopano kopano-server/mysql/admin-user string root
kopano kopano-server/database-type select mysql
kopano kopano-server/db/dbname string kopanoserver
EOF
DEBIAN_FRONTEND=noninteractive apt -y install mariadb-server
/etc/init.d/mysql start
DEBIAN_FRONTEND=noninteractive apt -y install kopano-server kopano-utils
$ kopano-admin -l | grep -qs "SYSTEM.*Kopano"
=> Reproduced in the container

I ran this three ways now:
1. from bionic
2. from bionic-proposed
3. from bionic, but interactive instead of debconf-set-selections

=> Only the one in proposed is failing.

Upgrading the working #3 (with manual config) makes it break as well.
Note there is a new mariadb in proposed which could cause this.

The server / service is actually failed in those environments.
root@bionic-i386-proposed:~# systemctl restart kopano-server
root@bionic-i386-proposed:~# systemctl status kopano-server
● kopano-server.service - Kopano Server
   Loaded: loaded (/lib/systemd/system/kopano-server.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2017-11-21 13:09:03 UTC; 18s ago
     Docs: man:kopano-server(8)
  Process: 4319 ExecStart=/usr/sbin/kopano-server -F -c /etc/kopano/server.cfg (code=exited, status=127)
  Process: 4318 ExecStartPre=/bin/chown kopano /var/lib/kopano (code=exited, status=0/SUCCESS)
  Process: 4317 ExecStartPre=/bin/chown kopano /var/run/kopano (code=exited, status=0/SUCCESS)
  Process: 4316 ExecStartPre=/bin/mkdir -p /var/run/kopano (code=exited, status=0/SUCCESS)
Main PID: 4319 (code=exited, status=127)

Nov 21 13:09:03 bionic-i386-proposed systemd[1]: Starting Kopano Server...
Nov 21 13:09:03 bionic-i386-proposed systemd[1]: Started Kopano Server.
Nov 21 13:09:03 bionic-i386-proposed systemd[1]: kopano-server.service: Main process exited, code=exited, status=127/n/a
Nov 21 13:09:03 bionic-i386-proposed systemd[1]: kopano-server.service: Failed with result 'exit-code'.

Sometimes in the log there is a message more and I see it when running the server manually:
root@bionic-i386-proposed:~# /usr/sbin/kopano-server -F -c /etc/kopano/server.cfg
/usr/sbin/kopano-server: error while loading shared libraries: cannot apply additional memory protection after relocation: Permission denied

I remember seeing something like that in the past.
It was apparmor then and I see now such messages as well:

[2338345.982219] audit: type=1400 audit(1511269881.445:2191): apparmor="DENIED" operation="file_mprotect" namespace="root//lxd-bionic-i386-proposed_<var-lib-lxd>" profile="/usr/sbin/kopano-server" name="/usr/sbin/kopano-server" pid=23050 comm="kopano-server" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

The new version might need that an extension.

Christian Ehrhardt  (paelzer) on 2017-11-21

Changed in kopanocore (Ubuntu):
status:	New → Confirmed

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-21:

TL;DR - disabling apparmor helps

Details:
for the first deny it needs:
/usr/sbin/kopano-server r,

The next I'm blocked on is:
[2339582.514893] audit: type=1400 audit(1511271117.945:2210): apparmor="DENIED" operation="connect" info="Failed name lookup - disconnected path" error=-13 namespace="root//lxd-bionic-i386-interactive-no-conf_<var-lib-lxd>" profile="/usr/sbin/kopano-server" name="run/mysqld/mysqld.sock" pid=25026 comm="kopano-server" requested_mask="wr" denied_mask="wr" fsuid=113 ouid=112

For that it already has #include <abstractions/mysql> but the disconnected path kills that

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-21:

For the latter we have to add
/usr/sbin/kopano-server flags=(attach_disconnected) {

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-21:

Tested from ppa [1] and working partially.
Need to extend it for:

[2348775.582564] audit: type=1400 audit(1511280310.792:2314): apparmor="DENIED" operation="mkdir" namespace="root//lxd-bionic-i386_<var-lib-lxd>" profile="/usr/sbin/kopano-server" name="/var/lib/kopano/attachments/" pid=8987 comm="kopano-server" requested_mask="c" denied_mask="c" fsuid=0 ouid=0

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3048

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-21:

/var/lib/kopano/attachments/ r,
->
/var/lib/kopano/attachments/ rw,

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-21:

[2351852.074571] audit: type=1400 audit(1511283387.198:2357): apparmor="DENIED" operation="file_mprotect" namespace="root//lxd-bionic-i386_<var-lib-lxd>" profile="/usr/sbin/kopano-dagent" name="/usr/sbin/kopano-dagent" pid=6389 comm="kopano-dagent" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

needs:
/usr/sbin/kopano-dagent r,

Revision history for this message

Adam Conrad (adconrad) wrote on 2017-11-22:

Removing kopanocore from the bionic release pocket and making this a blocking bug for re-entry. Close it when it's not broken anymore.

tags:

added: block-proposed

Revision history for this message

Steve Langasek (vorlon) wrote on 2017-11-22:

block-proposed tag not needed and is chicken-and-egg; the autopkgtest failures already block.

tags:

removed: block-proposed

Revision history for this message

Steve Langasek (vorlon) wrote on 2017-11-22:

kopanocore-partial-fixes.debdiff Edit (2.7 KiB, text/plain)

I'm attaching a debdiff including the results of my own investigations into the failing autopkgtests. This gets a lot farther than before, but:

- on s390x 'kopano-admin -l' fails with an error because there is an endianness bug in kopanocore:
Wed Nov 22 04:45:22 2017: [warning] Object not found unknown user "匀夀匀吀䔀": 匀夀匀吀䔀
That string is:

$ echo '匀夀匀吀䔀' | iconv -f utf8 -t ucs-2le
SYSTE
$

and that's not a regression so is ignorable.

- on arm64 and ppc64el, kopano-dagent exits 75 instead of 0. I don't know if this is the reason, but I see the following denial in dmesg:

[ 567.593536] audit: type=1400 audit(1511327924.317:69): apparmor="DENIED" operation="mknod" profile="/usr/sbin/kopano-dagent" name="/usr/share/kopano-dagent/python/wraplogger.pyc" pid=9977 comm="kopano-dagent" requested_mask="c" denied_mask="c" fsuid=0 ouid=0

creating a .pyc file with mknod() is obviously crazy. I don't know whose bug this is.

aa-disable /usr/sbin/kopano-dagent lets the testsuite finish, so this evidently is the cause of the kopano-dagent failure.

Since this is a regression on ppc64el, and I didn't get to the bottom of it, I didn't bother testing on amd64/i386.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-22:

[2407384.116110] audit: type=1400 audit(1511338917.340:2516): apparmor="DENIED" operation="file_inherit" info="Failed name lookup - disconnected path" error=-13 namespace="root//lxd-bionic-i386-kopano_<var-lib-lxd>" profile="/usr/sbin/kopano-server//kopano_userscripts" name="run/mysqld/mysqld.sock" pid=18187 comm="createuser" requested_mask="wr" denied_mask="wr" fsuid=113 ouid=112

The sub profile also needs attach disconnect for mysql

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-22:

#10

[1936453.049664] audit: type=1400 audit(1511339314.263:153): apparmor="DENIED" operation="file_mprotect" namespace="root//lxd-cpaelzer-bionic-kopano_<var-lib-lxd>" profile="/usr/sbin/kopano-dagent" name="/usr/sbin/kopano-dagent" pid=73949 comm="kopano-dagent" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Dagent does the same mprotect on its bin, similar fix needed

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-22:

#11

More errors that need the follow disconnect for dagent and search.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-22:

#12

Now I'm at the crazy mknod that was discussed when people disabled the server profile.

[1937583.037882] audit: type=1400 audit(1511340444.273:203): apparmor="DENIED" operation="mknod" namespace="root//lxd-cpaelzer-bionic-kopano_<var-lib-lxd>" profile="/usr/sbin/kopano-dagent" name="/usr/lib/python2.7/dist-packages/MAPI/Util/AddressBook.pyc" pid=77123 comm="kopano-dagent" requested_mask="c" denied_mask="c" fsuid=0 ouid=0

and more like that.

But it is not "one-of-a-kind-crazy" see [1][2]

It seems that due to the way it is set up that hits any of its py files being compiled to pyc's.

Along that also found an issue on the kopano server socket.

[1]: https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/810270
[2]: https://github.com/globaleaks/Tor2web/issues/261

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-22:

#13

I started a discussion with security team, but the TL;DR to me seems that this is running pycompile for its own dagent/MAPI components out of the dagent context and therefore apparmor profile.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-22:

#14

At least after all we had before both nosetests are good.
Wrapping aup a new revision in the ppa for tests ...

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-22:

#15

Note: ppa is at https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3048

Tests now done on amd64, i386 and ppc64 (have no arm box, but as outlined that would at least be no regression).

While building I'm refreshing my test systems for the next round.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-22:

#16

DEBIAN_FRONTEND=noninteractive apt install -y kopano-server kopano-utils
systemctl status kopano-server
kopano-admin -c testadmin -p r00tme -e <email address hidden> -f "Your admin" -a 1

DEBIAN_FRONTEND=noninteractive apt install -y kopano-dagent
systemctl status kopano-dagent
cat <<EOF | kopano-dagent -n -v testadmin
From: <email address hidden>
To: <email address hidden>
Subject: the towers
EOF
echo $?

DEBIAN_FRONTEND=noninteractive apt install -y python-nose dpkg-dev
export AUTH_USER='testadmin'
export AUTH_PASS='r00tme'
apt source kopanocore
cd kopanocore-8.3.4/debian/tests/
nosetests -v

Also ran a set of qemu based autopkgtest runs.
With the fixes in the ppa all are solved now.

So with that convincing it might at least work a bit better than before I'm proposing [1] the changes for a 2nd look by someone else.

[1]: https://code.launchpad.net/~paelzer/ubuntu/+source/kopanocore/+git/kopanocore/+merge/334097

Manual Test derived from autopkgtest:
add-apt-repository ppa:ci-train-ppa-service/3048
apt update; apt upgrade
cat <<EOF | debconf-set-selections                                                  
kopano kopano-server/mysql/method select unix socket                                
kopano kopano-server/missing-db-package-error select abort                          
kopano kopano-server/internal/reconfiguring boolean true                            
kopano kopano-server/internal/skip-preseed boolean false                            
kopano kopano-server/dbconfig-upgrade boolean true                                  
kopano kopano-server/dbconfig-install boolean true                                  
kopano kopano-server/dbconfig-reinstall boolean true                                
kopano kopano-server/db/app-user string kopano-server                               
kopano kopano-server/mysql/admin-user string root                                   
kopano kopano-server/database-type select mysql                                     
kopano kopano-server/db/dbname string kopanoserver                                  
EOF
DEBIAN_FRONTEND=noninteractive apt install -y mariadb-server

DEBIAN_FRONTEND=noninteractive apt install -y kopano-server kopano-utils
systemctl status kopano-server
kopano-admin -c testadmin -p r00tme -e admin@example.com -f "Your admin" -a 1

DEBIAN_FRONTEND=noninteractive apt install -y kopano-dagent
systemctl status kopano-dagent
cat <<EOF | kopano-dagent -n -v testadmin                                           
From: hwm@example.com                                                               
To: testadmin@example.com                                                           
Subject: the towers                                                                 
EOF
echo $?

DEBIAN_FRONTEND=noninteractive apt install -y python-nose dpkg-dev
export AUTH_USER='testadmin'
export AUTH_PASS='r00tme'
apt source kopanocore
cd kopanocore-8.3.4/debian/tests/
nosetests -v

Also ran a set of qemu based autopkgtest runs.
With the fixes in the ppa all are solved now.

So with that convincing it might at least work a bit better than before I'm proposing [1] the changes for a 2nd look by someone else.

[1]:  https://code.launchpad.net/~paelzer/ubuntu/+source/kopanocore/+git/kopanocore/+merge/334097

Andreas Hasenack (ahasenack) on 2017-11-22

Changed in kopanocore (Ubuntu):
assignee:	nobody → ChristianEhrhardt (paelzer)
status:	Confirmed → In Progress

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-23:

#17

We discussed the odd pyc apparmor issues.
TL;DR this should be compiled on install, but it isn't.
Reasons:
1. non standard paths
2. not calling dh_python and similar

List of non compiled .py's in kopanocore:

# for py in $(dpkg -L $(dpkg -l | awk '/kopano|mapi/ {print $2}' | xargs) | grep '.py$'); do [ -f ${py}c ] || echo "${py}c missing"; done
/usr/lib/python2.7/dist-packages/kopano_backup/__init__.pyc missing
/usr/share/kopano-dagent/python/mapiplugin.pyc missing
/usr/share/kopano-dagent/python/pluginmanager.pyc missing
/usr/share/kopano-dagent/python/plugins/BMP2PNG.pyc missing
/usr/share/kopano-dagent/python/plugins/examplerules.pyc missing
/usr/share/kopano-dagent/python/plugins/movetopublic.pyc missing
/usr/share/kopano-dagent/python/plugintemplates.pyc missing
/usr/share/kopano-dagent/python/wraplogger.pyc missing
/usr/share/kopano-dagent/python/zconfig.pyc missing
/usr/share/kopano-dagent/python/zinterval.pyc missing
/usr/share/kopano-dagent/python/zunit.pyc missing
/usr/share/doc/kopano-gateway/optimize-imap.pyc missing
/usr/lib/python2.7/dist-packages/kopano_search/plugin_solr.pyc missing
/usr/lib/python2.7/dist-packages/kopano_search/plugin_xapian.pyc missing
/usr/share/kopano-spooler/python/mapiplugin.pyc missing
/usr/share/kopano-spooler/python/pluginmanager.pyc missing
/usr/share/kopano-spooler/python/plugins/disclaimer.pyc missing
/usr/share/kopano-spooler/python/plugintemplates.pyc missing
/usr/share/kopano-spooler/python/wraplogger.pyc missing
/usr/share/kopano-spooler/python/zconfig.pyc missing
/usr/share/kopano-spooler/python/zinterval.pyc missing
/usr/share/kopano-spooler/python/zunit.pyc missing
/usr/lib/python2.7/dist-packages/kopano/daemon/runner.pyc missing
/usr/lib/python2.7/dist-packages/MAPI/Util/Generators.pyc missing
/usr/lib/python2.7/dist-packages/MAPI/Util/Iterators.pyc missing
/usr/lib/python2.7/dist-packages/MAPI/Util/codepage.pyc missing

It already specifies X-Python-Version: 2.7 in d/control

I was able to fix this up with dh_python2 and some changes to control.
That should make sure also dependencies are covered better.
So I was able to drop the write apparmor rules and still work fine.

The only non pyc file left is a doc file which is ok I think.

We discussed the odd pyc apparmor issues.
TL;DR this should be compiled on install, but it isn't.
Reasons:
 1. non standard paths
 2. not calling dh_python and similar

List of non compiled .py's in kopanocore:

It already specifies X-Python-Version: 2.7 in d/control

I was able to fix this up with dh_python2 and some changes to control.
That should make sure also dependencies are covered better.
So I was able to drop the write apparmor rules and still work fine.

The only non pyc file left is a doc file which is ok I think.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-23:

#18

Ok, passed all tests with the new code.
Pushed the follow on fixes to the MP.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-11-23:

#19

Tested and acked on MP - uploading to bionic and hoping this fixes all outstanding issues.
Thanks to everybody involved.

I also rebased onto Debian-sid (sans the php-mapi renaming) and will suggest the same to be picked up by Debian.

Bug Watch Updater (bug-watch-updater) on 2017-11-23

Changed in kopanocore (Debian):
status:	Unknown → New

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-11-23:

#20

This bug was fixed in the package kopanocore - 8.3.4-4ubuntu4

---------------
kopanocore (8.3.4-4ubuntu4) bionic; urgency=medium

  [ Christian Ehrhardt ]
  * keep php7.1-mapi package name to avoid conclicts (LP: #1733572)
    - debian/php-mapi* renamed to debian/php7.1-mapi* (as it was before)
    - debian/control: changed name back to php7.1-mapi
  * fix 8.3.4-4ubuntu3 apparmor issues (LP: #1733591)
    - allow to read (file_mprotect) for kopano-server
    - allow to read (file_mprotect) for kopano-dagent
    - allow server to reach mysql.sock (follow disconnects)
    - allow kopano_userscripts to reach mysql.sock (follow disconnects)
    - allow kopano_dagent to reach mysql.sock (follow disconnects)
    - allow kopano_search to reach mysql.sock (follow disconnects)
    - allow dagent to reach kopano server socket
    - allow to create /var/lib/kopano/attachments
    - python files were not correctly installed
      - debian/rules: add python2 to dh call to correctly render pyc files
      - debian/control: add ${python:Depends} to all packages shiping python
        files
      - add all called dh addons as build depends

  [ Steve Langasek ]
  * Further fixes for new denies and behavior in 8.3.4-4ubuntu3 to fix
    failing autopkgtests (LP 1733591).
    - debian/apparmor/usr.sbin.kopano-server: use @{multiarch} instead of
      a wrongly hard-coded arch string.
    - debian/kopano-server.kopano-server.service: chown the subdirectory.
    - debian/kopano-server.dirs: create /var/lib/kopano/attachments, not just
      /var/lib/kopano.