move to main

[Ken VanDine]

Binary package hint: erlang

https://wiki.ubuntu.com/MainInclusionReport/Erlang

[Alexander Sack (asac)]

seems to duplicate a few system libs (i spotted zlib, pcre at least); can we make erlang use the system libs?

Also there are quite a few binary packages shipped by erlang source. Do we need all for couchdb? maybe we can just promote those needed for it?

Last but not least I think that this needs a quick security team review as it introduces a virtual machine that can run arbitrary code and might get run as root etc.

[Ken VanDine (ken-vandine)]

# erlang-base
# erlang-crypto
# erlang-inets
# erlang-xmerl
# erlang-abi

I think that is all we need for couchdb

[Alexander Sack (asac)]

talked to kees: security wise its ok if you could enable the testsuite by default and set it to fail-on-error.

[Elliot Murphy (statik)]

I've looked into the test suite, it's not going to be simple to enable. It's not nearly as sane as the test suite for mysql, for example. I'll write to the erlang developers at ericsson about this, and try to find a way to get the release tests run during package build, but the current test process involves building a test server, installing it somewhere, then kicking off the tests from inside a running erlang node. It's not feasible to get done today, which is when I found out about this request.

For anyone who is also looking at this test suite, the (sparse) instructions seem to be located at lib/test_server/README in the source tree.

[Elliot Murphy (statik)]

Worse than not simple, it's currently going to be impossible to enable the test suite. The open source release of erlang has the tests/ directories stripped out, so even though the makefiles has a 'release_tests' target, we can't run that in the package build because those tests simply don't exist in the source tree that we have to work with.

[Elliot Murphy (statik)]

I have asked the Erlang team about the missing tests using the erlang-bugs mailing list:

http://www.erlang.org/cgi-bin/ezmlm-cgi?2:mss:1487:200908:ikamdokeffmgahaocnkg

"Hi!

Running the release_tests: target in the Makefile doesn't work, at least
not in R13-B01. It seems that the root of the problems is that the
various tests/ subdirectories are not included in the source tarball,
although the test_server is. For example the makefile tries to cd into
lib/asn1/tests, and that directory doesn't exist.

Why do I care? I'm working on the Erlang packages for Ubuntu (and this
is probably relevant for Debian too). We're including CouchDB by default
in the next release of Ubuntu, and so this means that Erlang is getting
a lot more attention than it has before since it's going on the CD and
in the default Ubuntu desktop installation. The Ubuntu security team has
asked that the test suite be enabled as part of the Erlang package build
process, so that we can have a higher degree of confidence that applying
a bugfix has not broken anything - it's a standard thing that we do in
Ubuntu for all sorts of important foundational packages like languages,
databases, etc.

Is it deliberate that the tests are not included in the source tarball?
Is that something that could be changed?

We're about to start delivering Erlang to many many millions of Ubuntu
desktops in the next few weeks, and I would really really like to be
able to run 'make release_tests' anytime we need to apply a critical
patch or something when processing a security update sometime in the
next 18 months that we will be supporting the packages we have shipped
with Ubuntu 9.10.

Thanks for making Erlang(yay erlang rocks!) and for your consideration."

[Elliot Murphy (statik)]

Hi Ken, I've done all I can with this before freeze so bouncing to you.

[Martin Pitt (pitti)]

So it seems the test suite issue is not that trivial; if we need to apply a patch, we might need to run the tests from upstream's VCS for the time being. Eventually it is important to get them into the proper source package and run on package build, but I don't think it should block this MIR and karmic goal.

> seems to duplicate a few system libs (i spotted zlib, pcre at least); can we make erlang use the system libs?

This _is_ a major issue, though. It should build and link against the system libs instead of including its own copies. Both of those had security issues in the past.

[Elliot Murphy (statik)]

I have found this interesting changelog comment from april about disabling the lintian warning for embedded zlib:

"erlang (1:13.a-dfsg-1) experimental; urgency=low

  * New upstream beta release.
  * Added packages libwxgtk2.8-dev, freeglut3-dev to build dependencies to make
    wxErlang buildable.
  * Split erlang-nox and erlang-x11 packages into a bunch of smaller packages
    for individual Erlang/OTP applications. erlang-nox and erlang-x11 are still
    exist and depend on the correspondent packages.
  * Overridden lintian error 'embedded-zlib' because zlib included into Erlang
    VM is patched, so using system-wide zlib will lead to unknown concequences.
  * Bumped standards version to 3.8.1.

 -- Sergei Golovan < <email address hidden>> Fri, 10 Apr 2009 16:32:04 +0400"

Also, I found this discussion thread:

http://erlang.org/pipermail/erlang-questions/2007-July/027700.html
"> is there an easy way to avoid building against erlang's internal zlib
> so the one from the system is used? If not, could a simple way be
> added?
>
> V-Li
>

AFAIK no, because erlang's zlib is heavily patched :)"

The patching seems to be making zlib use different memory allocation routines. At the end of the thread there is a comment about possibly making it easier to use the system zlib in a future release.

[Elliot Murphy (statik)]

Also, just to be clear: it is impossible to run the upstream test suite because it is not released to the public at all currently.

[Martin Pitt (pitti)]

> AFAIK no, because erlang's zlib is heavily patched :)"

> The patching seems to be making zlib use different memory allocation routines. At the end of the thread there is a > comment about possibly making it easier to use the system zlib in a future release.

This embrace&extend approach doesn't exactly speak in favor of upstream's good development attitude :-(

Kees, do we have a list of embedded code copies, similar to http://svn.debian.org/wsvn/secure-testing/data/embedded-code-copies?op=file ? It should at least be added there. Can someone please open a new bug against erlang to remove the embedded copy?

Debian went through a major effort of removing embedded zlibs, since they were a real pain in the past several times. It feels sad to torpedo these efforts.

[Martin Pitt (pitti)]

Steve is in conversation with upstream, and they are positive about fixing the zlib issue.

One thing that is really important is making the test suite available, though. I don't want a language interpreter in main without a comprehensive test suite, since it's just about impossible to maintain and patch/fix it without one. Is that being discussed as well?

[Steve Alexander (stevea)]

I'm still talking with upstream. Based on that, I am optimistic that Erlang will use standard zlib libraries well in time for 10.04. Elliot has been having a conversation about the test suite, and he's optimistic that it will be available in time for 10.04 also.

While this means that Erlang for 9.10 would use a patched zlib, and would have no test suite, the situation will improve dramatically for 10.04.

[Martin Pitt (pitti)]

Thanks, Steve. Under these conditions I'm fine with approving this now.

[Martin Pitt (pitti)]

Promoted, according to component-mismatches.

[Kees Cook (kees)]

(For the record, similar to P-a-s, we share embedded-code-copies with Debian -- the security team has commit access to that repository if there is anything we need to add.)