Ubuntu nodejs package isn't ABI compatible with mainline nodejs.

Bug #1779863 reported by Nicolas Noble on 2018-07-03
34
This bug affects 3 people
Affects Status Importance Assigned to Milestone
nodejs (Debian)
New
Unknown
nodejs (Ubuntu)
Medium
Dan Streetman
Trusty
Undecided
Unassigned
Xenial
Undecided
Unassigned
Bionic
Medium
Dan Streetman
Cosmic
Medium
Dan Streetman

Bug Description

[impact]

Pre-built addons for nodejs built against the 8.10 version, which is what is included in Bionic, will fail to load on Bionic because the version of nodejs there is built using a newer ABI-incompatible openssl version.

[test case]

1. Run 'sudo apt install nodejs npm'
2. Run 'mkdir /tmp/lp.1779863'
3. Run 'cd /tmp/lp.1779863'
4. Run 'npm install grpc'
5. Note that it mentions installing a prebuilt binary from remote.
6. Run 'node' and enter the following code:
> const grpc = require('grpc')
> const creds = grpc.ServerCredentials.createSsl(null, [])
> const server = new grpc.Server()
> server.bind('0.0.0.0:8080', creds)
7. Observe that this results in a crash, with an error like:
node: symbol lookup error:
/tmp/lp.1779863/node_modules/grpc/src/node/extension_binary/node-v57-linux-x64-glibc/grpc_node.node:
undefined symbol: SSL_library_init
8. Run 'npm install node-webcrypto-ossl'.
9. Observe that compilation fails due to a header expectation mismatch.
10. Install nodejs from -proposed.
11. Repeat steps 6-9 and observe that the commands succeed without errors.

[regression potential]

Although this SRU changes the ABI of nodejs that is exposed to binary add-ons, the practical regression potential for this ABI change is minimal. The archive has been scanned to confirm there are no reverse-dependencies in Ubuntu which use this part of the ABI, and it is not feasible to build third-party binaries that are compatible with nodejs as shipped in 18.04 because the gyp build system used by the nodejs ecosystem exposes system headers that don't match the symbols exported by the current Ubuntu nodejs built against OpenSSL 1.1.

Thus, the greatest risk of regression is from someone manually working around this gyp incompatibility in order to build an add-on which uses these symbols. This risk is negligible.

Changing this to use openssl1.0 assumes that the security team will maintain security patches for openssl1.0.

There is no risk of regression in protocol compatibility by switching back from openssl 1.1 to openssl 1.0, because TLS 1.3 support has not yet landed in the openssl package in 18.04.

[other info]

alternately, this could be fixed by upgrading the nodejs package in Bionic (and Cosmic) to a newer nodejs - Debian has version 10.4.0 in experimental.

also debian 904274 has quite a bit of discussion.

original bug description below.
---

Background:
NodeJS has a native extension API: https://nodejs.org/api/addons.html
It's fairly understood by developers that NodeJS's ABI is stable, and that one module built using a version of nodejs should work on another semantically version compatible of nodejs.

NodeJS exposes various third party libraries to the native module developers. Quote from the addons developers page: "Node.js includes a number of other statically linked libraries including OpenSSL. These other libraries are located in the deps/ directory in the Node.js source tree. Only the libuv,i OpenSSL, V8 and zlib symbols are purposefully re-exported by Node.js and may be used to various extents by Addons."

It's fairly understood by developers that native modules have the same ABI guarantee than the rest of the node API.

The NodeJS ecosystem uses native modules extensively, and it's fairly common for developers to publish precompiled versions of their extensions so that the typical end-user can simply npm install their dependencies without worrying about having a compiler installed. Some packages will do their own thing (see for instance https://www.npmjs.com/package/uws), while others will rely on third party extensions to facilitate their work. See for instance prebuild (https://www.npmjs.com/package/prebuild) that has a handful of dependents, or node-pre-gyp (https://www.npmjs.com/package/node-pre-gyp) that has north of 350 dependents. So the nodejs ecosystem has roughly 400 native packages that are publishing prebuilt versions of their extensions.

Problem with the Ubuntu nodejs package:
Put simply, it breaks prebuilt packages that depend on OpenSSL. NodeJS 8.10.0 officially comes with OpenSSL 1.0.2n, while the NodeJS 8.10.0 that comes with the Ubuntu package exposes OpenSSL 1.1.0g.

Since there are ABI breakages between OpenSSL 1.0.2 and 1.1.0, these ABI breakages are bubbling up to any prebuilt native addon.

Here is an example:

https://github.com/nicolasnoble/openssl-nodejs-ubuntu-demo

If you build this package under the mainline nodejs, it will try to import the following symbols from OpenSSL 1.0.2:
 . SSL_library_init
 . SSLeay_version

Whereas if you build it under Ubuntu's nodejs, it will try to import the following symbols instead from OpenSSL 1.1.0:
 . OPENSSL_init_ssl
 . OpenSSL_version

Therefore, trying to load one prebuilt module from one version of the runtime to another will result in a symbol loading error:

node: symbol lookup error: /home/pixel/node-openssl-addon-example/build/Release/openssl_example.node: undefined symbol: SSL_library_init

Incidentally, nodejs 10.5.0 uses OpenSSL 1.1.0h, and compiling the same demo module with this version of node will try to import the proper symbols. Obviously, since the module will be built for the wrong version of the nodejs runtime, it won't load, but the SSL symbols are now proper.

This creates weird bug reports for nodejs extension developers, such as https://github.com/grpc/grpc-node/issues/341

Another example is uws. Trying to use uws in ubuntu's nodejs will result in the same sort of failures. Which means there are at least two packages available out there that are affected by this issue.

I don't think this is easily solvable, and all of my suggestions for fixing it have severe cons. Ubuntu won't want to downgrade their system's OpenSSL for this. Maybe there's a way to get another openssl package for 1.0.2, and have the nodejs runtime for Ubuntu depend on it. Another possible solution would be to radically upgrade nodejs to 10, so that the ABI of OpenSSL will then match properly. But this may be viewed as a too radical upgrade.

One sort of mitigation option would be to get node-pre-gyp and prebuild to recognize that it's running with this version of the nodejs runtime, so that it can recognize and take actions, such as downloading an ubuntu-specific version of the prebuilt extension, or recompiling from sources. This obviously would help mitigating the issue for a good portion of existing packages that are using node-pre-gyp and prebuild, but for packages that are doing their own thing such as uws, this solution wouldn't work properly.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: nodejs 8.10.0~dfsg-2
ProcVersionSignature: Ubuntu 4.15.0-24.26-generic 4.15.18
Uname: Linux 4.15.0-24-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Tue Jul 3 05:34:28 2018
EcryptfsInUse: Yes
InstallationDate: Installed on 2017-03-08 (482 days ago)
InstallationMedia: Ubuntu 16.10 "Yakkety Yak" - Release amd64 (20161012.2)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: nodejs
UpgradeStatus: Upgraded to bionic on 2018-05-10 (54 days ago)

Nicolas Noble (grumpycoder) wrote :
Dan Streetman (ddstreet) wrote :

I think I don't quite understand the use case fully - the problem here is some binary packages, provided not by Ubuntu but through NPM, aren't binary-compatible with nodejs due to openssl lib ABI differences with the NPM-built binary? If you can provide a bit more detail on this, maybe with a specific example or way to reproduce seeing the error, it would help a lot.

Also for bionic, the openssl1.0 version is available:

$ sudo apt install openssl1.0 libssl1.0.0

$ apt-cache policy libssl1.0.0
libssl1.0.0:
  Installed: 1.0.2n-1ubuntu5.1
  Candidate: 1.0.2n-1ubuntu5.1
  Version table:
 *** 1.0.2n-1ubuntu5.1 500
        500 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.2n-1ubuntu5 500
        500 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

Is the issue here that nodejs is statically built with the newer openssl lib?

Nicolas Noble (grumpycoder) wrote :

I've created an issue on nodejs' tracker to discuss this also: https://github.com/nodejs/node/issues/21897

Let me rephrase what you just said a bit, because I think you're getting it a bit incorrectly.

People are distributing binaries through npm, and these binaries are expected to work directly when being loaded inside the nodejs runtime because there are ABI contracts from the nodejs runtime.

These contracts include exposing OpenSSL symbols. More specifically, NodeJS extensions aren't supposed to link against OpenSSL directly. They get OpenSSL's symbols transitively through the NodeJS binary. This is also true for zlib and libuv. Ubuntu (transitively from Debian) is shipping a version of the NodeJS runtime that breaks these ABI contracts by linking in OpenSSL 1.1 instead of 1.0. More specifically, the symbols and ABI exposed from NodeJS 8.x are supposed to be that of OpenSSL 1.0, but since your runtime is linked against 1.1, it exposes this ABI instead, which transitively breaks native modules that are expecting the 1.0 ABI of OpenSSL.

Nicolas Noble (grumpycoder) wrote :

Also if you want to reproduce it, you simply need to npm install grpc on Ubuntu using Ubuntu's nodejs. Loading and using gRPC (I can build a quick nodejs demo project if you want) will then fail with the symbol mismatch I described initially.

Or you can also use the reproduction case I linked initially: https://github.com/nicolasnoble/openssl-nodejs-ubuntu-demo

If you build this extension using the nodejs 8 available at https://nodejs.org/dist/latest-v8.x/ and then try to use the code in Ubuntu's nodejs, it'll fail. One way to quickly try this is to have two users on a single system. One user is using nvm to install node 8. That first user builds the demo code. Then the second user uses the system's node to run it. This will demonstrate the failure.

Elana Hashman (ehashman) wrote :

Hi all,

Because this bug was filed against the unmodified package version from Debian, I also filed this bug against Debian upstream: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=904274

@ddstreet it is pretty common for end users to install a language runtime (e.g. python, nodejs, etc.) and package manager (e.g. pip, npm, etc,) from the OS while then using that package manager to download libraries/dependencies from the language-specific registry. I think it's pretty important to ensure that the language runtimes we ship in Debian/Ubuntu uphold the ABI contracts that upstream languages establish for distributing compiled native extensions.

In order to fix this in Bionic, I believe we need to change the builddep/dependency to depend on the package that provides openssl 1.0.2.

In Buster, I'm suggesting that we upgrade to node 10.x to fix the issue.

- e

Changed in nodejs (Debian):
status: Unknown → New
Dan Streetman (ddstreet) wrote :

From reading the debian bug report, it seems like there is considerable discussion that still needs to happen separate from debian or ubuntu packaging. Additionally, this seems to not be an issue in debian, once they update nodejs in the debian repository to the latest version.

So, I'll ignore upstream and debian discussion and changes, since that seems to be ongoing, and IIUC won't impact whatever change we make to nodejs in ubuntu (for current releases). Also, I'll mark this bug as affecting only bionic (and possibly cosmic, since debian hasn't yet updated nodejs to 0.10). For trusty and xenial, again IIUC, nodejs is built with the correct openssl so this ABI mismatch problem doesn't exist there.

> NodeJS 8.10.0 officially comes with OpenSSL 1.0.2n

I think this is an important piece of info here, and I wasn't able to find any "official" documentation that clearly states this - can you provide a link to any docs for this? I'm not disputing it, but if we're going to rebuild nodejs with a different openssl version it would be good to be able to point to upstream docs to justify it.

> if you want to reproduce it, you simply need to npm install grpc on Ubuntu using
> Ubuntu's nodejs. Loading and using gRPC (I can build a quick nodejs demo project if you want)

yes, please - I'm not familiar with nodejs or npm, so while I can do 'sudo apt install nodejs ; sudo npm install grpc ; git clone https://github.com/nicolasnoble/openssl-nodejs-ubuntu-demo', I'm not entirely sure what exactly to do after that to reproduce this. It's important (or at least, very helpful) to have specific clear steps to reproduce and verify the bug, and verify the fixed package (even to those not familiar with nodejs/npm).

Also, I've built nodejs for bionic in this ppa, altering the build and runtime deps to use bionic's openssl1.0 and libssl1.0-dev; can you test with this build and see if it fixes the problem?
https://launchpad.net/~ddstreet/+archive/ubuntu/lp1779863

Changed in nodejs (Ubuntu Trusty):
status: New → Invalid
Changed in nodejs (Ubuntu Xenial):
status: New → Invalid
Changed in nodejs (Ubuntu Bionic):
status: New → In Progress
Changed in nodejs (Ubuntu Cosmic):
status: New → In Progress
Changed in nodejs (Ubuntu Bionic):
importance: Undecided → Medium
Changed in nodejs (Ubuntu Cosmic):
importance: Undecided → Medium
Dan Streetman (ddstreet) on 2018-07-23
Changed in nodejs (Ubuntu Bionic):
assignee: nobody → Dan Streetman (ddstreet)
Changed in nodejs (Ubuntu Cosmic):
assignee: nobody → Dan Streetman (ddstreet)
Elana Hashman (ehashman) wrote :

>> NodeJS 8.10.0 officially comes with OpenSSL 1.0.2n
>
> I think this is an important piece of info here, and I wasn't able to find any "official"
> documentation that clearly states this - can you provide a link to any docs for this?

I don't know if this is officially documented anywhere but here's the vendored OpenSSL in NodeJS upstream as of the 8.10.0 release, confirming the dep is OpenSSL 1.0.2n: https://github.com/nodejs/node/blob/2fce636f578b63fc0115e6c9f68157f6c4517145/deps/openssl/openssl/README#L2

Elana Hashman (ehashman) wrote :

Oops, GitHub expanded that link to the full commit when I copied it. Here's demonstrating it's the same one as the 8.10.0 tag: https://github.com/nodejs/node/blob/v8.10.0/deps/openssl/openssl/README#L2

Release notes are here https://github.com/nodejs/node/releases/tag/v8.10.0

There's also a "documentation" in nodejs' release database:
https://nodejs.org/dist/index.json

On Mon, Jul 23, 2018, 15:56 Elana Hashman <email address hidden>
wrote:

> Oops, GitHub expanded that link to the full commit when I copied it.
> Here's demonstrating it's the same one as the 8.10.0 tag:
> https://github.com/nodejs/node/blob/v8.10.0/deps/openssl/openssl/README#L2
>
> Release notes are here
> https://github.com/nodejs/node/releases/tag/v8.10.0
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1779863
>
> Title:
> Ubuntu nodejs package isn't ABI compatible with mainline nodejs.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/nodejs/+bug/1779863/+subscriptions
>

Dan Streetman (ddstreet) wrote :

@ehashman, @grumpycoder, let me know if you're able to test with the deb from my ppa, so we can at least confirm it does fix this issue.

Nicolas Noble (grumpycoder) wrote :

I can confirm that your nodejs from your ppa makes prebuilt binaries work
fine again, yes.

On Tue, Jul 24, 2018 at 2:35 PM, Dan Streetman <email address hidden>
wrote:

> @ehashman, @grumpycoder, let me know if you're able to test with the deb
> from my ppa, so we can at least confirm it does fix this issue.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1779863
>
> Title:
> Ubuntu nodejs package isn't ABI compatible with mainline nodejs.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/nodejs/+bug/
> 1779863/+subscriptions
>

Dan Streetman (ddstreet) wrote :

Jérémy (@kapouer), as the maintainer of nodejs for Debian, what's your opinion on Ubuntu changing nodejs's build and runtime deps specifically (and only) for Ubuntu Bionic, to use the openssl1.0 (currently version 1.0.2n-1ubuntu5.1) instead of openssl (currently version 1.1.0g-2ubuntu4.1)? It seems like that is the correct thing to do for Bionic.

For the Ubuntu devel release in progress (Cosmic) I believe it will sync the later nodejs version before its release, but I'll leave this bug tracking it until it is updated to nodejs 10.4.0 (or later).

Dan Streetman (ddstreet) wrote :

@ubuntu-security, since this proposed change to nodejs involves altering its build/runtime deps to use openssl 1.0 instead of the newer openssl 1.1, please review the details and comment if you have any concerns.

Dan Streetman (ddstreet) wrote :
tags: added: sts-sponsor-ddstreet
Dan Streetman (ddstreet) wrote :

@grumpycoder, I added the SRU-required template to the bug description; please correct my statements in the template if needed.

description: updated
Dan Streetman (ddstreet) wrote :

$ reverse-depends -r bionic -b -l nodejs | wc -l
1090

I think we'll have to rebuild all 1090 of these Ubuntu-provided packages after updating Bionic's nodejs openssl dep.

description: updated
Dan Streetman (ddstreet) on 2018-07-25
description: updated
Marc Deslauriers (mdeslaur) wrote :

Since nodejs isn't maintained by the security team, we have no comment on the switch to openssl1.0 from a security point of view.

tags: added: patch
Nicolas Noble (grumpycoder) wrote :

The template changes look good to me.

On Wed, Jul 25, 2018 at 9:21 AM, Ubuntu Foundations Team Bug Bot <
<email address hidden>> wrote:

> ** Tags added: patch
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1779863
>
> Title:
> Ubuntu nodejs package isn't ABI compatible with mainline nodejs.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/nodejs/+bug/
> 1779863/+subscriptions
>

Dan Streetman (ddstreet) wrote :

I've uploaded the patched nodejs to the Bionic queue, as well as discussed it in #ubuntu-release; the SRU approvers will either accept or reject it now, and I (or someone) will still need to go through all the Ubuntu packages that have a dependency on nodejs, to check which (if any) will need to be rebuilt with the new nodejs (since the ABI will be different).

Adam Conrad (adconrad) wrote :

@ddstreet: Can we get the rdep analysis now, rather than later? I think that's critical to understanding if we should or shouldn't accept the node upload and go through with this madness.

Dan Streetman (ddstreet) wrote :

$ lsb_release -c
Codename: bionic

$ apt download $( reverse-depends -r bionic -l nodejs )
...

$ ls *.deb | wc -l
1296

$ rm *all.deb
$ ls *.deb | wc -l
20

So of the 1296 debs that Depends: nodejs, only 20 of them actually create binary debs:

$ ls -1
codelite-plugins_10.0+dfsg-2_amd64.deb
emscripten_1.22.1-1build1_amd64.deb
libkf5purpose-bin_5.44.0-0ubuntu1_amd64.deb
netdata_1.9.0+dfsg-1_amd64.deb
node-almond_0.3.3+dfsg-1_amd64.deb
node-groove_2.5.0-2ubuntu1_amd64.deb
node-iconv_2.3.0-3build1_amd64.deb
nodejs-dev_8.10.0~dfsg-2_amd64.deb
node-jstimezonedetect_1.0.6-1_amd64.deb
node-leveldown_1.5.0+dfsg-1ubuntu1_amd64.deb
node-mapnik_3.7.1+dfsg-3_amd64.deb
node-node-expat_2.3.15-4build2_amd64.deb
node-node-stringprep_0.8.0-3ubuntu1_amd64.deb
node-sqlite3_3.1.8+ds1-2build2_amd64.deb
node-srs_0.4.8+dfsg-3ubuntu1_amd64.deb
node-websocket_1.0.23-3build2_amd64.deb
node-ws_1.1.0+ds1.e6ddaae4-3ubuntu1_amd64.deb
node-xmlhttprequest_1.6.0-1_amd64.deb
node-zipfile_0.5.11+ds-2ubuntu1_amd64.deb
qtwebchannel5-examples_5.9.5-0ubuntu1_amd64.deb

Dan Streetman (ddstreet) wrote :
Download full text (5.2 KiB)

$ for d in *.deb ; do dpkg -x $d ${d%_amd64.deb} ; done

$ for f in $( find * -type f ) ; do file "$f" | grep ELF | cut -d ':' -f 1 ; done | cut -d '/' -f 1 | sort | uniq
codelite-plugins_10.0+dfsg-2
libkf5purpose-bin_5.44.0-0ubuntu1
netdata_1.9.0+dfsg-1
node-groove_2.5.0-2ubuntu1
node-iconv_2.3.0-3build1
node-leveldown_1.5.0+dfsg-1ubuntu1
node-mapnik_3.7.1+dfsg-3
node-node-expat_2.3.15-4build2
node-node-stringprep_0.8.0-3ubuntu1
node-sqlite3_3.1.8+ds1-2build2
node-srs_0.4.8+dfsg-3ubuntu1
node-websocket_1.0.23-3build2
node-ws_1.1.0+ds1.e6ddaae4-3ubuntu1
node-zipfile_0.5.11+ds-2ubuntu1
qtwebchannel5-examples_5.9.5-0ubuntu1

$ for f in $( find * -type f ) ; do file "$f" | grep ELF | cut -d ':' -f 1 ; done
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/codelite-lldb
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/Outline.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/WebTools.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/Copyright.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/EditorConfigPlugin.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/UnitTestsPP.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/CallGraph.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/Wizards.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/SnipWiz.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/git.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/SpellCheck.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/cscope.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/DatabaseExplorer.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/wxFormBuilder.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/SFTP.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/QMakePlugin.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/WordCompletion.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/Tail.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/MemCheck.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/CodeFormatter.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/ContinuousBuild.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/Subversion.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/codelitephp.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/HelpPlugin.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/abbreviation.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/LLDBDebugger.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/CodeLiteDiff.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/cppchecker.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/ExternalTools.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/Tweaks.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/ZoomNavigator.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/AutoSave.so
codelite-plugins_10.0+dfsg-2/usr/lib/codelite/CMakePlugin.so
libkf5purpose-bin_5.44.0-0ubuntu1/usr/lib/x86_64-linux-gnu/libReviewboardHelpers.so.5.44.0
libkf5purpose-bin_5.44.0-0ubuntu1/usr/lib/x86_64-linux-gnu/libPhabricatorHelpers.so.5.44.0
libkf5purpose-bin_5.44.0-0ubuntu1/usr/lib/x86_64-linux-gnu/libexec/kf5/purposeprocess
libkf5purpose-bin_5.44.0-0ubuntu1/usr/lib/x86_64-linux-gnu/qt5/plugins/kf5/purpose/youtubeplugin.so
libkf5purpose-bin_5.44.0-0ubuntu1/usr/lib/x86_64-linux-gnu/qt5/plugins/kf5/purpose/imgurplugin.so
libkf5purpose-bin_5.44.0-0ubuntu1/usr/l...

Read more...

Dan Streetman (ddstreet) wrote :

$ for f in $( find * -type f ) ; do file "$f" | grep ELF | cut -d ':' -f 1 ; done > file-list

Assuming that the openssl init symbol will be included:

$ for f in $( cat file-list ) ; do objdump -T "$f" | grep OPENSSL_init_ssl ; done
$ for f in $( cat file-list ) ; do objdump -T "$f" | grep SSL_library_init ; done

And for good measure:

$ for f in $( cat file-list ) ; do objdump -T "$f" | grep SSL ; done
$ for f in $( cat file-list ) ; do objdump -T "$f" | grep ssl ; done

Looks like none of the nodejs reverse-deps provided by Ubuntu actually link against openssl, so I think we should be ok with this change; it should not affect Ubuntu-provided nodejs packages, unless I'm missing something.

Robie Basak (racb) wrote :

If the SRU team decides to make this change, I think it's worth making it clear to the wider community whether we're just making this change on this occasion because everything has aligned to make it possible easily in this specific case (openssl 1.0 remains in Bionic, etc), or if this ABI stability promise is actually one we realistically expect to keep in the future, or if we're deferring that decision until the wider community has figured out how to handle this situation in distributions. Just because we happen to be able to do it this time due to the versions available doesn't necessarily mean that we'll be able to do it again.

Steve Langasek (vorlon) wrote :

From the regression potential analysis:

> Any external nodejs addons that were built specifically for the
> current Bionic nodejs will also start failing, until rebuilt against
> the upstream nodejs 8.10 version, or the new Bionic nodejs package.

I consider this a blocker for SRU. We do not guarantee ABI compatibility with third-party builds of the packages we ship; but we DO guarantee ABI compatibility with older versions of the same package as published in Ubuntu stable releases. Making this change to nodejs to cause it to build against openssl 1.0 instead of openssl 1.1 means that any user who has already built addons against the current Ubuntu nodejs ABI in Ubuntu 18.04 will have those addons break on upgrade to the new version of nodejs. That is not an acceptable experience in an SRU ("I installed this SRU and the software I had deployed on nodejs has ceased to work with symbol errors"). While it's regrettable that the nodejs ABI in 18.04 is incompatible with upstream, a regression of this magnitude makes this a non-starter for SRU.

Changed in nodejs (Ubuntu Bionic):
status: In Progress → Won't Fix

An upload of nodejs to bionic-proposed has been rejected from the upload queue for the following reason: "known to introduce a regression in nodejs ABI vs. 18.04 GA".

Steve Langasek (vorlon) wrote :

(I think this should also be wontfix for the devel series since we should not make bionic and cosmic incompatible with one another, until such time as nodejs is updated to a later version that does openssl 1.1 upstream; but I'll let someone else handle that bug task.)

Dan Streetman (ddstreet) wrote :

> I think this should also be wontfix for the devel series

cosmic is currently nodejs 8.11, and debian still has nodejs 10.4 in -experiemental; @kapouer indicated he hopes to promote nodejs 10.4 to -unstable at some point (soon) but it's unclear if that will be in time to make it into cosmic (probably, unlikely).

I'll leave this bug as inprogress for cosmic, unless anyone else disagrees and wants to change it to wontfix, since there is still a possibility (but of course no promise at all) that debian will promote nodejs 10.4 in time to pull that into cosmic.

Changed in nodejs (Ubuntu Bionic):
assignee: Dan Streetman (ddstreet) → nobody
Changed in nodejs (Ubuntu Cosmic):
assignee: Dan Streetman (ddstreet) → nobody
Nicolas Noble (grumpycoder) wrote :

I'm sorry, but I must heavily disagree with the assessment here, and I am asking you to reconsider your "won't fix". No breakage is possible since there can't be existing native modules built manually by people due to this other bug, which is the corollary of this present one: https://github.com/nodejs/node-gyp/issues/1415 - node-gyp may be able to fix this in the future somehow and detect and use the system's headers in the case of a mismatch, but it's currently not the case.

Basically, node's build system is exposing OpenSSL 1.0 headers to native addons at build time, which are themselves API compatible with the prebuilt OpenSSL 1.1 library. Therefore, it's impossible to currently build any native addon manually on Ubuntu's nodejs that depend on OpenSSL.

In other words, you can't break existing native modules built by people, because people can't build native modules *due to this bug*. Your rejection is based on an uroboros where you don't want to fix the problem because you're afraid of breaking people that do not exist due to the problem you're rejecting to fix.

If you prefer, I can file a new bug to talk about the fact that you can't currently build native addons that depend on OpenSSL due to mismatching building headers from nodejs' build system and the API you're exposing, but this really is the same resolution than this current bug here.

Robie Basak (racb) wrote :

> In other words, you can't break existing native modules built by people, because people can't build native modules *due to this bug*.

Thank you for explaining that. It's not currently what the bug description currently says. Please could those interested in this bug work on getting that accurate?

Please see https://irclogs.ubuntu.com/2018/08/03/%23ubuntu-devel.html#t16:44 for a further question on this.

Steve Langasek (vorlon) wrote :

Nicolas, thanks for the clarification. Just to be sure, is node-gyp the *only* way to build binary add-ons for node, or is it possible that some users have built add-ons using some other build system?

While it's theorically possible to bypass node-gyp, I'm not aware of any
actively used method that would do so. The node-gyp is a bundled dependency
of npm, and is the code that parses the native module information (the
bindings.gyp file that is), in order to generate the Makefile that will
build the module. This Makefile will contain all the appropriate hard-coded
information pertaining to the version of the runtime this module needs to
be build against, including the path to the openssl headers that node-gyp
just downloaded for the runtime in its local cache folder.

On Fri, Aug 3, 2018, 20:01 Steve Langasek <email address hidden>
wrote:

> Nicolas, thanks for the clarification. Just to be sure, is node-gyp the
> *only* way to build binary add-ons for node, or is it possible that some
> users have built add-ons using some other build system?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1779863
>
> Title:
> Ubuntu nodejs package isn't ABI compatible with mainline nodejs.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/nodejs/+bug/1779863/+subscriptions
>

Elana Hashman (ehashman) wrote :

I also disagree with the assessment of "wontfix".

The ABI incompatibility with upstream is not just "regrettable", but an actual bug. It's not supported behaviour; it's an undocumented ABI deviation, and as soon as upstream became aware of it, they filed an issue. The SRU policy states that it covers:

> Bugs which represent severe regressions from the previous release of Ubuntu.

Breaking the ABI contract that the rest of the NodeJS ecosystem relies on when the previous LTS did not represents a severe regression, IMO.

Debian (and by proxy Ubuntu) did not make clear to end users that we were not honouring upstream ABI compatibility with this package when we chose to link against an incompatible OpenSSL. Users typically have a default expectation of compatibility unless clearly stated otherwise. Python wheels are compatible with distro Pythons; why wouldn't compiled node extensions be compatible with distro node? In my experience, many users don't even know there's a difference between the upstream and apt versions!

Because ABI incompatibility is subtle and difficult for end users to diagnose, it is even more important that we get this addressed in Ubuntu rather than expecting users to work around us. The SRU policy recognizes this: "Users of the official release ... are less experienced with Ubuntu and with Linux, and expect a reliable system which does not require their intervention." In this case, users would need to uninstall the distro nodejs and figure out how to use upstream instead to work around the incompatibility, which constitutes a lot of extra friction.

The majority use case is "I want to `apt-get install node` and `npm install` my deps", and I don't know why we should break those users in favour of ABI compat for the people building their native dependencies against the Ubuntu node ABI, if such users even exist. nodejs/node-gyp#1415 seems to suggest they don't. The fact that no bugs have been filed against Debian or Ubuntu about native dependencies failing to compile due to said bug is stronger evidence that users aren't doing this.

I think the downside is much greater in not patching.

At minimum, we have a responsibility to make sure end users are aware of this issue. Can we make sure to include a NEWS.Debian file? Are there other appropriate channels?

Robie Basak (racb) wrote :
Download full text (4.0 KiB)

On Sat, Aug 04, 2018 at 04:44:05PM -0000, Elana Hashman wrote:
> The ABI incompatibility with upstream is not just "regrettable", but an
> actual bug. It's not supported behaviour; it's an undocumented ABI
> deviation, and as soon as upstream became aware of it, they filed an
> issue.

The problem is wider than this. Distributions aren't set up to maintain
binary compatibility in the way that you expect.

I appreciate the difficulties that this causes the ecosystem. However,
binary compatibility with elements not shipped with the distribution is
not a guarantee that binary distributions (like Debian, Ubuntu and
others) have traditionally ever made.

*Declaring* binary compatibility/incompatibility correctly is a separate
matter and is possibly a bug here. For example: if I grab a binary built
elsewhere and try to run it on my system, I may receive an error telling
me that the linker cannot find "libc.so.5". It would correctly be
reporting that binary dependencies that are expected to be provided by
my system are not present. The equivalent should be happening with
Node.js and third party modules affected by this issue, and if it isn't,
that's a bug. This bug report says "will fail to load" though, so
perhaps binary compatibility checks are working as expected.

In this specific case it looks like it is possible to rebuild nodejs
against openssl1.0 to resolve the problem for now. However, this is only
possible because openssl1.0 happens to also be packaged in Bionic. This
is a lucky coincidence. Had it been removed by the time of Bionic's
release, we'd be far more stuck than we are now. I want to make sure
that everybody involved understands this, because this demonstrates how
the current expectations of the Node.js ecosystem mismatches the way
distributions actually work.

In the general case, distributions, in a particular release, may well
not package the particular versions of dependencies that you declare to
be part of your ABI. They may use other versions because it minimises
distribution maintenance work to only package one version, and
dependencies are used by other projects, too.

It has been suggested that this would be the responsibility of
distribution nodejs maintainers to resolve. However, in the general case
it cannot be resolved. Other packaged projects may require a newer
version of the dependency. That Node.js considers the dependency part of
its "official ABI" is currently neither a reason to hold those projects
back, nor a reason to package multiple versions of dependencies just for
nodejs' sake. Distribution maintainers will patch nodejs as required to
get it building against the newer version of the dependency so that the
distribution can ship on schedule. That this might break binaries built
outside the distribution has never been considered within the scope of
general distribution development workflow, except to the extent that
distribution binaries correctly declare their own ABIs and that
diverging on ABI in a distribution-specific way that is not upstream (in
this case upstream means OpenSSL, not Node.js) is generally undesirable.

I hope this explains the actual problem: it's not as simple as just
expecting/requirin...

Read more...

Nicolas Noble (grumpycoder) wrote :

I must specify something from your statements. The fact the native module doesn't load really is undesired behavior. The normal nodejs behavior is make sure that the module is going to load while installing it, by checking ABI compatibility using various tags. It is expected and supported that nodejs reports its runtime version, and other parameters such as platform it's compiled for, to the external world.

A very typical installation path would then be for example:

    "module_path": "src/node/extension_binary/{node_abi}-{platform}-{arch}-{libc}"

And if a pre-compiled binary doesn't exist there, the installation system will trigger a compilation in order to generate a module that fits. For node8 for instance, {node_abi} will resolve to "node-v57".

The issue here is that you're transparently masquerading as "nodejs" while not really being "nodejs". For instance, Electron, which is a nodejs-like environment, but has a different ABI, will report the {node_abi} parameter completely differently. Instead of reporting "node-v57" as above, the Electron 2 runtime which is also based on node8, will report "electron-v2.0". This makes our lives as npm package maintainers easy, since we can then select the proper binary based on it.

So one another solution I'd then see would be for you to bite the bullet, and stop calling your runtime "nodejs", because, well, it's not really nodejs. More specifically, your "node8" runtime should report "node-debian-v57" instead of "node-v57", so we, npm package maintainers, can do the right thing on our end and either (1) start distributing node-debian specific prebuilt binaries or (2) let the npm install realize this is a different runtime than the prebuilt binaries we have and let it just recompile for this different runtime.

But your obstination on releasing a nodejs runtime that's not really nodejs while 100% masquerading as the official nodejs will eventually force us to discourage our users to use your runtime, because there's nothing we could do to handle the subtle bugs you're introducing on us.

Robie Basak (racb) wrote :

On Tue, Aug 07, 2018 at 03:59:27AM -0000, Nicolas Noble wrote:
> So one another solution I'd then see would be for you to bite the
> bullet, and stop calling your runtime "nodejs", because, well, it's not
> really nodejs.

Sure. This falls under correctly "declaring binary compatibility" in my
analysis.

When it comes to this kind of thing, we rely on a single arbitrator to
define how to do this exactly so that all distributions, as well as
binaries built directly from upstream sources, correctly interact and
agree on what constitutes compatibility. Upstream works best to act as
the arbitrator since you're unilaterally in a position to define what
constitutes binary compatibility. Downstreams can't really do that. I
suggest, then, that you define exactly how to do this correctly, and
incorporate this mechanism into your build system (or at least
officially document it) so that all distributions do it the right way.

For example: it's not really "node-debian-v57" either; it's
"node-openssl1.1-v57". Ubuntu, Debian and all other distributions that
might release with nodejs linked to OpenSSL 1.1 would all share binary
compatibility, AIUI, so it would be overkill to force them to lose
binary compatibility between each other just because this compatibility
is ill defined. The build system should correctly arrange for the report
to be correct based on how it was built. If this were to happen, all
distributions would simply use it, and you'd get your "declare your
binary compatibility correctly" wish granted by default.

> But your obstination on releasing a nodejs runtime that's not really
> nodejs while 100% masquerading as the official nodejs will eventually
> force us to discourage our users to use your runtime, because there's
> nothing we could do to handle the subtle bugs you're introducing on us.

I suspect what will happen here is that we'll end up rebuilding nodejs
against 1.0 in 18.04. But this won't solve the problem for next time.

From your wording I don't think you've understood the distribution
ecosystem and why what you are doing and your expectations are a problem
for distributions to be able to meet in the general case. Distribution
users *expect* package dependencies to be de-duplicated, and to receive
security support for those dependencies from a single source. This is
the reason that "apt-get install ..." Just Works. You rely on this too:
for everything on your system that works without you caring for them
specifically.

Let's try not split our users into two factions, and instead figure out
how to solve this problem well for everyone. But I think to begin to do
that you first need to understand why distributions work the way they
do. I'm happy to spend more time with you on this. Please feel free to
ping me on IRC for a more interactive explanation of this, or to arrange
some other medium.

Nicolas Noble (grumpycoder) wrote :
Download full text (3.7 KiB)

My current schedule will make IRC being difficult, sorry.

The problem is that users are *already* split into two factions. They
expect apt-get to work, but they also expect npm install to work too. The
former is fine on your end, because you control the whole ecosystem here,
but it's perfectly reasonable for people to also expect that doing npm
install just works too, and right now that's not the case.

On Tue, Aug 7, 2018 at 10:51 AM, Robie Basak <email address hidden>
wrote:

> On Tue, Aug 07, 2018 at 03:59:27AM -0000, Nicolas Noble wrote:
> > So one another solution I'd then see would be for you to bite the
> > bullet, and stop calling your runtime "nodejs", because, well, it's not
> > really nodejs.
>
> Sure. This falls under correctly "declaring binary compatibility" in my
> analysis.
>
> When it comes to this kind of thing, we rely on a single arbitrator to
> define how to do this exactly so that all distributions, as well as
> binaries built directly from upstream sources, correctly interact and
> agree on what constitutes compatibility. Upstream works best to act as
> the arbitrator since you're unilaterally in a position to define what
> constitutes binary compatibility. Downstreams can't really do that. I
> suggest, then, that you define exactly how to do this correctly, and
> incorporate this mechanism into your build system (or at least
> officially document it) so that all distributions do it the right way.
>
> For example: it's not really "node-debian-v57" either; it's
> "node-openssl1.1-v57". Ubuntu, Debian and all other distributions that
> might release with nodejs linked to OpenSSL 1.1 would all share binary
> compatibility, AIUI, so it would be overkill to force them to lose
> binary compatibility between each other just because this compatibility
> is ill defined. The build system should correctly arrange for the report
> to be correct based on how it was built. If this were to happen, all
> distributions would simply use it, and you'd get your "declare your
> binary compatibility correctly" wish granted by default.
>
> > But your obstination on releasing a nodejs runtime that's not really
> > nodejs while 100% masquerading as the official nodejs will eventually
> > force us to discourage our users to use your runtime, because there's
> > nothing we could do to handle the subtle bugs you're introducing on us.
>
> I suspect what will happen here is that we'll end up rebuilding nodejs
> against 1.0 in 18.04. But this won't solve the problem for next time.
>
> >From your wording I don't think you've understood the distribution
> ecosystem and why what you are doing and your expectations are a problem
> for distributions to be able to meet in the general case. Distribution
> users *expect* package dependencies to be de-duplicated, and to receive
> security support for those dependencies from a single source. This is
> the reason that "apt-get install ..." Just Works. You rely on this too:
> for everything on your system that works without you caring for them
> specifically.
>
> Let's try not split our users into two factions, and instead figure out
> how to solve this problem well for everyone. But I think to begin to d...

Read more...

Nicolas Noble (grumpycoder) wrote :
Download full text (4.4 KiB)

Also, I am but a simple npm package maintainer whose users are currently
reporting weird issues when using Ubuntu packages. I did the thorough
investigation of what the actual issue is, but I am not a nodejs developer.
I have strictly no power whatsoever in resolving this issue one way or
another. And I personally don't really care what the actual solution is, as
long as my users stop reporting that doing npm install of my package fails
for them on Ubuntu. Please do engage with the actual nodejs developers in
order to resolve this.

On Tue, Aug 7, 2018 at 11:20 AM, Nicolas Noble <email address hidden>
wrote:

> My current schedule will make IRC being difficult, sorry.
>
> The problem is that users are *already* split into two factions. They
> expect apt-get to work, but they also expect npm install to work too. The
> former is fine on your end, because you control the whole ecosystem here,
> but it's perfectly reasonable for people to also expect that doing npm
> install just works too, and right now that's not the case.
>
> On Tue, Aug 7, 2018 at 10:51 AM, Robie Basak <email address hidden>
> wrote:
>
>> On Tue, Aug 07, 2018 at 03:59:27AM -0000, Nicolas Noble wrote:
>> > So one another solution I'd then see would be for you to bite the
>> > bullet, and stop calling your runtime "nodejs", because, well, it's not
>> > really nodejs.
>>
>> Sure. This falls under correctly "declaring binary compatibility" in my
>> analysis.
>>
>> When it comes to this kind of thing, we rely on a single arbitrator to
>> define how to do this exactly so that all distributions, as well as
>> binaries built directly from upstream sources, correctly interact and
>> agree on what constitutes compatibility. Upstream works best to act as
>> the arbitrator since you're unilaterally in a position to define what
>> constitutes binary compatibility. Downstreams can't really do that. I
>> suggest, then, that you define exactly how to do this correctly, and
>> incorporate this mechanism into your build system (or at least
>> officially document it) so that all distributions do it the right way.
>>
>> For example: it's not really "node-debian-v57" either; it's
>> "node-openssl1.1-v57". Ubuntu, Debian and all other distributions that
>> might release with nodejs linked to OpenSSL 1.1 would all share binary
>> compatibility, AIUI, so it would be overkill to force them to lose
>> binary compatibility between each other just because this compatibility
>> is ill defined. The build system should correctly arrange for the report
>> to be correct based on how it was built. If this were to happen, all
>> distributions would simply use it, and you'd get your "declare your
>> binary compatibility correctly" wish granted by default.
>>
>> > But your obstination on releasing a nodejs runtime that's not really
>> > nodejs while 100% masquerading as the official nodejs will eventually
>> > force us to discourage our users to use your runtime, because there's
>> > nothing we could do to handle the subtle bugs you're introducing on us.
>>
>> I suspect what will happen here is that we'll end up rebuilding nodejs
>> against 1.0 in 18.04. But this won't solve the problem for next time.
>>...

Read more...

Steve Langasek (vorlon) on 2018-08-07
description: updated
Steve Langasek (vorlon) wrote :

Nicolas, I accept your correction regarding the risk of binary incompatibility with locally-built binaries referencing the Ubuntu 18.04 nodejs ABI. I have clarified the 'regression potential' in the bug description, and am willing to accept the SRU given my current understanding; however, the bug is missing a clear test case. The bug description currently points to https://github.com/nicolasnoble/openssl-nodejs-ubuntu-demo, but there is no documentation there, so you have to know how to build a node project to reproduce the bug. Can you give step-by-step directions here for what to run from the commandline?

Changed in nodejs (Ubuntu Bionic):
status: Won't Fix → Incomplete
Seth Arnold (seth-arnold) wrote :

On Tue, Aug 07, 2018 at 05:51:56PM -0000, Robie Basak wrote:
> For example: it's not really "node-debian-v57" either; it's
> "node-openssl1.1-v57". Ubuntu, Debian and all other distributions that

You'd probably also like this to enumerate all libraries that node
re-exports, if you choose to go this route.

Thanks

Nicolas Noble (grumpycoder) wrote :

Apologies for the late reply.

My github repository is a quick effort in trying to expose the ABI problem,
but a more thorough (and straightforward) way to actually reproduce would
be to do the following:

First case, when using a module that ships with prebuilt binaries. With the
"nodejs" and "npm" Ubuntu packages installed, in a directory somewhere,
first do `npm install grpc`. Notice how it's mentioning installing a
prebuilt binary from remote. Next, run node, and enter the following code:
> const grpc = require('grpc')
> const creds = grpc.ServerCredentials.createSsl(null, [])
> const server = new grpc.Server()
> server.bind('0.0.0.0:8080', creds)

The last line will result in a crash of the nodejs runtime, that says
something like
node: symbol lookup error:
/tmp/ubuntu-crash/node_modules/grpc/src/node/extension_binary/node-v57-linux-x64-glibc/grpc_node.node:
undefined symbol: SSL_library_init

Second case, when using a module that only ships source code. With the
"nodejs" and "npm" Ubuntu packages installed, try the command `npm install
node-webcrypto-ossl`. Compilation will start, but will fail due to header
expectation mismatch.

If you retry the same steps using any of the official nodejs runtime
instead of the Ubuntu package, the first case will not crash the runtime
(it'll fail because we're not providing valid SSL certificates, but that's
normal behavior), and the second case will compile properly at installation
time.

On Wed, Aug 8, 2018 at 4:56 PM, Seth Arnold <email address hidden>
wrote:

> On Tue, Aug 07, 2018 at 05:51:56PM -0000, Robie Basak wrote:
> > For example: it's not really "node-debian-v57" either; it's
> > "node-openssl1.1-v57". Ubuntu, Debian and all other distributions that
>
> You'd probably also like this to enumerate all libraries that node
> re-exports, if you choose to go this route.
>
> Thanks
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1779863
>
> Title:
> Ubuntu nodejs package isn't ABI compatible with mainline nodejs.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/nodejs/+bug/
> 1779863/+subscriptions
>

Steve Langasek (vorlon) wrote :

Hello Nicolas, or anyone else affected,

Accepted nodejs into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nodejs/8.10.0~dfsg-2ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

description: updated
Changed in nodejs (Ubuntu Bionic):
status: Incomplete → Fix Committed
tags: added: verification-needed verification-needed-bionic
Elana Hashman (ehashman) wrote :

Hi Steve,

I have completed the first half of the testing process and confirmed the bug. (Transcript attached.)

However, nodejs is not yet available in bionic-proposed as it seems the build failed. Please let me know when a successfully built package is available to test.

Cheers,

- e

Dan Streetman (ddstreet) wrote :

I'll take a look at the FTBFS for this today - it didn't fail when built in my ppa so not sure what happened yet.

Dan Streetman (ddstreet) on 2018-08-14
Changed in nodejs (Ubuntu Bionic):
assignee: nobody → Dan Streetman (ddstreet)
Changed in nodejs (Ubuntu Cosmic):
assignee: nobody → Dan Streetman (ddstreet)
Dan Streetman (ddstreet) wrote :

The failure is due to unfortunate timing of one of the test cert's crl expiry (Next Update date):

$ openssl crl -inform PEM -text -noout -in test/fixtures/keys/ca2-crl.pem
Certificate Revocation List (CRL):
        Version 1 (0x0)
    Signature Algorithm: sha512WithRSAEncryption
        Issuer: /<email address hidden>
        Last Update: Nov 12 21:31:47 2015 GMT
        Next Update: Aug 7 21:31:47 2018 GMT

The security team is releasing a nodejs update with new certs, so I'll rebase and reupload this patch after that is released.

Dan Streetman (ddstreet) wrote :

Hello Nicolas, or anyone else affected,

Accepted nodejs into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nodejs/8.10.0~dfsg-2ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Dan Streetman (ddstreet) wrote :

re: cosmic debdiff, debian appears to have added code to 8.11 version that breaks the build with openssl 1.0.2, so i'll need to look into that.

@grumpycoder, or anyone else, can you confirm that nodejs version 8.11 still "requires" openssl 1.0?

The official list of library dependency versions can be found here:
https://nodejs.org/dist/index.json

8.11.4 is listed as linked against OpenSSL 1.0.2p.

On Fri, Aug 31, 2018 at 12:37 PM, Dan Streetman <<email address hidden>
> wrote:

> re: cosmic debdiff, debian appears to have added code to 8.11 version
> that breaks the build with openssl 1.0.2, so i'll need to look into
> that.
>
> @grumpycoder, or anyone else, can you confirm that nodejs version 8.11
> still "requires" openssl 1.0?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1779863
>
> Title:
> Ubuntu nodejs package isn't ABI compatible with mainline nodejs.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/nodejs/+bug/
> 1779863/+subscriptions
>

Elana Hashman (ehashman) wrote :

I have tested the steps with nodejs and nodejs-dev in bionic-proposed, however I get a different error. I am guessing that there is something wrong with the credential initializing:

ubuntu@ubuntu-bionic:~$ node
> const grpc = require('grpc')
undefined
> const creds = grpc.ServerCredentials.createSsl(null, [])
undefined
> const server = new grpc.Server()
undefined
> server.bind('0.0.0.0:8080', creds)
E0908 19:25:28.193325796 27119 security_connector.cc:1149] Handshaker factory creation failed with TSI_INVALID_ARGUMENT.
E0908 19:25:28.193531437 27119 server_secure_chttp2.cc:84] {"created":"@1536434728.193508636","description":"Unable to create secure server with credentials of type Ssl.","file":"../deps/grpc/src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.cc","file_line":63,"security_status":1}
0

I was able to successfully `npm install node-webcrypto-ossl` this time.

Nicolas Noble (grumpycoder) wrote :

Yes - this is expected behavior for not passing down proper SSL
certificates to the constructor.

On Sat, Sep 8, 2018 at 12:27 PM, Elana Hashman <email address hidden> wrote:

> I have tested the steps with nodejs and nodejs-dev in bionic-proposed,
> however I get a different error. I am guessing that there is something
> wrong with the credential initializing:
>
> ubuntu@ubuntu-bionic:~$ node
> > const grpc = require('grpc')
> undefined
> > const creds = grpc.ServerCredentials.createSsl(null, [])
> undefined
> > const server = new grpc.Server()
> undefined
> > server.bind('0.0.0.0:8080', creds)
> E0908 19:25:28.193325796 27119 security_connector.cc:1149] Handshaker
> factory creation failed with TSI_INVALID_ARGUMENT.
> E0908 19:25:28.193531437 27119 server_secure_chttp2.cc:84]
> {"created":"@1536434728.193508636","description":"Unable to create secure
> server with credentials of type Ssl.","file":"../deps/grpc/
> src/core/ext/transport/chttp2/server/secure/server_secure_
> chttp2.cc","file_line":63,"security_status":1}
> 0
>
>
> I was able to successfully `npm install node-webcrypto-ossl` this time.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1779863
>
> Title:
> Ubuntu nodejs package isn't ABI compatible with mainline nodejs.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/nodejs/+bug/
> 1779863/+subscriptions
>

Patch for Cosmic nodejs to build with openssl 1.0.2
and fix test-case regression seen on libuv 1.22.0-3
(which is unrelated to the openssl version change.)

This built successfully on LP PPA; build log [1].

On previous builds there was one test-case timeout
on parallel/test-net-listen-after-destroying-stdin
which happened only on amd64 (not i386), but seems
to be resolved now or be intermittent.

[1] https://launchpadlibrarian.net/388245545/buildlog_ubuntu-cosmic-amd64.nodejs_8.11.2~dfsg-1ubuntu2_BUILDING.txt.gz

Steve Langasek (vorlon) wrote :

uploaded to cosmic.

Changed in nodejs (Ubuntu Cosmic):
status: In Progress → Fix Committed

Good news as of this morning.

The nodejs in cosmic-proposed now builds correctly on all architectures,
as s390x now passes due to the libuv1 fix uploaded to cosmic-proposed yesterday (LP #1792647).

Now there's the autopkgtest failures to look at:
- First, the new libuv1 failing in cosmic-proposed only for armhf (related to udp/tcp bind(), so apparently unrelated)
- Second, the nodejs reverse deps for cosmic-proposed (reproduced some of them yesterday, and they were intermittent)
- Third, likewise, for bionic-proposed.

> - First, the new libuv1 failing in cosmic-proposed only for armhf (related to udp/tcp bind(), so apparently unrelated)

Confirmed by asking for a re-test of cosmic-release version, on which the same tests fail as in cosmic-proposed.
The failures will be marked as badtests and the libuv1 package should migrate.

Today looked more at the autopkgtest failures with ddstreet.

> - Second, the nodejs reverse deps for cosmic-proposed (reproduced some of them yesterday, and they were intermittent)

Several intermittent. Dan triggered retests for many of them. Checking tomorrow.

node-gulp had a permanent issue, unrelated to this change, but submitted fixes in LP 1793226 and BTS 909151, asked Steve for review/sponsorship.

Continues tomorrow.

> - Third, likewise, for bionic-proposed.

Those seemed all benign, intermittent errors (no longer show up in the pending-sru page [1] after retesting)
except for node-tap, which is unrelated to this change (happens w/ version in cosmic-release as well).

All good.

[1] https://people.canonical.com/~ubuntu-archive/pending-sru.html

> - Second, the nodejs reverse deps for cosmic-proposed

From yesterday -- node-gulp autopkgtests are now green on all archs.

Today -- submitted 2 more fixes
- node-mime-types on LP 1793367 (applied fix from Debian)
- node-unicode-data on LP 1793392 (submitted fix to Debian, BTS 909222)

These last 2 should clear the remaining 'Regression' status on all archs,
and we should be good to go on cosmic-proposed from a point of view of the
nodejs reverse deps / autopkgtests (see the update-excuses page [1]).

Asked Steve for review/sponsorship on them.

[1] https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nodejs - 8.11.2~dfsg-1ubuntu2

---------------
nodejs (8.11.2~dfsg-1ubuntu2) cosmic; urgency=medium

  [ Dan Streetman ]
  * Force dependency on openssl1.0
    The newer openssl has ABI change, and this version of nodejs
    requires building with the older 1.0.2 version of openssl.
    (LP: #1779863)

  [ Mauricio Faria de Oliveira ]
  * d/p/revert-26388feb760c42d5a342a08507f13a3a268918da.patch:
    Fix build errors with openssl 1.0.2 due to patch for openssl 1.1.
    (LP: #1779863)
  * d/p/cd2b80c1f57faa349232276e113fe09be5da1106.patch: Fix test-case
    failure (parallel/test-child-process-stdout-ipc) with libuv 1.21+.

 -- Mauricio Faria de Oliveira <email address hidden> Wed, 12 Sep 2018 16:29:53 -0300

Changed in nodejs (Ubuntu Cosmic):
status: Fix Committed → Fix Released

The 2 fixes from yesterday were uploaded, and resolved the pending test-case failures for nodejs rdeps on Cosmic, which allowed the migration from cosmic-proposed to cosmic-release.

Now nodejs w/ OpenSSL 1.0 is in Cosmic (version 8.11.2~dfsg-1ubuntu2).

Verification for bionic-proposed is successful.

Changing verification-needed{-bionic} tags to done.

The last autopkgtest failure in nodejs rdeps for bionic
(node-tap) had a patch submitted in LP #1793612, and
since it's a test-server problem, not code/runtime,
Steve kindly marked it as non-blocking for this SRU.

Procedure:
=========

Enabled 'bionic-proposed universe' in apt sources.list.

$ sudo apt install nodejs npm

$ dpkg -s nodejs npm | grep Version
Version: 8.10.0~dfsg-2ubuntu0.3
Version: 3.5.2-0ubuntu4

$ mkdir /tmp/lp.1779863
$ cd /tmp/lp.1779863

Test 1)
------

$ npm install grpc
...
[grpc] Success: "/tmp/lp.1779863/node_modules/grpc/src/node/extension_binary/node-v57-linux-x64-glibc/grpc_node.node" is installed via remote
...

$ echo $?
0

$ node
> const grpc = require('grpc')
undefined
> const creds = grpc.ServerCredentials.createSsl(null, [])
undefined
> const server = new grpc.Server()
undefined
> server.bind('0.0.0.0:8080', creds)
E0921 13:06:16.712298391 29233 security_connector.cc:1160] Handshaker factory creation failed with TSI_INVALID_ARGUMENT.
E0921 13:06:16.712493281 29233 server_secure_chttp2.cc:84] {"created":"@1537535176.712464610","description":"Unable to create secure server with credentials of type Ssl.","file":"../deps/grpc/src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.cc","file_line":63,"security_status":1}
0
<Ctrl-D>

This did not fail with symbol lookup error for SSL_library_init (reported in bug description).
It fails in an expected manner, for not passing SSL certificates to the constructor (comment #51).

Test 2)
------

> $ npm install node-webcrypto-ossl
...
make: Entering directory '/tmp/lp.1779863/node_modules/node-webcrypto-ossl/build'
...
  SOLINK_MODULE(target) Release/obj.target/nodessl.node
  COPY Release/nodessl.node
make: Leaving directory '/tmp/lp.1779863/node_modules/node-webcrypto-ossl/build'
...
└─┬ node-webcrypto-ossl@1.0.38
  ├─┬ mkdirp@0.5.1
  │ └── minimist@0.0.8
  ├── tslib@1.9.3
  └── webcrypto-core@0.1.22
...

$ echo $?
0

The build succeeds, without any compilation errors (reported in bug description).

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nodejs - 8.10.0~dfsg-2ubuntu0.3

---------------
nodejs (8.10.0~dfsg-2ubuntu0.3) bionic; urgency=medium

  * Force dependency on openssl1.0
    The newer openssl has ABI change, and this version of nodejs
    requires building with the older 1.0.2 version of openssl.
    (LP: #1779863)

 -- Dan Streetman <email address hidden> Mon, 23 Jul 2018 16:12:30 -0400

Changed in nodejs (Ubuntu Bionic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for nodejs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.