[Feature] Update the memkind(1.7) package with support for Knights Mill

[Piotr Luc]

Update the memkind package with support for Knights Mill.
The patch is under review.

Target Release: 18.10
Version: 1.7 | latest

[Piotr Luc (piotrluc)]

The version v1.5.0 contains the KNM changes.

[quanxian (quanxian-wang)]

17.04 contains v1.1.0. Any plan to update memkind to 1.5.0?

[Leann Ogasawara (leannogasawara)]

We should look to get this updated for 17.10 and examine if the support is feasible to backport as an SRU for 17.04.

[Leann Ogasawara (leannogasawara)]

I've assigned a task for memkind for 17.10.

[quanxian (quanxian-wang)]

@Canonical, what is the status?

[Leann Ogasawara (leannogasawara)]

I see version 1.5.0 is currently in artful-proposed:

https://launchpad.net/ubuntu/+source/memkind/1.5.0-0ubuntu1

However it appears that it is failing to build. Eg. build log for amd64 is here:

https://launchpadlibrarian.net/336372549/buildlog_ubuntu-artful-amd64.memkind_1.5.0-0ubuntu1_BUILDING.txt.gz

Given this package is in Universe, Canonical offers no official support for this package. I suspect if you want this fixed and available in Artful, Intel will need to assist with providing patches.

[Leann Ogasawara (leannogasawara)]

Additional feedback from the Canonical Foundations Team is as follows:

A newer version of memkind appears to require a newer version of jemalloc than what is currently present in the Ubuntu archive (Debian is still at jemalloc 3.6.0 in unstable; so is Ubuntu; memkind requires jemalloc 5). The team did do the work to update the packaging, and went ahead and uploaded it knowing it would fail to build and stall in -proposed. They noted that memkind does bundle jemalloc in its sources, but it is best practice not to use bundled copies, so they did not enable the package to use this. jemalloc 5 is now available in Debian experimental; but this is an soname transition and since we are past feature freeze this is going to miss 17.10. We can sync jemalloc 5 as soon as the archive opens for 18.04 and get this resolved.

[quanxian (quanxian-wang)]

@michal

Would you provide help to Canonical for this issue? Thanks

[Michal Lewandowski (m.lewandowski)]

As far as I understand we wait now until jemalloc 5 gets into 18.04 archive. There are no plans to make memking 1.5.0 compatible with older jemalloc.

[quanxian (quanxian-wang)]

let's move it 18.04. I will use this as new request for memkind 1.6 as discussed in the Canonical/Intel meeting. Any question, let me know.

[Paweł K (pkochane)]

Since memkind 1.7 was recently released, could this request be updated to enable memkind 1.7 in 18.04?
Memkind 1.7 can be found at https://github.com/memkind/memkind/releases/tag/v1.7.0

Regarding having jemalloc sources inside memkind, we do not think that disabling such behavoiur is a good idea. Memkind is using unstable jemalloc's API, so we cannot guarentee how memkind libary would behave in case of changes of jemalloc. Also, we are testing memkind with specific jemalloc version (currently 5.0) and we cannot guarantee performance results with different version of jemalloc. Would it be possible to treat memkind as exception?

[quanxian (quanxian-wang)]

yes, I think so.

@Paul, if there is any concerns, just input your comment. Thanks

[Jeff Lane  (bladernr)]

IN Bionic at version 1.1:
https://packages.ubuntu.com/bionic/libmemkind-dev
https://packages.ubuntu.com/bionic/libmemkind-progs
https://packages.ubuntu.com/bionic/libmemkind0

jemalloc seems to be 3.6, not version 5 mentioned in this bug.
https://packages.ubuntu.com/bionic/libjemalloc1

[Paweł K (pkochane)]

What do you mean by "jemalloc seems to be 3.6, not version 5 mentioned in this bug."?
Is there any action required from memkind developers?

[Jeff Lane  (bladernr)]

Per Leann's comment #7 above:

+++
Additional feedback from the Canonical Foundations Team is as follows:

A newer version of memkind appears to require a newer version of jemalloc than what is currently present in the Ubuntu archive (Debian is still at jemalloc 3.6.0 in unstable; so is Ubuntu; memkind requires jemalloc 5). The team did do the work to update the packaging, and went ahead and uploaded it knowing it would fail to build and stall in -proposed. They noted that memkind does bundle jemalloc in its sources, but it is best practice not to use bundled copies, so they did not enable the package to use this. jemalloc 5 is now available in Debian experimental; but this is an soname transition and since we are past feature freeze this is going to miss 17.10. We can sync jemalloc 5 as soon as the archive opens for 18.04 and get this resolved.
+++

Bionic still has jemalloc 3.6.0, it appears the update to version 5 did not occur. I'll ping the Foundations team and see if there is anything we can do but as we are past FF again, I don't know that there is anything that can be done. That said, If you were to snap this instead, you could ensure the snap included any jemalloc version you need or desire. As they indicated, it's preferable to not include experimental or different versions of shared libraries in a deb package, with debs it's entirely possible that could lead to a conflict or errors with other packages looking for those libs. (I'm not saying that will happen in this case, just that it opens the door for that possibility).

But a snap is self contained and you can have any library at any version you need regardless of what other tools are using or calling. Additionally, Intel could create the snap packages and push them to the snap store at your convenience regardless of where we are in the current development cycle, and regardless of any freeze dates.

[Jeff Lane  (bladernr)]

Update. This will not make 18.04. As of today, memkind 1.7 does finally build using jemalloc 5, previous versions would now, as I understand it.

However, jemalloc has a large number of reverse depenencies in Ubuntu, and so bumping it to version 5 would require not just work for memkind, but for a whole host of other packages and it is not possible to do that work in the 3 weeks remaining of this cycle.

This could be re-visited for 18.10.

[Jeff Lane  (bladernr)]

Errr... "previous versions would NOT, as I understand it."

[Paweł K (pkochane)]

Jeff,
As I wrote in Comment 11 (https://bugs.launchpad.net/intel/+bug/1637546/comments/11), memkind has a copy of jemalloc 5.0 in its sources. This internally used jemalloc is hidden inside memkind library (its symbols are not exported outside). Memkind is also using jemallocs unstable API, which might change or even be removed in future versions of jemalloc. Changes in jemalloc could have a big impact on performance and library stability, so we decided to have its copy inside memkind library.
Generally, memkind does not require external jemalloc and it is also not providing API to internally used jemalloc.
Regards,
Paweł

[Launchpad Janitor (janitor)]

This bug was fixed in the package memkind - 1.7.0-0ubuntu1

---------------
memkind (1.7.0-0ubuntu1) bionic; urgency=medium

  * New upstream release.
  * Add versioned build-dependency on libjemalloc-dev (>= 5).
  * Refresh patch for upstream build system changes.
  * Refresh copyright.

 -- Steve Langasek <email address hidden> Mon, 02 Apr 2018 20:07:56 +0000