Sahara

[DIB] Vanilla 2 hadoop doesn't contains hadoop native library

Bug #1304995 reported by Sergey Reshetnyak
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Sahara
Fix Released
Medium
Sergey Reshetnyak
Sahara 2014.2 "juno"

Bug Description

Step to reproduce:
1. Create cluster with vanilla 2 plugin
2. Run MR job
3. Get warning:
14/04/09 10:52:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Need to add hadoop native libraries in hadoop distribution.

Tags: dib
Sergey Reshetnyak (sreshetniak)
Changed in sahara:
assignee: nobody → Sergey Reshetnyak (sreshetniak)
OpenStack Infra (hudson-openstack)
Changed in sahara:
status: New → In Progress
Revision history for this message
Matthew Farrellee (mattf) wrote :

what is the functional impact of this bug?

Revision history for this message
Sergey Reshetnyak (sreshetniak) wrote :

This needed for performance reasons.

Revision history for this message
Matthew Farrellee (mattf) wrote :

that's a good high level motivator. please provide more details.

Revision history for this message
Sergey Reshetnyak (sreshetniak) wrote :

For hadoop v1 this libraries have in deb/rpm packages. Hadoop v2 is installed from tarball and this tarball contains native libraries for only 32-bit systems. They don't work in 64-bit systems.

Revision history for this message
Sergey Lukjanov (slukjanov) wrote :

Sounds reasonable.

Changed in sahara:
milestone: none → icehouse-rc2
importance: Undecided → Medium
Sergey Lukjanov (slukjanov)
Changed in sahara:
milestone: icehouse-rc2 → juno-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to sahara-image-elements (master)

Reviewed: https://review.openstack.org/83036
Committed: https://git.openstack.org/cgit/openstack/sahara-image-elements/commit/?id=da3d4b6b95307cdca8e1a717ee93a4211eaa13ca
Submitter: Jenkins
Branch: master

commit da3d4b6b95307cdca8e1a717ee93a4211eaa13ca
Author: Sergey Reshetnyak <email address hidden>
Date: Wed Mar 26 15:21:32 2014 +0400

    Add inject hadoop native libs to vanilla hadoop 2

    Closes-bug: #1304995

    Change-Id: I25238bc29b23a132116b114416589c9a6210fea2

Changed in sahara:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in sahara:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in sahara:
milestone: juno-1 → 2014.2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to sahara-extra (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/486448

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to sahara-extra (master)

Reviewed: https://review.openstack.org/486448
Committed: https://git.openstack.org/cgit/openstack/sahara-extra/commit/?id=54ba24e1fa65ab9484532253d4bff5b18ab0feb2
Submitter: Jenkins
Branch: master

commit 54ba24e1fa65ab9484532253d4bff5b18ab0feb2
Author: Jeremy Freudberg <email address hidden>
Date: Sun Jul 23 23:56:33 2017 +0000

    Add Hadoop native libs to common-artifacts

    The Hadoop 2.7.1 native libs found on sahara-files host were actually
    the wrong thing; additionally, we are no longer using that host.

    Instructions for building these libs are found in the README of the
    "hadoop" element in SIE.

    Change-Id: I07ed1164266a2d2d2ad63ef12c8364935c9b95cc
    Partial-Bug: #1705942
    Related-Bug: #1304995

Luigi Toscano (ltoscano)
tags: added: dib
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.