[hadoop-swiftfs] cannot read large object that uploaded by hadoop command

Bug #1428939 reported by Kazuki OIKAWA
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Sahara
Fix Released
High
Kazuki OIKAWA

Bug Description

If large object uploaded by hadoop command, cannot read that object.
Because uploaded segment object name is wrong.

$ hadoop fs -copyFromLocal ./largeFile swift://test.sahara/large
$ hadoop fs -cat swift://test.sahara/large | wc -c
0
$ swift list test
large
swift://test.sahara/large/000001
swift://test.sahara/large/000002
swift://test.sahara/large/000003
swift://test.sahara/large/000004

Kazuki OIKAWA (k.oikw)
Changed in sahara:
assignee: nobody → Kazuki OIKAWA (k.oikw)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to sahara-extra (master)

Fix proposed to branch: master
Review: https://review.openstack.org/162005

Revision history for this message
Luigi Toscano (ltoscano) wrote :

I see the the proposed fix is for sahara-extra. Does it affect the copy of hadoop-swiftfs upstream? If yes, is there a proposed patch against that version, or is there a plan to use the sahara-extra version always (if not already)?

Revision history for this message
Kazuki OIKAWA (k.oikw) wrote :

> Does it affect the copy of hadoop-swiftfs upstream?
yes

I recommend using sahara-extra version

I think we need to discuss swiftfs maintenance policy.

Currently, swiftfs of apache hadoop repository is hard to merge new patch.
Because swift understood reviewer is a few, maybe.

Revision history for this message
Luigi Toscano (ltoscano) wrote :

I understand the slowness of the review, but keeping a fork is always complicated. I would try to push the patch upstream anyway first. Maybe the maintainership of that component upstream can be extended to someone else too from "our" (Sahara) world to speed up things?

Moreover, right now the swiftfs_hadoop element from sahara-image-elements, which installs the Sahara version of the library if I'm not mistaken, is used only for vanilla and spark, so any change in swiftfs/hadoop library usage should impact that file as well.

Changed in sahara:
milestone: none → kilo-3
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to sahara-extra (master)

Reviewed: https://review.openstack.org/162005
Committed: https://git.openstack.org/cgit/openstack/sahara-extra/commit/?id=c88a30012dc4e451b9f72db2684691bb2111e7e7
Submitter: Jenkins
Branch: master

commit c88a30012dc4e451b9f72db2684691bb2111e7e7
Author: Kazuki OIKAWA <email address hidden>
Date: Fri Mar 6 13:11:42 2015 +0900

    Fix segment object naming that uploaded at qualified URL

    Change-Id: Ifdac5efe2c79a4ffed96f6fd66e58d2a263925fc
    Closes-Bug: 1428939

Changed in sahara:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in sahara:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in sahara:
milestone: kilo-3 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.