'openstack object create' and 'swift upload' can not upload big (5432MB) file to container.

[Alexander Koryagin]

Hello,
Please take a look at the issue below:
    'openstack object create' and 'swift upload' can not upload big (5432MB) file to container.

Error messages:
 - Unable to establish connection to http://10.109.4.6:8080/swift/v1/TESTBUCKET01//tmp/ObjStor_somefile
 - ('Connection aborted.', error(32, 'Broken pipe'))

My env: MOS 9.0 ISO# 327.

Actions:
1) OK - Create file:
  # fallocate -l 5432M /tmp/ObjStor_somefile
  # ls -alph /tmp/ObjStor_somefile
      -rw-r--r-- 1 root root 5.4G May 18 07:41 /tmp/ObjStor_somefile

2) NOK - With a help of 'openstack' try to put file to container:
  # . openrc
  # openstack container create TESTBUCKET01 -f json
      [
        {
          "account": "v1",
          "container": "TESTBUCKET01",
          "x-trans-id": "tx00000000000000000065e-00573c1cd0-5ea5-default"
        }
      ]

  # openstack object create TESTBUCKET01 /tmp/ObjStor_somefile -f json
      >>> Unable to establish connection to http://10.109.4.6:8080/swift/v1/TESTBUCKET01//tmp/ObjStor_somefile

  # openstack object list TESTBUCKET01 --long -f json
      []

3) NOK - With a help of swift try to upload big file to container:
  # swift post TESTBUCKET02
  # swift upload TESTBUCKET02 /tmp/ObjStor_somefile
      >>> ('Connection aborted.', error(32, 'Broken pipe'))

  # swift list -l TESTBUCKET02
      {empty}

4) OK - With a help of swift with 'segment' option try to upload big file to container:
  # swift post TESTBUCKET03
  # swift upload TESTBUCKET03 -S 1024M /tmp/ObjStor_somefile
      tmp/ObjStor_somefile segment 5
      tmp/ObjStor_somefile segment 3
      tmp/ObjStor_somefile segment 1
      tmp/ObjStor_somefile segment 4
      tmp/ObjStor_somefile segment 2
      tmp/ObjStor_somefile segment 0
      tmp/ObjStor_somefile

  # swift list -l TESTBUCKET03
             0 2016-05-18 08:01:53 None tmp/ObjStor_somefile
             0

  # swift list -l TESTBUCKET03_segments
    1073741824 2016-05-18 08:01:52 None tmp/ObjStor_somefile/1463557270.178049/5695864832/1073741824/00000000
    1073741824 2016-05-18 08:01:47 None tmp/ObjStor_somefile/1463557270.178049/5695864832/1073741824/00000001
    1073741824 2016-05-18 08:01:49 None tmp/ObjStor_somefile/1463557270.178049/5695864832/1073741824/00000002
    1073741824 2016-05-18 08:01:42 None tmp/ObjStor_somefile/1463557270.178049/5695864832/1073741824/00000003
    1073741824 2016-05-18 08:01:49 None tmp/ObjStor_somefile/1463557270.178049/5695864832/1073741824/00000004
     327155712 2016-05-18 07:54:50 None tmp/ObjStor_somefile/1463557270.178049/5695864832/1073741824/00000005
    5695864832

[Roman Podoliaka (rpodolyaka)]

Swift team, please take a look.

[Alyona Kiseleva (akiselyova)]

Swift maximum file size is 5GB and if you want to upload bigger files to this storage, it has to be segmented, it's expected behavior according documentation (see http://docs.openstack.org/developer/swift/overview_large_objects.html, "Swift has a limit on the size of a single uploaded object; by default this is 5GB").
So, use swift option -S is decision.

What about "openstack object create" command, in fact I don't know, how it works inside, but if it uses something like swift upload, it could check file size and use segmented upload in case of big files. But I'm not sure, that it's supported feature in openstack cli, no info about large file support in manual to this command. It's question to somebody, more familiar with internal work of openstack cli.

[Alexander Koryagin (akoryagin)]

Hello,

So we have a clear error here:
. . .
2) NOK - With a help of 'openstack' try to put file to container:
  # . openrc
  # openstack container create TESTBUCKET01 -f json
      [
        {
          "account": "v1",
          "container": "TESTBUCKET01",
          "x-trans-id": "tx00000000000000000065e-00573c1cd0-5ea5-default"
        }
      ]

  # openstack object create TESTBUCKET01 /tmp/ObjStor_somefile -f json
      >>> Unable to establish connection to http://10.109.4.6:8080/swift/v1/TESTBUCKET01//tmp/ObjStor_somefile

  # openstack object list TESTBUCKET01 --long -f json
      []
. . .
And there is no option for segmentation in 'openstack object create'.

Why do you think that this issue is 'invalid'?

[Alyona Kiseleva (akiselyova)]

Ok, in that case it needs to be reassigned, I'm not a specialist in openstack cli code.

[Dina Belova (dbelova)]

Assigning to mos-oslo team. Please take a look and say if you're the right people to work through this.

[Dmitry Mescheryakov (dmitrymex)]

Dina, probably we are, but I would rather question importance of that issue. This is definitely not a high bug - there is a clear solution - use swift client. Moreover, this is more like a feature request for openstackclient, not a bug at all.

[Dmitry Mescheryakov (dmitrymex)]

Agreed with Dina that that is a feature request, hence moving to wishlist priority and assigning to 10.0.