OptionParser on PPC64LE does not split args correctly

[Tony Reix]

The issue appears on PPC64 LE.
It is OK on (Ubuntu) x86_64.
Version of Python is 2.7.6 on both systems.
The issue is demonstrated while running AVRO Python tests (in lang/py : ant test).

- On Ubuntu / PPC64 LE :

$ pwd
..../lang/py
$ export PYTHONPATH=./build/src
$ /home/tony/AVRO/avro-FromFreshIBMSOEGitHub/lang/py/build/test/../scripts/avro cat /tmp/REIX --fields 'first, last'
{"first": "daffy"}
{"first": "bugs"}
{"first": "tweety"}
{"first": "road"}
{"first": "wile"}
{"first": "pepe"}
{"first": "foghorn"}
Usage: avro cat|write [options] FILE [FILE...]

avro: error: Can't open last - [Errno 2] No such file or directory: 'last'
$ echo $?
2

The issue is that 'first, last' is not parsed correctly. The white space between "," and "last" is parsed in a way that makes "last" been interpreted as a "FILE" instead of an option.

Without the white space, that works fine:

$ /home/tony/AVRO/avro-FromFreshIBMSOEGitHub/lang/py/build/test/../scripts/avro cat /tmp/REIX --fields 'first,last'
{"last": "duck", "first": "daffy"}
{"last": "bunny", "first": "bugs"}
{"last": "", "first": "tweety"}
{"last": "runner", "first": "road"}
{"last": "e", "first": "wile"}
{"last": "le pew", "first": "pepe"}
{"last": "leghorn", "first": "foghorn"}
$ echo $?
0

- On Ubuntu / x86_64, both 'first, last' and 'first,last' are OK:

$ pwd
..../lang/py
$ export PYTHONPATH=./build/src
$ build/test/../scripts/avro cat /tmp/REIX --fields 'first, last'
{"last": "duck", "first": "daffy"}
{"last": "bunny", "first": "bugs"}
{"last": "", "first": "tweety"}
{"last": "runner", "first": "road"}
{"last": "e", "first": "wile"}
{"last": "le pew", "first": "pepe"}
{"last": "leghorn", "first": "foghorn"}
$ echo $?
0

Source code of: /home/tony/AVRO/avro-FromFreshIBMSOEGitHub/lang/py/build/test/../scripts/avro is:

def main(argv=None):
    import sys
    from optparse import OptionParser, OptionGroup

    argv = argv or sys.argv

    parser = OptionParser(description="Display/write for Avro files",
                      version="1.7.4",
                      usage="usage: %prog cat|write [options] FILE [FILE...]")
....

So, OptionParser() does not work on PPC64 like it does on x86_64.

[Tony Reix (tony-reix)]

In Avro, the issue appears as:

  [py-test] ERROR: test_fields (test_script.TestCat)
  [py-test] ----------------------------------------------------------------------
  [py-test] Traceback (most recent call last):
  [py-test] File "/home/tony/AVRO/avro-FromFreshIBMSOEGitHub/lang/py/build/test/test_script.py", line 169, in test_fields
  [py-test] out = self._run('--fields', 'first, last')
  [py-test] File "/home/tony/AVRO/avro-FromFreshIBMSOEGitHub/lang/py/build/test/test_script.py", line 109, in _run
  [py-test] out = check_output([SCRIPT, "cat", self.avro_file] + list(args))
  [py-test] File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
  [py-test] raise CalledProcessError(retcode, cmd, output=output)
  [py-test] CalledProcessError: Command '['/home/tony/AVRO/avro-FromFreshIBMSOEGitHub/lang/py/build/test/../scripts/avro', 'cat', '/tmp/tmp3Oiy6H', '--fields', 'first, last']' returned non-zero exit status 2

If the string 'first, last' is replaced by 'first,last' (no space), another issue appears (that does not appear on x86_64), with the string '' :

  [py-test] ERROR: test_fields (test_script.TestCat)
  [py-test] Traceback (most recent call last):
  [py-test]
  [py-test] File "/home/tony/AVRO/avro-FromFreshIBMSOEGitHub/lang/py/build/test/test_script.py", line 173, in test_fields
  [py-test] Schema: ["string", "null", "long"]
  [py-test] out = self._run('--fields', '')
  [py-test] Datum: None
  [py-test] File "/home/tony/AVRO/avro-FromFreshIBMSOEGitHub/lang/py/build/test/test_script.py", line 109, in _run
  [py-test] Codec: deflate
  [py-test] out = check_output([SCRIPT, "cat", self.avro_file] + list(args))
  [py-test] Round Trip Data: [None, None, None, None, None, None, None, None, None, None]
  [py-test] File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
  [py-test] Round Trip Data Length: 10
  [py-test] raise CalledProcessError(retcode, cmd, output=output)
  [py-test] Correct Round Trip: True
  [py-test]
  [py-test] CalledProcessError: Command '['/home/tony/AVRO/avro-FromFreshIBMSOEGitHub/lang/py/build/test/../scripts/avro', 'cat', '/tmp/tmpJJp2gX', '--fields', '']' returned non-zero exit status 2

The issue in Python is triggered by this Avro test file:
    lang/py/build/test/test_script.py

class TestCat(unittest.TestCase):
.....
     def test_fields(self):
     ...
        # Field selection (with comma and space)
        out = self._run('--fields', 'first, last')
        assert json.loads(out[0]) == {'first': 'daffy', 'last': 'duck'}

        # Empty fields should get all
        out = self._run('--fields', '')

So, there are 2 issues with OptionParser():
 - 'first, last'
 - '' (empty field)

[Matthias Klose (doko)]

please can you check this in current utopic (14.10) as well?

[Breno Leitão (breno-leitao)]

Tony,

what is the contect of /tmp/REIX?

I am trying to reproduce it on 14.10, using a simple text file at /tmp/REIX and I got:

ubuntu@ubuntu:~/avro/avro/trunk/lang/py/build/scripts-2.7$ avro cat /tmp/REIX --fields 'first,last'
panic: ord() expected a character, but string of length 0 found

[Tony Reix (tony-reix)]

Hi Breno,

I have changed the source code of the Avro Python tests in order to save the temporary file used by the test. I saved it in a file I named /tmp/REIX . This file contains non-asci data. I gonna attach it to this bug. Also, I think this file depends on the machine architecture.

About the "avro" command you used, maybe you are not using the correct one. I do not see any scripts-2.7 directory .
Maybe you have to try:
  cd lang/py/build/scripts
  ./avro cat /tmp/REIX --fields 'first, last'

Tony

[Tony Reix (tony-reix)]

/tmp/REIX file for testing.

[Tony Reix (tony-reix)]

Hi Matthias,
We have checked, and there is no 14.10 version yet for PPC64LE.

[Tony Reix (tony-reix)]

I've tried to use PDb on Ubuntu/PPC64LE for debugging the issue.

However, with Pdb, the issue disappears...
We have in "opts" variable the correct value for fields: 'fields': 'first, last'

$ pdb ../../lang/py/build/scripts/avro cat /tmp/REIX --fields 'first, last'

build/scripts/avro: line 243:
  opts, args = parser.parse_args(argv[1:])

Pdb) c
> /home/tony/AVRO/avro-FromFreshIBMSOEGitHub/lang/py/build/scripts/avro(243)main()
-> opts, args = parser.parse_args(argv[1:])
(Pdb) n
...
(Pdb) p opts
<Values at 0x3fff9b8274d0: {'count': inf, 'format': 'json', 'skip': 0, 'input_type': None, 'print_schema': False, 'filter': None, 'header': False, 'fields': 'first, last', 'output': '-', 'schema': None}>
(Pdb) p args
['cat', '/tmp/REIX']

.......

(Pdb) n
> /home/tony/AVRO/avro-FromFreshIBMSOEGitHub/lang/py/build/scripts/avro(125)cat()
-> print_avro(avro, opts)
(Pdb) n
{"last": "duck", "first": "daffy"}
{"last": "bunny", "first": "bugs"}
{"last": "", "first": "tweety"}
{"last": "runner", "first": "road"}
{"last": "e", "first": "wile"}
{"last": "le pew", "first": "pepe"}
{"last": "leghorn", "first": "foghorn"}
> /home/tony/AVRO/avro-FromFreshIBMSOEGitHub/lang/py/build/scripts/avro(113)cat()
-> for filename in args:
(Pdb) n
--Return--
> /home/tony/AVRO/avro-FromFreshIBMSOEGitHub/lang/py/build/scripts/avro(113)cat()->None
-> for filename in args:
(Pdb) n
--Return--
> /home/tony/AVRO/avro-FromFreshIBMSOEGitHub/lang/py/build/scripts/avro(250)main()->None
-> cat(opts, args)
(Pdb) n
--Return--
> /home/tony/AVRO/avro-FromFreshIBMSOEGitHub/lang/py/build/scripts/avro(261)<module>()->None
-> main()
(Pdb) n
--Return--
> <string>(1)<module>()->None
(Pdb) n
The program finished and will be restarted

Launching the same command withoug pdb:
    ../../lang/py/build/scripts/avro cat /tmp/REIX --fields 'first, last'
raised the same issue:

{"first": "daffy"}
{"first": "bugs"}
{"first": "tweety"}
{"first": "road"}
{"first": "wile"}
{"first": "pepe"}
{"first": "foghorn"}
Usage: avro cat|write [options] FILE [FILE...]

avro: error: Can't open last - [Errno 2] No such file or directory: 'last'

[Tony Reix (tony-reix)]

Looking at ./Lib/optparse.py of Python 2.7.6 :
   def parse_args(self, args=None, values=None):
....
   stop = self._process_args(largs, rargs, values)

  _process_args()
       _process_args()
           _process_long_opt()

Probably the issue is in :

    def _process_long_opt(self, rargs, values):
 ...

??

[Breno Leitão (breno-leitao)]

Tony,

You can check the same bug on Ubuntu 14.10 daily image that could be found at:

http://cdimage.ubuntu.com/ubuntu-server/daily/current/

[Tony Reix (tony-reix)]

Hi Breno
OK. There is a place where to get an install image for Ubuntu 14.10 for PPC64 LE.
Since the PPC64/LE machines I use are managed by another team, it may be complicated/long to get them agree to install a machine with something unstable. Anyway, I'll ask if it is possible.

Now, why do you think that the issue is in the Ubuntu system and not simply in the port of the Python 2.7.6 on PPC64/LE ?
I don't know a lot about Python, however I guess that there is probably C code that may have an issue.

I'll probably check with other versions of Python 2, in order to know if the issue is specific to some sub-version of Python 2, or only to this 2.7.6 .

[Matthias Klose (doko)]

Am 09.10.2014 um 15:49 schrieb Tony Reix:
> Hi Breno
> OK. There is a place where to get an install image for Ubuntu 14.10 for PPC64 LE.
> Since the PPC64/LE machines I use are managed by another team, it may be complicated/long to get them agree to install a machine with something unstable. Anyway, I'll ask if it is possible.
>
> Now, why do you think that the issue is in the Ubuntu system and not simply in the port of the Python 2.7.6 on PPC64/LE ?
> I don't know a lot about Python, however I guess that there is probably C code that may have an issue.

I don't think there is any ppc64 specific. I'd like to understand the issue
first, and a reproducer with a single test case would be much appreciated.

Of course, this could be a miscompilation of python2.7 on ppc64el, but that is a
wild guess, given that the python testsuite for this build doesn't show any errors.

[Tony Reix (tony-reix)]

I've found the issue. I had changed the python shell in order to add some trace options and that generated the issue. Removing the change did fix the issue. And, looking in depth at Avro results, I found that the final result is mixed with the traces, and that the result is OK.
So, that is my mistake. Sorry. Gonna close the defect.

[Tony Reix (tony-reix)]

No issue with Python. My mistake.