python3.4.3 SRU breaks awscli

[Nelson Elhage]

The recently (~6h ago) python3.4.3 SRU update contains backwards-incompatible changes to the Python language, which in particular break the version of `awscli` packaged in Ubuntu Trusty. From a fully-updated Trusty machine:

<email address hidden>:~# aws
Traceback (most recent call last):
  File "/usr/bin/aws", line 23, in <module>
    sys.exit(main())
  File "/usr/bin/aws", line 19, in main
    return awscli.clidriver.main()
  File "/usr/share/awscli/awscli/clidriver.py", line 44, in main
    driver = create_clidriver()
  File "/usr/share/awscli/awscli/clidriver.py", line 53, in create_clidriver
    event_hooks=emitter)
  File "/usr/share/awscli/awscli/plugin.py", line 49, in load_plugins
    plugin.awscli_initialize(event_hooks)
  File "/usr/share/awscli/awscli/handlers.py", line 73, in awscli_initialize
    register_removals(event_handlers)
  File "/usr/share/awscli/awscli/customizations/removals.py", line 32, in register_removals
    'verify-email-address'])
  File "/usr/share/awscli/awscli/customizations/removals.py", line 45, in remove
    self._create_remover(remove_commands))
  File "/usr/lib/python3/dist-packages/botocore/hooks.py", line 64, in register
    self._verify_accept_kwargs(handler)
  File "/usr/lib/python3/dist-packages/botocore/hooks.py", line 84, in _verify_accept_kwargs
    argspec = inspect.getargspec(func)
  File "/usr/lib/python3.4/inspect.py", line 936, in getargspec
    raise ValueError("Function has keyword-only arguments or annotations"
ValueError: Function has keyword-only arguments or annotations, use getfullargspec() API which can support them

This bug has been fixed upstream via https://github.com/aws/aws-cli/issues/800, but the version in Ubuntu hasn't been updated.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: python3.4 3.4.3-1ubuntu1~14.04.1
ProcVersionSignature: Ubuntu 3.13.0-63.103-generic 3.13.11-ckt25
Uname: Linux 3.13.0-63-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.13
Architecture: amd64
Date: Wed Sep 23 21:06:39 2015
Ec2AMI: ami-8d5b5dbd
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-west-2c
Ec2InstanceType: c4.8xlarge
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: python3.4
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Marc Deslauriers (mdeslaur)]

ACK on the debdiff, uploaded to queue for the SRU team.

[Matthias Klose (doko)]

not sure what the best thing to do ... either revert that, or fix the packages affected by this incompatible upstream change.

[Adam Conrad (adconrad)]

Hello Nelson, or anyone else affected,

Accepted python-botocore into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-botocore/0.29.0+repack-2ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Adrien Siebert (asiebert)]

Hi Adam,

just successfully tested out 0.29.0+repack-2ubuntu0.1 (upgraded from 0.29.0+repack-2) on trusty-proposed − thanks for providing all the instructions to quickly get to it.

We had the exact same issue when using the aws-cli after the python3.4 upgrade, and from our limited testing it appear to be fixed with the proposed package.

[Slobodan Miskovic (slobodan)]

0.29.0+repack-2ubuntu0.1 fixes the problem for us

[Wayne Clancy (wayneclancy80)]

I'm still seeing the problem. Fully updated version of 14.04 LTS running in AWS ap-northeast-1. It appears have the latest and greatest python3-botocore.

Can this ticket please be re-opened?

root@ip-10-0-100-139:~# aws
Traceback (most recent call last):
  File "/usr/bin/aws", line 23, in <module>
    sys.exit(main())
  File "/usr/bin/aws", line 19, in main
    return awscli.clidriver.main()
  File "/usr/share/awscli/awscli/clidriver.py", line 44, in main
    driver = create_clidriver()
  File "/usr/share/awscli/awscli/clidriver.py", line 53, in create_clidriver
    event_hooks=emitter)
  File "/usr/share/awscli/awscli/plugin.py", line 49, in load_plugins
    plugin.awscli_initialize(event_hooks)
  File "/usr/share/awscli/awscli/handlers.py", line 73, in awscli_initialize
    register_removals(event_handlers)
  File "/usr/share/awscli/awscli/customizations/removals.py", line 32, in register_removals
    'verify-email-address'])
  File "/usr/share/awscli/awscli/customizations/removals.py", line 45, in remove
    self._create_remover(remove_commands))
  File "/usr/lib/python3/dist-packages/botocore/hooks.py", line 64, in register
    self._verify_accept_kwargs(handler)
  File "/usr/lib/python3/dist-packages/botocore/hooks.py", line 84, in _verify_accept_kwargs
    argspec = inspect.getargspec(func)
  File "/usr/lib/python3.4/inspect.py", line 936, in getargspec
    raise ValueError("Function has keyword-only arguments or annotations"
ValueError: Function has keyword-only arguments or annotations, use getfullargspec() API which can support them

--Packages installed---

ii awscli 1.2.9-2 all Universal Command Line Environment for AWS
ii dh-python 1.20140128-1ubuntu8.2 all Debian helper tools for packaging Python libraries and applications
ii libpython-stdlib:amd64 2.7.5-5ubuntu3 amd64 interactive high-level object-oriented language (default python version)
ii libpython2.7:amd64 2.7.6-8ubuntu0.2 amd64 Shared Python runtime library (version 2.7)
ii libpython2.7-minimal:amd64 2.7.6-8ubuntu0.2 amd64 Minimal subset of the Python language (version 2.7)
ii libpython2.7-stdlib:amd64 2.7.6-8ubuntu0.2 amd64 Interactive high-level object-oriented language (standard library, version 2.7)
ii libpython3-stdlib:amd64 3.4.0-0ubuntu2 amd64 interactive high-level object-oriented language (default python3 version)
ii libpython3.4-minimal:amd64 3.4.3-1ubuntu1~14.04.1 amd64 Minimal subset of the Python language (version 3.4)
ii libpython3.4-stdlib:amd64 3.4.3-1ubuntu1~14.04.1 amd64 Interactive high-level object-oriented language (standard library, version 3.4)
ii python 2.7.5-5ubuntu3 amd64 interactive high-level object-oriented language (default version)
ii python-apt 0.9.3.5ubuntu1 amd64 Python interface to libapt-pkg
ii python-apt-common ...

[Silvio SantoZ (pardinilist)]

@adconrad: I can confirm that python3-botocore (0.29.0+repack-2ubuntu0.1) fixes the problem. Thanks!
@wayne: no, you don't have the updated botocore. Try this:

wget "http://launchpadlibrarian.net/218773283/python3-botocore_0.29.0%2Brepack-2ubuntu0.1_all.deb"
dpkg -i python3-botocore_0.29.0+repack-2ubuntu0.1_all.deb

[Tom Wills (r-mail-q)]

I can run awscli successfully after upgrading to 0.29.0+repack-2ubuntu0.1 from trusty-proposed. Thanks.

[Barry Warsaw (barry)]

Since this is fixed in boto, are the Python bugtasks still relevant?

[Matthias Klose (doko)]

won't fix in python3.4 for trusty; let's update other packages in the archive if we find them. Third party software is much more likely to expect the 3.4.3 behaviour over the 3.4.0 behaviour.

[Jaap-Andre de Hoop (j-dehoop)]

Just update python3-botocore from ubuntu-proposed and aws (awscli) is running without a problem (on two computers) again,

[Stefan Horning (stefan-horning)]

I don't know what I am doing wrong but I am still seing the same error when executing the aws command even after I installed the python-botocore (0.29.0+repack-2ubuntu0.1) package from the proposed repo.

Steps
1. Added deb http://archive.ubuntu.com/ubuntu/ trusty-proposed restricted main multiverse universe to sources.list
2. apt-get update
3. apt-get install python-botocore, which correctly installed the patched versios as far I can see:
Selecting previously unselected package python-ply.
(Reading database ... 94892 files and directories currently installed.)
Preparing to unpack .../python-ply_3.4-3ubuntu2_all.deb ...
Unpacking python-ply (3.4-3ubuntu2) ...
Selecting previously unselected package python-jmespath.
Preparing to unpack .../python-jmespath_0.2.1-1_all.deb ...
Unpacking python-jmespath (0.2.1-1) ...
Selecting previously unselected package python-dateutil.
Preparing to unpack .../python-dateutil_1.5+dfsg-1ubuntu1_all.deb ...
Unpacking python-dateutil (1.5+dfsg-1ubuntu1) ...
Selecting previously unselected package python-botocore.
Preparing to unpack .../python-botocore_0.29.0+repack-2ubuntu0.1_all.deb ...
Unpacking python-botocore (0.29.0+repack-2ubuntu0.1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up python-ply (3.4-3ubuntu2) ...
Setting up python-jmespath (0.2.1-1) ...
Setting up python-dateutil (1.5+dfsg-1ubuntu1) ...
Setting up python-botocore (0.29.0+repack-2ubuntu0.1) ...

4. execute aws and still get
Traceback (most recent call last):
  File "/usr/bin/aws", line 23, in <module>
    sys.exit(main())
  File "/usr/bin/aws", line 19, in main
    return awscli.clidriver.main()
  File "/usr/share/awscli/awscli/clidriver.py", line 44, in main
    driver = create_clidriver()
  File "/usr/share/awscli/awscli/clidriver.py", line 53, in create_clidriver
    event_hooks=emitter)
  File "/usr/share/awscli/awscli/plugin.py", line 49, in load_plugins
    plugin.awscli_initialize(event_hooks)
  File "/usr/share/awscli/awscli/handlers.py", line 73, in awscli_initialize
    register_removals(event_handlers)
  File "/usr/share/awscli/awscli/customizations/removals.py", line 32, in register_removals
    'verify-email-address'])
  File "/usr/share/awscli/awscli/customizations/removals.py", line 45, in remove
    self._create_remover(remove_commands))
  File "/usr/lib/python3/dist-packages/botocore/hooks.py", line 64, in register
    self._verify_accept_kwargs(handler)
  File "/usr/lib/python3/dist-packages/botocore/hooks.py", line 84, in _verify_accept_kwargs
    argspec = inspect.getargspec(func)
  File "/usr/lib/python3.4/inspect.py", line 936, in getargspec
    raise ValueError("Function has keyword-only arguments or annotations"
ValueError: Function has keyword-only arguments or annotations, use getfullargspec() API which can support them

Or do I have to reinstall awscli for some reason too?

[Stefan Horning (stefan-horning)]

Here some system information of where I tried it:

Linux ip-172-16-1-173 3.13.0-65-generic #105-Ubuntu SMP Mon Sep 21 18:50:58 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty

[Stefan Horning (stefan-horning)]

Ok the trick seems to be that you have to activate the proposed repo before installing awscli and any package it brings onto the system. So if you already have awscli installed you should to a apt-get remove awscli AND a apt-get autoclean and then to a fresh install of awscli, afther that the bug is fixed.

[Stefan Horning (stefan-horning)]

I of course meant `apt-get autoremove`

[Jeff Hansen (sqrammi)]

When will python-botocore hit updates? I still get the error right now on a fresh Ubuntu instance.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-botocore - 0.29.0+repack-2ubuntu0.1

---------------
python-botocore (0.29.0+repack-2ubuntu0.1) trusty; urgency=low

  * debian/patches/support-python3.4.1: Fix use of inspect with Python
    3.4.1 or later. (LP: #1499075)

 -- Evan Broder <email address hidden> Wed, 23 Sep 2015 17:35:34 -0400

[Chris J Arges (arges)]

The verification of the Stable Release Update for python-botocore has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.