tempest config error in standalone upgrade in reproducer only: TypeError: must be string or buffer, not callable-iterator

[wes hayutin]

2019-04-02 21:59:57 | 2019-04-02 21:59:57.768 81931 INFO config_tempest.constants [-] Creating image 'cirros-0.3.5-x86_64-disk.img'
2019-04-02 21:59:57 | 2019-04-02 21:59:57.818 81931 INFO config_tempest.constants [-] Uploading image 'cirros-0.3.5-x86_64-disk.img' from '/home/zuul/tempest/etc/cirros-0.3.5-x86_64-disk.img'
2019-04-02 21:59:57 | 2019-04-02 21:59:57.923 81931 INFO tempest.lib.common.rest_client [req-9982d869-6422-460b-a621-ea47f57532aa ] Request (main): 201 POST http://192.168.24.1:9292/v2/images 0.104s
2019-04-02 22:00:31 | 2019-04-02 22:00:31.082 81931 WARNING urllib3.connectionpool [-] Retrying (Retry(total=9, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(32, 'Broken pipe'))': /v2/images/891e291a-9cd9-49c6-ab5e-31e81473059d/file: error: [Errno 32] Broken pipe
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 CRITICAL tempest [-] Unhandled error: TypeError: must be string or buffer, not callable-iterator
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest Traceback (most recent call last):
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/bin/discover-tempest-config", line 10, in <module>
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest sys.exit(main())
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/main.py", line 576, in main
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest verbose=args.verbose
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/main.py", line 519, in config_tempest
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest image.create_tempest_images(conf)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/services/image.py", line 104, in create_tempest_images
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest image_dest=img_path)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/services/image.py", line 150, in find_or_upload_image
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest image = self._upload_image(image_name, image_dest)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/services/image.py", line 196, in _upload_image
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest self.client.store_image_file(image['id'], data)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/services/image/v2/images_client.py", line 151, in store_image_file
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest body=data, chunked=True)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 664, in request
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest body=body, chunked=chunked)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 555, in _request
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest chunked=chunked
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 590, in raw_request
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest body=body, chunked=chunked)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/common/http.py", line 112, in request
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest *args, **new_kwargs)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 72, in request
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest **urlopen_kw)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 150, in request_encode_body
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest return self.urlopen(method, url, **extra_kw)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 323, in urlopen
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest response = conn.urlopen(method, u.request_uri, **kw)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest **response_kw)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest chunked=chunked)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest conn.request(method, url, **httplib_request_kw)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib64/python2.7/httplib.py", line 1041, in request
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest self._send_request(method, url, body, headers)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest self.endheaders(body)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest self._send_output(message_body)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib64/python2.7/httplib.py", line 885, in _send_output
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest self.send(message_body)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib64/python2.7/httplib.py", line 857, in send
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest self.sock.sendall(data)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest File "/usr/lib64/python2.7/socket.py", line 224, in meth
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest return getattr(self._sock,name)(*args)
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest TypeError: must be string or buffer, not callable-iterator
2019-04-02 22:00:31 | 2019-04-02 22:00:31.086 81931 ERROR tempest 

See the attached log for more details

[wes hayutin (weshayutin)]

I think this is no longer happening.. I had a tripleo-standalone-upgrade job complete using the reproducer work on Sunday. Will try again to be sure.

[wes hayutin (weshayutin)]

working locally for me w/ a f28 host, and centos-7-standalone job.

closing.

019-04-26 17:53:36 | 2019-04-26 17:53:36.331 45 DEBUG config_tempest.constants [-] Setting [network-feature-enabled] api_extensions = default-subnetpools,qos,availability_zone,network_availability_zone,auto-allocated-topology,ext-gw-mode,binding,agent,subnet_allocation,dhcp_agent_scheduler,trunk-details,address-scope,external-net,standard-attr-tag,flavors,segment,net-mtu,network-ip-availability,qos-default,quotas,revision-if-match,availability_zone_filter,l3-ha,filter-validation,subnet-segmentid-writable,multi-provider,quota_details,l2_adjacency,trunk,extraroute,net-mtu-writable,empty-string-filtering,subnet-service-types,port-mac-address-regenerate,standard-attr-timestamp,provider,service-type,qos-rule-type-details,l3-flavors,port-security,extra_dhcp_opt,port-security-groups-filtering,standard-attr-revisions,pagination,sorting,standard-attr-segment,segments-peer-subnet-host-routes,security-group,l3_agent_scheduler,fip-port-details,router_availability_zone,rbac-policies,project-id,qos-bw-limit-direction,standard-attr-description,ip-substring-filtering,router,allowed-address-pairs,ip_allocation,qos-fip,binding-extended set /usr/lib/python2.7/site-packages/config_tempest/tempest_conf.py:103
2019-04-26 17:53:36 | 2019-04-26 17:53:36.331 45 DEBUG config_tempest.constants [-] Setting [volume-feature-enabled] api_extensions = os-hosts,os-vol-tenant-attr,os-quota-sets,os-types-manage,os-volume-encryption-metadata,os-snapshot-actions,backups,os-volume-actions,os-used-limits,os-volume-type-access,consistencygroups,os-vol-host-attr,encryption,os-availability-zone,capabilities,cgsnapshots,os-types-extra-specs,os-snapshot-manage,os-vol-mig-status-attr,os-volume-unmanage,os-volume-manage,os-image-create,os-extended-services,os-extended-snapshot-attributes,os-snapshot-unmanage,qos-specs,os-quota-class-sets,os-volume-transfer,os-vol-image-meta,os-admin-actions,os-services,scheduler-stats set /usr/lib/python2.7/site-packages/config_tempest/tempest_conf.py:103
2019-04-26 17:53:36 | 2019-04-26 17:53:36.331 45 INFO config_tempest.constants [-] Creating configuration file /var/lib/tempest/tempest/etc/tempest.conf
2019-04-26 17:53:37 | +--------------------------------+-----------------------------------------------------------------+
2019-04-26 17:53:37 | | Name | EntryPoint |
2019-04-26 17:53:37 | +--------------------------------+-----------------------------------------------------------------+
2019-04-26 17:53:37 | | zaqar_tests | zaqar_tempest_plugin.plugin:ZaqarTempestPlugin |
2019-04-26 17:53:37 | | watcher_tests | watcher_tempest_plugin.plugin:WatcherTempestPlugin |
2019-04-26 17:53:37 | | vitrage_tests | vitrage_tempest_plugin.plugin:VitrageTempestPlugin |
2019-04-26 17:53:37 | | trove_tests | trove_tempest_plugin.plugin:TroveTempestPlugin |
2019-04-26 17:53:37 | | tripleo_common_tests | tripleo_common_tempest_plugin.plugin:TripleOCommonTempestPlugin |
2019-04...