zuul job dsvm-grenade error

[zhangyuhe]

Many tasks have the following errors

2019-12-19 05:32:35.240 | ERROR nova.cmd.manage [None req-12e00068-1f43-43ea-9608-98b5c83b16bc None None] Error attempting to run <function fill_virtual_interface_list at 0x7f67744092f0>: ModuleNotFoundError: No module named 'memcache'
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage Traceback (most recent call last):
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/cmd/manage.py", line 504, in _run_migration
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  found, done = migration_meth(ctxt, count)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1015, in wrapper
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  return fn(*args, **kwargs)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/virtual_interface.py", line 279, in fill_virtual_interface_list
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  _set_or_delete_marker_for_migrate_instances(cctxt, marker)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 223, in wrapped
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  return f(context, *args, **kwargs)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/virtual_interface.py", line 305, in _set_or_delete_marker_for_migrate_instances
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  instance.create()
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  return fn(self, *args, **kwargs)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/instance.py", line 629, in create
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  self._load_ec2_ids()
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/instance.py", line 949, in _load_ec2_ids
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  self.ec2_ids = objects.EC2Ids.get_by_instance(self._context, self)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  result = fn(cls, context, *args, **kwargs)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/ec2.py", line 231, in get_by_instance
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  ec2_ids = cls._get_ec2_ids(context, instance)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/ec2.py", line 217, in _get_ec2_ids
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  ec2_ids['instance_id'] = id_to_ec2_inst_id(context, instance.uuid)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/ec2.py", line 57, in id_to_ec2_inst_id
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  int_id = get_int_id_from_instance_uuid(context, instance_id)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/ec2.py", line 36, in memoizer
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  _CACHE = cache_utils.get_client(expiration_time=_CACHE_TIME)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/cache_utils.py", line 54, in get_client
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  _get_default_cache_region(expiration_time=expiration_time))
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/cache_utils.py", line 65, in _get_default_cache_region
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  cache.configure_cache_region(CONF, region)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/oslo_cache/core.py", line 235, in configure_cache_region
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  '%s.' % conf.cache.config_prefix)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 608, in configure_from_config
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  "%sreplace_existing_backend" % prefix, False
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 436, in configure
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  _config_argument_dict, _config_prefix
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/api.py", line 95, in from_config_dict
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage  for key in config_dict
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 207, in __init__
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage  super(MemcacheArgs, self).__init__(arguments)
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 115, in __init__
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage  self._imports()
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 287, in _imports
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage  import memcache # noqa
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage ModuleNotFoundError: No module named 'memcache'

full log:
https://6081989ad82b8be7d120-53e9ef0ccb08d29c43f34421d87aa388.ssl.cf1.rackcdn.com/698632/3/check/networking-ovn-dsvm-grenade/72e94e8/logs/grenade.sh.txt.gz

[Brian Haley (brian-haley)]

This doesn't look like a bug in networking-ovn, but instead with nova, I added them as an affected project.

[zhangyuhe (ikukaku)]

hey haley
I think this is a problem in oslo.cache

[Matt Riedemann (mriedem)]

nova is configured to use memcache as the caching backend in this job:

https://6081989ad82b8be7d120-53e9ef0ccb08d29c43f34421d87aa388.ssl.cf1.rackcdn.com/698632/3/check/networking-ovn-dsvm-grenade/72e94e8/logs/etc/nova/nova.conf.txt.gz

[cache]
memcache_servers = localhost:11211
backend = dogpile.cache.memcached
enabled = True

I see dogpile.cache==0.9.0 installed:

https://6081989ad82b8be7d120-53e9ef0ccb08d29c43f34421d87aa388.ssl.cf1.rackcdn.com/698632/3/check/networking-ovn-dsvm-grenade/72e94e8/logs/pip3-freeze.txt.gz

And I see memcached is installed:

https://6081989ad82b8be7d120-53e9ef0ccb08d29c43f34421d87aa388.ssl.cf1.rackcdn.com/698632/3/check/networking-ovn-dsvm-grenade/72e94e8/logs/dpkg-l.txt.gz

ii memcached 1.5.6-0ubuntu1.2 amd64 high-performance memory object caching system

The errors drop off in recent days as well - was this fixed or worked around somehow?

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ModuleNotFoundError%3A%20No%20module%20named%20'memcache'%5C%22&from=7d

[Ghanshyam Mann (ghanshyammann)]

Its not occurring now.

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22ModuleNotFoundError:%20No%20module%20named%20'memcache'%5C%22&from=7d

I have not seen this recently and grenade job working fine. I do not know exact fix for that but marking incomplete for now and we can reopen this if show up again.

[Takashi Kajinami (kajinamit)]

This does not look like an actual problem in oslo.cache, but it's very likely a deployment problem

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]