assert len(_active) == 1 in _after_fork

Bug #1821307 reported by Attila Fazekas
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
oslo.service
New
Undecided
Unassigned

Bug Description

When you directly start for ex. g-api, n-cpu, c-api:

Exception ignored in: <function _after_fork at 0x7f666969e620>
Traceback (most recent call last):
  File "/usr/lib64/python3.7/threading.py", line 1335, in _after_fork
    assert len(_active) == 1
AssertionError:

The message usually does not appears either in journal or in the service log,
but it is visible on the console.

Python 3.7.2

No direct evidence for it is the root cause for another issue,
but might be related.

Revision history for this message
Attila Fazekas (afazekas) wrote :
Revision history for this message
XiaojueGuan (xiaojuegaun) wrote :
Download full text (4.1 KiB)

➜ nova git:(master) tox -e py37 nova.tests.functional.test_servers
py37 develop-inst-nodeps: /home/steveguan/code/Openstack/nova
py37 installed: alembic==1.3.1,amqp==2.5.2,appdirs==1.4.3,attrs==19.3.0,automaton==1.17.0,Babel==2.7.0,bandit==1.6.2,bcrypt==3.1.7,cachetools==3.1.1,castellan==1.4.0,certifi==2019.11.28,cffi==1.13.2,chardet==3.0.4,cliff==2.16.0,cmd2==0.8.9,colorama==0.4.1,coverage==4.5.4,cryptography==2.8,cursive==0.2.2,ddt==1.2.2,debtcollector==1.22.0,decorator==4.4.1,dnspython==1.15.0,dogpile.cache==0.9.0,eventlet==0.25.1,extras==1.0.0,fasteners==0.14.1,fixtures==3.0.0,flake8==2.6.2,future==0.18.2,futurist==1.10.0,gabbi==1.49.0,gitdb2==2.0.6,GitPython==3.0.5,greenlet==0.4.15,hacking==1.1.0,idna==2.8,importlib-metadata==1.1.0,iso8601==0.1.12,Jinja2==2.10.3,jmespath==0.9.4,jsonpatch==1.24,jsonpath-rw==1.4.0,jsonpath-rw-ext==1.2.2,jsonpointer==2.0,jsonschema==3.2.0,keystoneauth1==3.18.0,keystonemiddleware==7.0.1,kombu==4.6.6,linecache2==1.0.0,lxml==4.4.2,Mako==1.1.0,MarkupSafe==1.1.1,mccabe==0.5.3,microversion-parse==0.2.1,mock==3.0.5,monotonic==1.5,more-itertools==8.0.0,mox3==0.28.0,msgpack==0.6.2,munch==2.5.0,netaddr==0.7.19,netifaces==0.10.9,networkx==2.4,-e git+https://github.com/openstack/nova.git@2f31c8d1b97c2b64581d8f824b6547f93dfd94fb#egg=nova,numpy==1.17.4,openstacksdk==0.38.0,os-brick==2.10.0,os-client-config==1.33.0,os-resource-classes==0.5.0,os-service-types==1.7.0,os-traits==2.0.0,os-vif==1.17.0,os-win==4.3.2,os-xenapi==0.3.4,osc-lib==1.14.1,oslo.cache==1.38.0,oslo.concurrency==3.31.0,oslo.config==6.12.0,oslo.context==2.23.0,oslo.db==5.0.2,oslo.i18n==3.25.0,oslo.log==3.45.0,oslo.messaging==10.4.0,oslo.middleware==3.38.1,oslo.policy==2.4.0,oslo.privsep==1.33.3,oslo.reports==1.30.0,oslo.rootwrap==5.16.1,oslo.serialization==2.29.2,oslo.service==1.40.2,oslo.upgradecheck==0.3.2,oslo.utils==3.42.1,oslo.versionedobjects==1.36.1,oslo.vmware==2.34.1,oslotest==3.8.1,osprofiler==2.9.0,ovs==2.11.0,ovsdbapp==0.18.0,packaging==19.2,paramiko==2.6.0,Paste==3.2.3,PasteDeploy==2.0.1,pbr==5.4.4,pluggy==0.13.1,ply==3.11,prettytable==0.7.2,psutil==5.6.7,psycopg2==2.8.4,py==1.8.0,pyasn1==0.4.8,pyasn1-modules==0.2.7,pycadf==2.10.0,pycodestyle==2.5.0,pycparser==2.19,pydot==1.4.1,pyflakes==1.2.3,pyinotify==0.9.6,PyJWT==1.7.1,PyMySQL==0.9.3,PyNaCl==1.3.0,pyOpenSSL==19.1.0,pyparsing==2.4.5,pyperclip==1.7.0,pypowervm==1.1.24,pyroute2==0.5.7,pyrsistent==0.15.6,pytest==5.3.1,python-barbicanclient==4.9.0,python-cinderclient==5.0.0,python-dateutil==2.8.1,python-editor==1.0.4,python-glanceclient==2.17.0,python-ironicclient==3.1.0,python-keystoneclient==3.22.0,python-mimeparse==1.6.0,python-neutronclient==6.14.0,python-subunit==1.3.0,pytz==2019.3,PyYAML==5.2,repoze.lru==0.7,requests==2.22.0,requests-mock==1.6.0,requestsexceptions==1.4.0,retrying==1.3.3,rfc3986==1.3.2,Routes==2.4.1,simplejson==3.17.0,six==1.13.0,smmap2==2.0.5,sortedcontainers==2.1.0,SQLAlchemy==1.3.11,sqlalchemy-migrate==0.13.0,sqlparse==0.3.0,statsd==3.3.0,stestr==2.5.1,stevedore==1.31.0,suds-jurko==0.6,taskflow==3.8.0,Tempita==0.5.2,tenacity==6.0.0,testresources==2.0.1,testscenarios==0.5.0,testtools==2.3.0,tooz==1.67.0,traceback2==1.4.0,unittest2==1.1.0,urllib3==1.25.7,vine==1.3.0,vo...

Read more...

Changed in oslo.service:
assignee: nobody → XiaojueGuan (xiaojuegaun)
Revision history for this message
XiaojueGuan (xiaojuegaun) wrote :

os.register_at_fork(*, before=None, after_in_parent=None, after_in_child=None)
Register callables to be executed when a new child process is forked using os.fork() or similar process cloning APIs. The parameters are optional and keyword-only. Each specifies a different call point.

before is a function called before forking a child process.

after_in_parent is a function called from the parent process after forking a child process.

after_in_child is a function called from the child process.

These calls are only made if control is expected to return to the Python interpreter. A typical subprocess launch will not trigger them as the child is not going to re-enter the interpreter.

Functions registered for execution before forking are called in reverse registration order. Functions registered for execution after forking (either in the parent or in the child) are called in registration order.

Note that fork() calls made by third-party C code may not call those functions, unless it explicitly calls PyOS_BeforeFork(), PyOS_AfterFork_Parent() and PyOS_AfterFork_Child().

There is no way to unregister a function.

Availability: Unix.

New in version 3.7.

https://docs.python.org/3/library/os.html

Revision history for this message
XiaojueGuan (xiaojuegaun) wrote :
Download full text (36.5 KiB)

GLOB sdist-make: /Users/steveguan/code/OpenStack/oslo.policy/setup.py
py37 inst-nodeps: /Users/steveguan/code/OpenStack/oslo.policy/.tox/.tmp/package/1/oslo.policy-2.4.2.dev1.zip
py37 installed: alabaster==0.7.12,appdirs==1.4.3,Babel==2.7.0,bandit==1.5.1,certifi==2019.11.28,cffi==1.13.2,chardet==3.0.4,cliff==2.16.0,cmd2==0.8.9,coverage==4.5.4,cryptography==2.8,debtcollector==1.22.0,decorator==4.4.1,docutils==0.15.2,dogpile.cache==0.9.0,dulwich==0.19.14,extras==1.0.0,fixtures==3.0.0,flake8==2.6.2,future==0.18.2,gitdb2==2.0.6,GitPython==3.0.5,hacking==1.1.0,idna==2.8,imagesize==1.1.0,iso8601==0.1.12,Jinja2==2.10.3,jmespath==0.9.4,jsonpatch==1.24,jsonpointer==2.0,keystoneauth1==3.18.0,linecache2==1.0.0,MarkupSafe==1.1.1,mccabe==0.5.3,mock==3.0.5,mox3==0.28.0,msgpack==0.6.2,munch==2.5.0,netaddr==0.7.19,netifaces==0.10.9,openstackdocstheme==1.31.2,openstacksdk==0.39.0,os-client-config==1.33.0,os-service-types==1.7.0,oslo.config==6.12.0,oslo.context==2.23.0,oslo.i18n==3.25.0,oslo.policy==2.4.2.dev1,oslo.serialization==2.29.2,oslo.utils==3.42.1,oslotest==3.8.1,packaging==19.2,pbr==5.4.4,prettytable==0.7.2,pycodestyle==2.0.0,pycparser==2.19,pyflakes==1.2.3,Pygments==2.5.2,pyparsing==2.4.5,pyperclip==1.7.0,python-mimeparse==1.6.0,python-subunit==1.3.0,pytz==2019.3,PyYAML==5.2,reno==2.11.3,requests==2.22.0,requests-mock==1.6.0,requestsexceptions==1.4.0,rfc3986==1.3.2,six==1.13.0,smmap2==2.0.5,snowballstemmer==2.0.0,Sphinx==2.2.2,sphinxcontrib-apidoc==0.3.0,sphinxcontrib-applehelp==1.0.1,sphinxcontrib-devhelp==1.0.1,sphinxcontrib-htmlhelp==1.0.2,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.2,sphinxcontrib-serializinghtml==1.1.3,stestr==2.5.1,stevedore==1.31.0,testtools==2.3.0,traceback2==1.4.0,unittest2==1.1.0,urllib3==1.25.7,voluptuous==0.11.7,wcwidth==0.1.7,wrapt==1.11.2
py37 run-test-pre: PYTHONHASHSEED='727722955'
py37 run-test: commands[0] | stestr run --slowest
{0} oslo_policy.tests.test_checks.CheckRegisterTestCase.test_register_check_decorator [0.001768s] ... ok
{0} oslo_policy.tests.test_checks.FalseCheckTestCase.test_call [0.000564s] ... ok
{0} oslo_policy.tests.test_checks.GenericCheckTestCase.test_constant_string_mismatch [0.004352s] ... ok
{1} oslo_policy.tests.test_checks.AndCheckTestCase.test_call_all_false [0.002167s] ... ok
{1} oslo_policy.tests.test_checks.GenericCheckTestCase.test_constant_literal_mismatch [0.001790s] ... ok
{1} oslo_policy.tests.test_checks.OrCheckTestCase.test_add_check [0.000673s] ... ok
{1} oslo_policy.tests.test_checks.OrCheckTestCase.test_call_first_true [0.000762s] ... ok
{1} oslo_policy.tests.test_checks.OrCheckTestCase.test_pop_check [0.000589s] ... ok
{0} oslo_policy.tests.test_checks.GenericCheckTestCase.test_deep_credentials_dictionary_lookup [0.002038s] ... ok
{1} oslo_policy.tests.test_checks.RoleCheckTestCase.test_format_value [0.001964s] ... ok
{0} oslo_policy.tests.test_checks.NotCheckTestCase.test_call_false [0.000896s] ... ok
{1} oslo_policy.tests.test_checks.RoleCheckTestCase.test_no_roles_case [0.001597s] ... ok
{1} oslo_policy.tests.test_checks.TrueCheckTestCase.test_str [0.000528s] ... ok
{0} oslo_policy.tests.test_checks.RoleCheckTestCase.test_accept [0.001403s] ... ok
{0} oslo_policy....

Changed in oslo.service:
assignee: XiaojueGuan (xiaojuegaun) → nobody
Revision history for this message
XiaojueGuan (xiaojuegaun) wrote :

this error is gone, shall we close this issue

Revision history for this message
Satish Patel (satish-txt) wrote :

Look like i am seeing same error in my compute (nova-compute) logs - https://bugs.launchpad.net/nova/+bug/1910837

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.