Tempest has a number of negative test cases that check for failure codes from the API calls. But some of these are treated as errors with tracebacks in the log. Here are two examples from the floating ip tests. I have been struggling with this general issue while stress testing nova. There needs to be some way to scan a log file, or use publish_errors, and understand if the system is misbehaving. Use of the ERROR log level would be a good way to do that but there are too many places where log.error or log.exception is called that should really be DEBUG or INFO. I think this is as a huge problem for testing.
/var/log/nova/nova-network.log:2012-04-09 15:06:32 ERROR nova.rpc.amqp [req-90cca942-5980-4ebe-8139-282a6567e6f9 5b963fedd517476f899bd73a7fad4d06 18cdc649708644a68030088165282e05] Exception during message handling
/var/log/nova/nova-network.log:2012-04-09 15:06:32 ERROR nova.rpc.amqp [req-90cca942-5980-4ebe-8139-282a6567e6f9 5b963fedd517476f899bd73a7fad4d06 18cdc649708644a68030088165282e05] Returning exception Floating ip not found for id 99923926838806. to caller
/var/log/nova/nova-network.log:2012-04-09 15:06:32 ERROR nova.rpc.amqp [req-90cca942-5980-4ebe-8139-282a6567e6f9 5b963fedd517476f899bd73a7fad4d06 18cdc649708644a68030088165282e05] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 252, in _process_data\n rval = node_func(context=ctxt, **node_args)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 258, in wrapped\n return func(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 555, in get_floating_ip\n return dict(self.db.floating_ip_get(context, id).iteritems())\n', ' File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 232, in floating_ip_get\n return IMPL.floating_ip_get(context, id)\n', ' File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 120, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 613, in floating_ip_get\n raise exception.FloatingIpNotFound(id=id)\n', 'FloatingIpNotFound: Floating ip not found for id 99923926838806.\n']
/var/log/nova/nova-network.log:2012-04-09 15:06:32 ERROR nova.rpc.amqp [req-a9345519-8aab-4591-8c1b-532416b0e2ac 5b963fedd517476f899bd73a7fad4d06 18cdc649708644a68030088165282e05] Exception during message handling
/var/log/nova/nova-network.log:2012-04-09 15:06:32 ERROR nova.rpc.amqp [req-a9345519-8aab-4591-8c1b-532416b0e2ac 5b963fedd517476f899bd73a7fad4d06 18cdc649708644a68030088165282e05] Returning exception Floating ip not found for address 0.0.0.0. to caller
/var/log/nova/nova-network.log:2012-04-09 15:06:32 ERROR nova.rpc.amqp [req-a9345519-8aab-4591-8c1b-532416b0e2ac 5b963fedd517476f899bd73a7fad4d06 18cdc649708644a68030088165282e05] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 252, in _process_data\n rval = node_func(context=ctxt, **node_args)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 258, in wrapped\n return func(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 567, in get_floating_ip_by_address\n address).iteritems())\n', ' File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 304, in floating_ip_get_by_address\n return IMPL.floating_ip_get_by_address(context, address)\n', ' File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 120, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 774, in floating_ip_get_by_address\n raise exception.FloatingIpNotFoundForAddress(address=address)\n', 'FloatingIpNotFoundForAddress: Floating ip not found for address 0.0.0.0.\n']
/
Or maybe CRITICAL should be used for internal errors.