Deb package libapache2-mod-wsgi not found with fuel-createmirror

[Oleg S. Gelbukh]

Steps to reproduce:

1. Install cluster
2. Run fuel-createmirror:

  fuel-createmirror -U
  fuel-createmirror -M

3. Install controller in the cluster

Expected result:

Controller deployed.

Actual result:

Deployment failed with multiple 404 Not Found errors in Puppet logs while verifying various OpenStack resources. The root cause seems to be in missing deb package 'libapache2-mod-wsgi':

2015-09-22T09:36:45.967565+00:00 info: (/Stage[main]/Apache::Mod::Wsgi/Apache::Mod[wsgi]/Package[libapache2-mod-wsgi]) Starting to evaluate the resource
2015-09-22T09:36:45.968372+00:00 debug: Executing '/usr/bin/dpkg-query -W --showformat '${Status} ${Package} ${Version}\n' libapache2-mod-wsgi'
2015-09-22T09:36:46.157221+00:00 debug: Executing '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install libapache2-mod-wsgi'
2015-09-22T09:36:46.356920+00:00 err: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install libapache2-mod-wsgi' returned 100: Reading package lists...
2015-09-22T09:36:46.357045+00:00 err: Building dependency tree...
2015-09-22T09:36:46.358112+00:00 err: Reading state information...
2015-09-22T09:36:46.358112+00:00 err: Package libapache2-mod-wsgi is not available, but is referred to by another package.
2015-09-22T09:36:46.358112+00:00 err: This may mean that the package is missing, has been obsoleted, or
2015-09-22T09:36:46.358112+00:00 err: is only available from another source
2015-09-22T09:36:46.358112+00:00 err:
2015-09-22T09:36:46.358112+00:00 err: E: Package 'libapache2-mod-wsgi' has no installation candidate
2015-09-22T09:36:46.358112+00:00 err:
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:188:in `execute'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/provider/command.rb:23:in `execute'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/provider.rb:237:in `block in has_command'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/provider.rb:444:in `block in create_class_and_instance_method'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/provider/package/apt.rb:68:in `install'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/type/package.rb:67:in `block (3 levels) in <module:Puppet>'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/property.rb:197:in `call_valuemethod'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/property.rb:498:in `set'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/property.rb:581:in `sync'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:191:in `sync'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:174:in `apply'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:187:in `eval_resource'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `call'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block (2 levels) in evaluate'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:327:in `block in thinmark'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
2015-09-22T09:36:46.358112+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:326:in `thinmark'
2015-09-22T09:36:46.358526+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block in evaluate'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `block in apply'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `block in apply_catalog'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block (2 levels) in run'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block in run'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
2015-09-22T09:36:46.359600+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'
2015-09-22T09:36:46.359600+00:00 err: /usr/bin/puppet:4:in `<main>'

Build info:
{"build_id": "295", "build_number": "295", "release_versions": {"2014.1.3-5.2.9": {"VERSION": {"build_id": "2015-08-25_11-18-06", "ostf_sha": "9d4392fb97252c84dee7d29ee33c706d78b5bbef", "build_number": "64", "api": "1.0", "nailgun_sha": "2f32c02b4efa63431dcabb542417ad8aaa06a208", "tempest_sha": "adcbf964d60194e530b4c0b8d4d805c1b5827ba4", "rally_sha": "f389628a065ea53745fca840ee08f5b1de903aba", "production": "docker", "fuelmain_sha": "57a1d1fb6b9181e507cbe14a04fdd84887547486", "astute_sha": "103c76dcd7a79398480307457886e8be543968ed", "feature_groups": ["mirantis"], "release": "5.2.9", "fuellib_sha": "d3cd6c95fa199852f7e4ad4d85527065705be307"}}, "2015.1.0-7.0": {"VERSION": {"build_id": "295", "build_number": "295", "api": "1.0", "fuel-library_sha": "8e9a9ae51abbbd4edef1432809311004461eec94", "nailgun_sha": "16a39d40120dd4257698795f12de4ae8200b1778", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "2864459e27b0510a0f7aedac6cdf27901ef5c481", "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045", "release": "7.0", "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"}}, "2014.2-6.0": {"VERSION": {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}}, "2014.2.2-6.1": {"VERSION": {"build_id": "2015-06-29_20-18-55", "build_number": "18", "api": "1.0", "fuel-library_sha": "2e7a08ad9792c700ebf08ce87f4867df36aa9fab", "nailgun_sha": "99ec40817eb75510b13c00206e2698d47b5f86aa", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "1ea8017fe8889413706d543a5b9f557f5414beae", "fuel-ostf_sha": "8fefcf7c4649370f00847cc309c24f0b62de718d", "release": "6.1", "fuelmain_sha": "a3998372183468f56019c8ce21aa8bb81fee0c2f"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "8e9a9ae51abbbd4edef1432809311004461eec94", "nailgun_sha": "16a39d40120dd4257698795f12de4ae8200b1778", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "2864459e27b0510a0f7aedac6cdf27901ef5c481", "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045", "release": "7.0", "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"}

[Vladimir Kozhukalov (kozhukalov)]

formatted json build info: http://paste.openstack.org/show/473523/

[Vladimir Kozhukalov (kozhukalov)]

We need diagnostic snapshot from your env, but at least we need to know the following:
1) What were your steps prior to using fuel-createmirror?
2) How were repos configured for your cluseter? Did they point to the repository that was created by fuel-createmirror?
3) ls -l /var/www/nailgun

[Eugene Bogdanov (ebogdanov)]

Oleg, can you please clarify user impact:

- What particular scenarios this bug is reproduced with? Is it only valid after node was upgraded? Any other special conditions?
- How likely user is to encounter this issue - is it a common use case?

[Vladimir Kozhukalov (kozhukalov)]

It is still unclear if it is quite specific to the case used by Oleg or it is more generic. Oleg gave me access to his env where he experienced the issue. I'll try to find out the root cause of the issue till tomorrow.

[Vladimir Kozhukalov (kozhukalov)]

The root cause of the issue is as follows:

We upgrade/install (does not matter) the master node using tarball/iso and it is supposed that MOS repo is placed on the node into /var/www/nailgun/2015.1-7.0. Tarball used for the env was built not later than 14 Sep 2015 which means that MOS repo does not contain libapache2-mod-wsgi package. 'fuel-createmirror -M' command downloads only MOS updates, not MOS itself. It is not supposed to touch /var/www/2015.1-7.0 directory. In turn the command 'fuel-createmirror -U' tries to create partial upstream mirror
using online MOS [1] to find out if a particular package needs to be downloaded or not. I.e. if a package is available in online MOS, then it must not be downloaded from upstream. Then we try to deploy env with partial upstream (w/o mod wsgi package) + local MOS (w/o mod wsgi package).

I'd say it is not even a bug of fuel-createmirror, it is a bug of the whole approach how we deal with repos. We definitely need to deal with all repos the same way, no matter if it is upstream or mos. This thread is exactly about this [2]

[1] https://github.com/stackforge/fuel-mirror/blob/stable/7.0/util/partial_ubuntu.sh#L48-L61
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-September/073843.html

I don't think we need to close this bug. But it is not critial, so for 7.0 it won't be fixed. For 8.0 I have strong opinion that we need to change the whole approach how we treat repos.