Deployment fails with Zabbix plugin used on ubuntu local mirror, missing dependencies

[Denis Ipatov]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "525"
  build_id: "2015-06-19_13-02-31"
  nailgun_sha: "dbd54158812033dd8cfd7e60c3f6650f18013a37"
  python-fuelclient_sha: "4fc55db0265bbf39c369df398b9dc7d6469ba13b"
  astute_sha: "1ea8017fe8889413706d543a5b9f557f5414beae"
  fuel-library_sha: "2e7a08ad9792c700ebf08ce87f4867df36aa9fab"
  fuel-ostf_sha: "8fefcf7c4649370f00847cc309c24f0b62de718d"
  fuelmain_sha: "a3998372183468f56019c8ce21aa8bb81fee0c2f"

Fuel Zabbix monitoring plugin 1.0.0 installs Zabbix in HA on controllers. So there is no separate node with zabbix role. Puppet logs from the controllers display the actual error message I posted while opening the ticket.

This issues is a zabbix fuel plugin bug and needs to be entered in launchpad.

Here is the root cause analysis and possible workaround.

Symptoms
Deployment fails with the following line in puppet logs on the primary controller:

2015-08-11 00:55:52ERR (/Stage[main]/Plugin_zabbix::Controller/Package[zabbix-server-mysql]/ensure) zabbix-server-mysql : Depends: dbconfig-common (>= 1.8.19) but it is not installable
Steps to reproduce
Create local mirror for MOS and Ubuntu on Fuel Master:

[root@fuel nailgun]# fuel-createmirror
Download the Fuel Zabbix Monitoring plug-in from Mirantis plugins repository and install it:

  [root@fuel nailgun]# cd /tmp
  [root@fuel tmp]# wget http://plugins.mirantis.com/repository/z/a/zabbix_monitoring/zabbix_monitoring-1.0.0.noarch.rpm
  [root@fuel tmp]# fuel plugins --install /tmp/zabbix_monitoring_1.0.0.noarch.rpm
Create the environment in Fuel. You need to use 3 controllers setup (HA), otherwise the plug-in doesn't work. Networking, etc. settings do not matter.
On the Settings tab, check Zabbix check-box.
Click "Deploy" button and wait until the primary controller goes through Deploying OpenStack status

Root Cause
So in steps to reproduce, I mentioned creating a local repositories on master node first using fuel-createmirror command

fuel-createmirror takes the list of packages to add to ubuntu-partial local mirror from /opt/fuel-createmirror-6.1/config/requirements-deb.txt

The dbconfig-common is not explicitly called out there:

[root@fuel config]# cat /opt/fuel-createmirror-6.1/config/requirements-deb.txt | grep -c dbconfig
0
And it's not a dependency for any other package mentioned there since after running the fuel-createmirror command it's not available in the mirror:

[root@fuel nailgun]# ls -l /var/www/nailgun/ubuntu-part/pool/main/ | grep -c dbconfig
0
Installing zabbix_monitoring plugin (fuel plugins --install /tmp/zabbix_monitoring-1.0.0.noarch.rpm)
adds only the following packages:

[root@fuel nailgun]# ls -l /var/www/nailgun/plugins/zabbix_monitoring-1.0/repositories/ubuntu/
total 4532
-rw-r--r-- 1 root root 1349 Jul 1 13:52 Packages.gz
-rw-r--r-- 1 root root 252 Jul 1 13:52 Release
-rw-r--r-- 1 root root 321322 Jul 1 13:52 zabbix-agent_2.4.4-1+trusty_amd64.deb
-rw-r--r-- 1 root root 2723602 Jul 1 13:52 zabbix-frontend-php_2.4.4-1+trusty_all.deb
-rw-r--r-- 1 root root 1582564 Jul 1 13:52 zabbix-server-mysql_2.4.4-1+trusty_amd64.deb
And does not modify requirements-deb.txt, so re-running fuel-createmirror command will not give any benefits.

Package zabbix-server-mysql_2.4.4-1+trusty_amd64.deb is installed on the primary controller first with plug-in puppet manifests using regular apt-get install method. Apt is trying to resolve the dependencies and finds the unmet dependency of dbconfig-common, the partial ubuntu mirror that was created on fuel master doesn't have this package, so the corresponding task fails, causing the whole deployment to fail.

Manual workaround without the plug-in re-build
Edit /opt/fuel-createmirror-6.1/config/requirements-deb.txt to add dbconfig-common on the separate line in the end.

[root@fuel config]# cat /opt/fuel-createmirror-6.1/config/requirements-deb.txt | grep dbconfig
dbconfig-common
Re-run fuel-createmirror for ubuntu repository only (!!!Caution: see note after the command))

[root@fuel config]# fuel-createmirror --ubuntu
!!!Caution: by default fuel-createmirror wipes out the ubuntu local mirror before performing the operation, which may lead to loosing any manual changes to mirror or unintentional update of other packages. I believe this is controlled by /opt/fuel-createmirror-6.1/config/ubuntu.cfg setting DISABLE_CLEANUP="no", but I haven't actually tested disabling the cleanup myself.

After fuel-createmirror is finished, verify that the package is present:

[root@fuel nailgun]# ls -l /var/www/nailgun/ubuntu-part/pool/main/ | grep dbconfig
-rw-r--r-- 1 root root 467850 Aug 12 03:13 dbconfig-common_1.8.47+nmu1_all.deb
If you are using image based provisioning (default option in 6.1), for changes to take effect you have to force creating new nodes images for provisioning:
* Either create new environment
* Or manually delete image files from fuel master:

# Get ID of the environment
[root@fuel nailgun]# fuel environment list
id | status | name | mode | release_id | pending_release_id
---|------------|--------|------------|------------|-------------------
8 | deployment | Zabbix | ha_compact | 2 | None
# ID of the environment is 8

# Remove all the images for this environment
[root@fuel nailgun]# rm /var/www/nailgun/targetimages/env_8_*

root@node-28:~# less /var/log/puppet.log
2015-08-12 05:23:02 +0000 /Stage[main]/Plugin_zabbix::Server::Config/Plugin_zabbix_configuration_import[Template_App_OpenStack_Ceilometer_Compute.xml Import] (err): Could not evaluate: undefined method `each' for 10:Fixnum
/etc/fuel/plugins/zabbix_monitoring-1.1/puppet/modules/plugin_zabbix/lib/puppet/provider/plugin_zabbix_configuration_import/ruby.rb:31:in `exists?'

Actually the deployment still fails, but on much later stage, after all the node are in Read status , post-deployment task is querying Zabbix server and gets HTTP500.

root@node-28:~# less /var/log/zabbix/zabbix_server.log
Did not find 'SnmpAdminString' in module #-1 (/usr/share/snmp/mibs/NET-SNMP-AGENT-MIB.txt)
Did not find 'netSnmpObjects' in module NET-SNMP-MIB (/usr/share/snmp/mibs/NET-SNMP-AGENT-MIB.txt)
Did not find 'netSnmpModuleIDs' in module NET-SNMP-MIB (/usr/share/snmp/mibs/NET-SNMP-AGENT-MIB.txt)
Did not find 'netSnmpNotifications' in module NET-SNMP-MIB (/usr/share/snmp/mibs/NET-SNMP-AGENT-MIB.txt)
Did not find 'netSnmpGroups' in module NET-SNMP-MIB (/usr/share/snmp/mibs/NET-SNMP-AGENT-MIB.txt)
Did not find 'DisplayString' in module #-1 (/usr/share/snmp/mibs/NET-SNMP-AGENT-MIB.txt)
Did not find 'RowStatus' in module #-1 (/usr/share/snmp/mibs/NET-SNMP-AGENT-MIB.txt)
Did not find 'TruthValue' in module #-1 (/usr/share/snmp/mibs/NET-SNMP-AGENT-MIB.txt)

[Polina Petriuk (ppetriuk)]

Here is the full list of package names I had to add to /opt/fuel-createmirror-6.1/config/requirements-deb.txt to make deployment of MOS6.1 successful:

dbconfig-common
snmpd
snmp-mibs-downloader
php5-mysql
iptstate

[Swann Croiset (swann-w)]

the fix is here: https://review.openstack.org/#/c/241275/