diskimage-create -p hdp -i centos -v 2 tries to install packages that do not exist

[Matthew Farrellee]

during 'elements/hadoop-hdp/install.d/40-install-hdp' the following output is provided

No package hadoop-native available.
No package hadoop-pipes available.
No package hadoop-sbin available.
No package mysql-client available.
No package hdp_mon_dashboard available.

it's unclear if this prevents the resulting disk image from working (because i've not been able to successfully create a disk image)

[Robert Nettleton (rnettleton)]

I'll look into this.

Could the filer please post the actual error encountered while trying to generate the images?

I've been using:

Ubuntu 12.0.4 LTS

without any issues.

[Matthew Farrellee (mattf)]

the output is in the bug description

[Robert Nettleton (rnettleton)]

1. Is there any other output when the image generation fails?

2. Could the filer provide some info on the version of Fedora/Red Hat used in the failing case?

[Matthew Farrellee (mattf)]

1. this isn't about image generation failing, it's about the code to install hadoop trying to install packages that aren't available.
2. disimage-create.sh was run on f20 and el6, but the host filesystem should not be relevant here. this issue is coming from a chroot portion of the build.

[Robert Nettleton (rnettleton)]

I've run the latest HDP image generation scripts on Ubuntu, and the image generation completes successfully.

I am seeing the same "No Package Available" messages. I'm not sure why this is occurring, but I don't think this should affect the image itself. In most cases, Ambari will still be able to install these packages at cluster provisioning time.

That being said, I'll investigate this further, to determine why the packages are not installed. It is possible that a repo is not configured correctly before the DIB generation process starts.

As you mentioned, this problem seems to occur across platforms.

The message only seems to occur during the generation of the HDP 2.0.6 image, so perhaps this has something to do with the repos used when the install-packages calls are made.

I'll dig into this further.

[Matthew Farrellee (mattf)]

the yum install line in elements/hadoop-hdp/install.d/40-install-hdp is the same for HDP1 and HDP2. it may be that in HDP2 those packages have different names or are unnecessary.

afaik the goal of the images w/ HDP pre-installed is to allow for running on clusters where network access is not available. so relying on ambari to finish the package install is not a good thing.

[Robert Nettleton (rnettleton)]

I've investigated the package names mentioned below, and HDP 2.0.6 does not require these packages. The error reported is just a cosmetic error, and should not affect the correctness of the HDP image produced by diskimage-builder.

I'm testing out a patch fix for this issue, and hope to submit it sometime over the next few days.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/81630

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/81630
Committed: https://git.openstack.org/cgit/openstack/sahara-image-elements/commit/?id=947cb1d13ec73ea28528026155ed49dbb4fdbfbd
Submitter: Jenkins
Branch: master

commit 947cb1d13ec73ea28528026155ed49dbb4fdbfbd
Author: Bob Nettleton <email address hidden>
Date: Wed Mar 19 16:59:57 2014 -0400

    Removes unnecessary/incorectly-named packages from HDP install script

    This patch fixes bug 1292519. Some install packages were listed
      that were not available in the repositories used for the HDP
      image installs.
    Some of these packages were only valid for HDP 1.3, and so have
      been removed from the HDP 2.0 install commands.
    Some of the packages listed in the HDP 1.3 install were only valid
      in HDP 2.0, so these package names have been removed from the
      HDP 1.3 install.
    Some packages in the HDP 1.3 install commands were incorrectly named
      and not actually required by HDP 1.3, so this patch removes those
      package names.
    The commands used to install mysql have been refactored into a
      common method, which is now used by both the HDP 1.3 and 2.0
      installs.
    This patch also removes other unavailable packages that were listed
      in warnings during the HDP image generation process. After some
      investigation, it was determined that these packages were either
      incorrectly referenced, or no longer needed for HDP.

    Fixes: bug # 1292519

    Change-Id: I77eddf4497116d578720e9aaacba386333789757