Astute should add astute.yaml symlink right after file upload

[Dmitry Ilyin]

Astute should create /etc/astute.yaml symlink to /etc/${role}.yaml symlink during the pre-deployment hook before the external pre-deployment tasks from the Fuel Library are processed.

diff --git a/lib/astute/deploy_actions.rb b/lib/astute/deploy_actions.rb
index 0d80c47..adfefdf 100644
--- a/lib/astute/deploy_actions.rb
+++ b/lib/astute/deploy_actions.rb
@@ -130,7 +130,8 @@ module Astute
       super
       @actions = [
         EnablePuppetDeploy.new,
- UploadFacts.new
+ UploadFacts.new,
+ ConnectFacts.new
       ]
     end

[Vladimir Sharshov (vsharshov)]

You suggest to link any role yaml to astute.yaml before run any hooks?

[Gil Meir (gilmeir-d)]

I discussed this with @dshulyak, the approach here will probably be to workaround this only in Mellanox plugin, since currently it's the only plugin who needs to read astute.yaml at the pre_deployment stage.

The workaround will be adding in the pre_deployment stage of the plugin a task that will run first, and will link /etc/astute.yaml to a file /etc/<role>.yaml for any random role of this list ['controller', 'primary-controller', 'compute', 'cinder'], since these are the roles supported in Mellanox deployments.

For next version we need to find a prettier solution in astute.

[Evgeniy L (rustyrobot)]

Hi Gil, I'm not sure if prettier way is possible, we have such feature as multiroles, which means the node can have several yaml files, in this case we cannot predict to which file the symlink should be made. Only plugin knows, what file should be used. Also this information is available in the documentation [1].

[1] http://docs.mirantis.com/openstack/fuel/fuel-6.0/plugin-dev.html#why-is-there-no-etc-astute-yaml-file-when-i-run-pre-deployment-task

[Dima Shulyak (dshulyak)]

I think that we can get rid from multiple configuration files, afaik the only reason they were required is that the site.pp was dependent on correct role in astute.yaml (to orchestrate things correctly), but now there is no such dependency and we can send role as array of roles (if necessary at all).

[Gil Meir (gilmeir-d)]

@evgeniyl, if we go according to @dshulyak then we'll be ok. Let's assume for this discussion that we won't.

I'm aware of multirole nodes, and according to the current design I couldn't use only one predefined file (like /etc/astute.yaml) for retrieving the data, but I had to find one of /etc/<role>.yaml for any legal value of <role>.

My first solution for mellanox-plugin was to read an arbirtary /etc/<role>.yaml, but then I got heads up from @dyilin about correctly reading the data model using hiera, which arranges the data according to the hierarchy in /etc/hiera.yaml.
This looked like a better solution which is consistent with your design.

My requirement for the plugin is reading only data from astute.yaml that is common to all <role>.yaml, therefore the plugin doesn't have one point for getting its data.
On one hand you could take out all data which is common to all <role>.yaml to another file, to prevent duplication of data,
and then I would have access to that file which always exists, but then you are not using hiera's advantages of providing one point to access all the data according to the predefined hierarchy.

Currently I have found this workaround for Fuel 6.1:

1. My first pre_deployment task in the plugin's tasks.yaml will be this one:
- role: '*'
  stage: pre_deployment
  type: shell
  parameters:
    cmd: ./link_astute_yaml.py
    timeout: 10

2. This is the code for the python script: http://pastebin.com/DD80MgPV

3. My plugin, which reads astute.yaml and needs to edit it (like changing the storage endpoint https://github.com/stackforge/fuel-web/blob/058d1adef486c116ab8c79379ea6f925db039177/nailgun/nailgun/orchestrator/deployment_serializers.py#L678), has to write these changes, so it writes the changed data to /etc/hiera/override/common.yaml, which is higher than others in hierarchy. The plugin doesn't edit EACH <role>.yaml since this is a better approach leveraging the new design with Hiera.

@dshulyak, @evgeniyl - let's decide on the best solution/workaround we can have for 6.1, I tested mine and seems it's working ok.

[Dima Shulyak (dshulyak)]

Gil, i think that proposed approach is the best way to solve this problem in 6.1.
And we discussed with @evgeniyl that in future releases we will be able to remove multiple files and use only one with array of roles.

[Gil Meir (gilmeir-d)]

This is the work around: https://review.openstack.org/#/c/170787/2/deployment_scripts/link_astute_file.sh
It's currently during code review.

@dshulyak - do you have a bug for 7.0 to merge all <role>.yaml to one yaml file with an array of roles?

[Bogdan Dobrelya (bogdando)]

Setting the bug status to Fix commited as the patch https://review.openstack.org/#/c/170787 was merged

[Alexander Kurenyshev (akurenyshev)]

Steps to verify:
1) Create new cluster (any config)
2) Deploy changes
3) Go to the any node (for ex. to the primary-controller) and perform: readlink /etc/astute.yaml

Expected result:
For the output of the command should be a symlink to the /etc/${role},yaml. In my case it was "/etc/primary-controller.yaml"

Verified on #496 ISO

[Vladimir Sharshov (vsharshov)]

Guys, this review https://review.openstack.org/#/c/170787 contain only workaround for our current astute.yaml workflow:

- https://review.openstack.org/#/c/170787/4/tasks.yaml
- https://review.openstack.org/#/c/170787/4/deployment_scripts/link_astute_file.sh

I mark it as 'Won't fix' for 6.1 and status - Medium, because it is simply to create workaround and now affect limit number of people. But we need to fix this problem in 7.0 and delete hack with link_astute_file.sh.

We need to rethink our workflow with Astute.yaml.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/207488

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/207488
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=94b505752b998fa30505febd00171fc6796e2be4
Submitter: Jenkins
Branch: master

commit 94b505752b998fa30505febd00171fc6796e2be4
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Thu Jul 30 17:39:24 2015 +0300

    Create astute.yaml symlink right after file upload

    This action allow for plugins developers to read some
    nessesary data which common for all roles and perform
    work in pre deployment stage.

    Change-Id: I39c647241386b3be5ddba66173507c27ae19b094
    Closes-Bug: #1430470

[Vladimir Khlyunev (vkhlyunev)]

ISO 298:
Verified on the cluster with detach-services plugins. Steps - https://bugs.launchpad.net/fuel/+bug/1430470/comments/9
http://paste.openstack.org/show/472573/ - all symlinks contains correct path to real config file

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/251886
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=583d68b1cbac03a4f9b6bb9f15921fbbb38159ee
Submitter: Jenkins
Branch: master

commit 583d68b1cbac03a4f9b6bb9f15921fbbb38159ee
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Thu Nov 26 18:17:29 2015 +0300

    Task base deployment

    * async task: puppet, shell, cobbler, sync, noop and reboot;
    * not async: upload file, upload files, copy files;
    * new report format based on task progress instead of log;
    * support union graph deployment: pre/post tasks included in main graph;
    * plugins tasks also included in main graph;
    * support previous failt tollerance strategy (percent of failed nodes).
    * add tests.

    Implements: blueprint task-based-deployment-astute
    Closes-Bug: #1506962
    Closes-Bug: #1430470

    Change-Id: Ibe24d047ab502b88ea0ae9f8c77f94440000d289

[ElenaRossokhina (esolomina)]

Verified on cluster with standalone-rabbit and miltyrole nodes. Links follow to correct files
[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "555"
  build_id: "555"
  fuel-nailgun_sha: "ed2e0cde96ae7bc064e689f7409470e69c57772e"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "33634ec27be77ecfb0b56b7e07497ad86d1fdcd3"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"