Fails to translate the derived polices

[Kanagaraj Manickam]

Tried to translate https://github.com/openstack/tacker/blob/master/samples/tosca-templates/vnfd/tosca-vnfd-scale.yaml and it fails to translate the policy SP1.

 while i investigate the issue, found that heat-translator expects the policy as tosca.policies.Scaling type while above VNFD has type as tosca.policies.tacker.Scaling which is derived from tosca.policies.Scaling.

[Sahdev Zala (spzala)]

Hi Kanagaraj,

What I see is in the template you have provided certain custom types doesn't have a definition associated with it or there is no imported template that provides it. For example scaling type. The following test template that I have created has tosca.policies.tacker.Scaling similar to what you have but with added 'policy_types' section and translator is working fine here,

tosca_definitions_version: tosca_simple_yaml_1_0

description: >
  Template for deploying servers based on policies.

policy_types:
  tosca.policies.tacker.Scaling:
    derived_from: tosca.policies.Scaling

topology_template:
  node_templates:
    my_server_1:
      type: tosca.nodes.Compute
      capabilities:
        host:
         properties:
           num_cpus: 2
           disk_size: 10 GB
           mem_size: 512 MB
        os:
         properties:
            # host Operating System image properties
            architecture: x86_64
            type: Linux
            distribution: RHEL
            version: 6.5
  policies:
    - asg:
        type: tosca.policies.tacker.Scaling
        description: Simple node autoscaling
        targets: [my_server_1]
        triggers:
          resize_compute:
            description: trigger
            condition:
              constraint: utilization greater_than 50%
              period: 60
              evaluations: 1
              method: average
        properties:
          min_instances: 2
          max_instances: 10
          default_instances: 3
          increment: 1

Please take a look and let me know if you have question. Thanks!

[Bob Haddleton (bob-haddleton)]

The tacker code automatically imports the tacker-specific definitions files based on the version of the template.

See https://github.com/openstack/tacker/blob/master/tacker/vnfm/tosca/utils.py#L85

https://github.com/openstack/tacker/blob/master/tacker/nfvo/nfvo_plugin.py#L173

https://github.com/openstack/tacker/blob/master/tacker/vnfm/plugin.py#L191

Standalone tosca-parser and heat-translator cannot parse Tacker VNFDs/NSDs without including the associated tacker imports.

[Sahdev Zala (spzala)]

Hi Bob, thanks for the input, it's helpful!! As you noticed, the stand alone tosca-parser requires the type definition which typically is provided either via normative type (as defined in the TOSCA toscaparser/elements/TOSCA_definition_1_0.yaml) or via NFV definition file (toscaparser/extensions/nfv/TOSCA_nfv_definition_1_0_0.yaml) or via custom definition as I provided in my example template above. I wonder when Tacker imports file it might not be provided to the tosca-parser at a time when it's required to resolve custom type - something to take a look in Tacker code? Thanks!

[Sahdev Zala (spzala)]

I tried several things but seems like tosca-parser works fine if type definition is provided. For example, if I update NFV definition file in the tosca-parser with (policy_types:
  tosca.policies.tacker.Scaling:
    derived_from: tosca.policies.Scaling) then without providing any custom types it works fine. Please make sure that Tacker is providing the tosca.policies.Scaling definition to the parser.

[Kanagaraj Manickam (kanagaraj-manickam)]

I have verified from tacker that, it passes the scaling definitions (https://github.com/openstack/tacker/blob/master/tacker/vnfm/tosca/lib/tacker_defs.yaml defined here) as part of ToscaTemplate passed to ToscaTranslator. so Translator should be able to use it , which expects for policy_types. But i think its missing to use.

[Sahdev Zala (spzala)]

OK, cool. Let's continue debug - so for tosca-parser to know the definition it has be via:
1. toscaparser/elements/TOSCA_definition_1_0.yaml
2. toscaparser/extensions/nfv/TOSCA_nfv_definition_1_0_0.yaml
Anything else is considered as custom types and needed to be imported in the service template.
3. *.yaml which is then imported in the service template
4. define types in the service template itself

I have verified all those in my local environment and that seems working fine. You aren't doing #1, #3 or #4 I guess. How do you pass tacker_defs.yaml to parser and expect the parser to handle it? I guess the issue is happening even before heat-translator, can you invoke tosca-parser and see if it passes parsing of it?

[Kanagaraj Manickam (kanagaraj-manickam)]

Sahdev, we have not introduced any recent changes in the way tacker passes the TOSCA custom type definitions to HT and been working fine across many releases, which means the HT is parsing the the TOSCA used in tacker. and we have not changed this logic.

In ToscaTemplate, tacker provides both custom type definitions in addition to the actual template. so I feel that HT should consider the custom types provided as part of ToscaTemplate object in addition to the above mentioned #1 and #2.

Kindly let me know if tacker should change the way it passes the custom types definition to HT?

[Sridhar Ramaswamy (srics-r)]

I concur w/ Kanagaraj's findings.

Custom type definitions is a back bone in making tosca-nfv profile usable. As bobh mentioned earlier we have been doing the imports (#3 above) for a while now and it was working fine.

[Sahdev Zala (spzala)]

Kanagarj and Sridhar, thanks for digging more into it. I totally agree and can't see TOSCA templates without custom types, and that's we have several template examples created and supported in the tosca-parser and HT. As I said above use of #3 requires an "import" statement in the TOSCA service template. For example, https://github.com/openstack/tacker/blob/master/samples/tosca-templates/vnfd/tosca-vnfd-scale.yaml is not importing any 'type definition'. In order to use custom type definition the tosca-parser requires to access it. i.e. 'imports' statement is required in the service template as I mentioned earlier.

I am all up to fix this, if it is in issue with TP or HT with the highest priority but if I can't recreate it and see that custom type support is working fine I am helpless :(.
@Bob, you know tosca-parser and Tacker well. If you can jump on here that would be great.

Here is an example of how the custom type works:

Custom type definition as scalingcustom_nfv.yaml:

tosca_definitions_version: tosca_simple_profile_for_nfv_1_0_0

description: >
  Template for deploying servers based on policies.

policy_types:
  tosca.policies.tacker.Scaling:
    derived_from: tosca.policies.Scaling

Service Template (note the imports statement):

tosca_definitions_version: tosca_simple_profile_for_nfv_1_0_0

description: >
  Template for deploying servers based on policies.

imports:
  - scalingcustom_nfv.yaml

topology_template:
  node_templates:
    my_server_1:
      type: tosca.nodes.Compute
      capabilities:
        host:
         properties:
           num_cpus: 2
           disk_size: 10 GB
           mem_size: 512 MB
        os:
         properties:
            # host Operating System image properties
            architecture: x86_64
            type: Linux
            distribution: RHEL
            version: 6.5
  policies:
    - asg:
        type: tosca.policies.tacker.Scaling
        description: Simple node autoscaling
        targets: [my_server_1]
        triggers:
          resize_compute:
            description: trigger
            condition:
              constraint: utilization greater_than 50%
              period: 60
              evaluations: 1
              method: average
        properties:
          min_instances: 2
          max_instances: 10
          default_instances: 3
          increment: 1

[Sahdev Zala (spzala)]

Please look at above example and my comments. If that doesn't make sense please help me recreate the issue and since I don't have Tacker environment if you can provide where tosca-parser is failing and more detail that would be very helpful to solve this.

[Bob Haddleton (bob-haddleton)]

I'll try to look at this later today. I think the first step is to take the existing template and add the two imports that Tacker adds automatically:

https://github.com/openstack/tacker/blob/master/tacker/vnfm/tosca/lib/tacker_defs.yaml
https://github.com/openstack/tacker/blob/master/tacker/vnfm/tosca/lib/tacker_nfv_defs.yaml

and see if the template can be processed using standalone TP.

Assuming it CAN then the problem is in Tacker. If it cannot, then the problem is likely in TP.

I should be able to try it after lunch my time.

Bob

[Sahdev Zala (spzala)]

@Bob, that sounds like a right approach to go forward. Thanks!! I was on IRC and asked Bharath to help just few minutes back too.

[bharaththiruveedula (bharath-ves)]

@spzala, as per our discussion,the problem is improper ordering of "targets" for scaling resource. The target attribute must be in parallel to "properties", but the template kanagaraj posted contains "targets" inside "properties".

@KanagarajM, please try the below template

tosca_definitions_version: tosca_simple_profile_for_nfv_1_0_0

description: sample-tosca-vnfd-scaling

metadata:
  template_name: sample-tosca-vnfd-scaling

topology_template:
  node_templates:
    VDU1:
      type: tosca.nodes.nfv.VDU.Tacker
      properties:
        image: cirros-0.3.4-x86_64-uec
        mgmt_driver: noop
        availability_zone: nova
        flavor: m1.tiny

    CP1:
      type: tosca.nodes.nfv.CP.Tacker
      properties:
        management: true
        order: 0
        anti_spoofing_protection: false
      requirements:
        - virtualLink:
            node: VL1
        - virtualBinding:
            node: VDU1

    VDU2:
      type: tosca.nodes.nfv.VDU.Tacker
      properties:
        image: cirros-0.3.4-x86_64-uec
        mgmt_driver: noop
        availability_zone: nova
        flavor: m1.tiny

    CP2:
      type: tosca.nodes.nfv.CP.Tacker
      properties:
        management: true
        order: 0
        anti_spoofing_protection: false
      requirements:
        - virtualLink:
            node: VL1
        - virtualBinding:
            node: VDU2

    VL1:
      type: tosca.nodes.nfv.VL
      properties:
        network_name: net_mgmt
        vendor: Tacker

  policies:
    - SP1:
        type: tosca.policies.tacker.Scaling
        targets: [VDU1, VDU2]
        properties:
          increment: 1
          cooldown: 120
          min_instances: 1
          max_instances: 3
          default_instances: 2

[Sridhar Ramaswamy (srics-r)]

catching up a bit, looks you all are further down in narrowing down the issue.

@spzala: quick note on the imports. The imports doesn't show up in tacker's tosca templates (including the sample you refer to [1]) because they happen implicitly. The imports are unconditionally added by the code in [2] & [3] before sending off to TP.

[1] https://github.com/openstack/tacker/blob/master/samples/tosca-templates/vnfd/tosca-vnfd-scale.yaml
[2] https://github.com/openstack/tacker/blob/master/tacker/vnfm/tosca/utils.py#L85
[3] https://github.com/openstack/tacker/blob/master/tacker/vnfm/plugin.py#L189

[Sahdev Zala (spzala)]

@Sridhar: thanks for looking at it and update, I really appreciate it.

AS Bharath commented above, there seems no problem in resolving custom types. The implicit references you mentioned are working fine. About the second issue of translated template not providing expected result is due to the fact that the Tacker template (https://github.com/openstack/tacker/blob/master/samples/tosca-templates/vnfd/tosca-vnfd-scale.yaml) is not correct in providing 'targets' field syntax as expected by TOSCA specification. The 'targets' key name seats at the same level as 'properties' (i.e. it is one of the main sections of policies) but the template has provided it as property so the parser treats it as property and the translation can't make use of it. If you correct template as Bharath mentioned above and as given below snippet, it seems working fine. Bharath has tested it with Tacker.

Correct syntax:
  policies:
    - SP1:
        type: tosca.policies.tacker.Scaling
        targets: [VDU1, VDU2]
        properties:
          increment: 1
          cooldown: 120
          min_instances: 1
          max_instances: 3
          default_instances: 2
Incorrect syntax as provided in the https://github.com/openstack/tacker/blob/master/samples/tosca-templates/vnfd/tosca-vnfd-scale.yaml
  policies:
    - SP1:
        type: tosca.policies.tacker.Scaling
        properties:
          increment: 1
          cooldown: 120
          min_instances: 1
          max_instances: 3
          default_instances: 2
          targets: [VDU1, VDU2]

[Kanagaraj Manickam (kanagaraj-manickam)]

sridhar, sahdev, thanks for continously looking into it.
tbh, thanks for spotting the root cause. i will try it and update the findings here

[Sahdev Zala (spzala)]

@Kanagaraj, no problem! Sure please try it, I know you are going to give it a priority but just a friendly reminder that this is the main issue right now that we all need to make sure we are fine with before TP and HT new release. So I appreciate your priority here. Thanks!

[Kanagaraj Manickam (kanagaraj-manickam)]

@Sahdev, i tried to test it and pasted the generated templates here
http://paste.openstack.org/show/594032/

I have highlighted the issues
1. outputs are missing from provider template for scaling unit
2. invalid Alarm is generated. supposed to be removed when as scaling policy does not include it. (there is an separate policy called AlarmPolicy)

Tacker needs these issues to get fixed from HT. Kindly help. thanks

[Sahdev Zala (spzala)]

@Kanagaraj, thanks!! Sure, looking at it.

[Sahdev Zala (spzala)]

@Kanagaraj, for #1 there is no 'outputs' section in the TOSCA template itself so translator can't create one. Bharath kindly run translator in Tacker env by adding outputs section in the TOSCA template and that will produce outputs in the main template
e.g. Adding outputs in the template as: policies:
    - SP1:
        type: tosca.policies.tacker.Scaling
        targets: [VDU1, VDU2]
        properties:
          increment: 1
          cooldown: 120
          min_instances: 1
          max_instances: 3
          default_instances: 2
  outputs:
    ip_address:
      value: { get_attribute: [ VDU1, public_address ] }

will create outputs in the translated template as:
outputs:
  ip_address:
    value:
      get_attr:
      - VDU1
      - networks
      - private
      - 0

Hope makes sense.

[Sahdev Zala (spzala)]

@Kanagaraj, or #2 to not to unnecessary create Alarm resource I have created this patch https://review.openstack.org/#/c/417240/ If you can please get it in your environment with latest heat-translator clone along with the below patch, hopefully you should be fine.
https://review.openstack.org/#/c/417204/

Thanks!

[Kanagaraj Manickam (kanagaraj-manickam)]

@sahdev, thanks for #2. for #1, i would like HT to provide these details as part of provider template generation, which is required by tacker. and this output should be part of scaling translation to facilitate getting the IP address. and this would be the case for in general, IMO. so when heat-tranaltor command is used to translate it would provide same output.

[Sahdev Zala (spzala)]

@Kanagaraj, np and thank you much for the quick feedback. So are we good with #2?

About #1, sorry but I didn't get the requirement completely. As we know, in my understanding the 'outputs' section which is optional in Heat or TOSCA or any other orchestration in general is a way to display some information to the user of template by an author of template.
Per original design in translator, if the input template has 'outputs' section the translator reads it and try to produce an available mapping in Heat. I am good changing/enhance this per Tacker's requirement but can you please provide more detail, ideally a blueprint with a small example that we can use to understand it better and later to test. Also, I believe this work probably is time consuming and can impact other other test templates we have too which you may agree with me too, can we consider this as a new feature that can be made available in 0.8.0 (not in 0.7.0 which is about to release). I am good on releasing 0.8.0 or 0.7.1 sooner than later when this feature, along with any other near term need from Tacker, is available. I sincerely hope you would agree with me here. Thanks!

[Kanagaraj Manickam (kanagaraj-manickam)]

@sahdev, I tried to use the new Python API you have introduced for translate and its generating the HOT with following invalid entries in place:
  VDU2:
    type: OS::Nova::Server
    properties:
      user_data_format: SOFTWARE_CONFIG
      availability_zone: nova
      image: cirros-0.3.4-x86_64-uec
      flavor: m1.tiny
      networks:
      - port: '{ get_resource: CP2 }'
      config_drive: false

Here the port part is having invalid char '. which is causing the template validate failure. so can you please fix it?

Regarding the requirments mentioned above:
Tacker mainly looking for the IP address of the VM created as part of stack. so can you please add output section. similar to below:

{'mgmt_ip-VDU2': 'CP2', 'mgmt_ip-VDU1': 'CP1'}

Here the VDU and CP are self explanatory. in place of CP, pls provide the IP address of VUDs

[Kanagaraj Manickam (kanagaraj-manickam)]

@sahdev, it seems now HT broken the alarm policy translation.

pls refer http://paste.openstack.org/show/594279/

kindly check it once.

[Sahdev Zala (spzala)]

@Kanagaraj, thanks! sure I will be looking at it today on both issues - ' char and #594279 About output translator can't add one if template doesn't have it, again something we need more time to look into this requirement. Hope you agree. Thanks!

[Sahdev Zala (spzala)]

@Kanagaraj, can you please provide expected alarm translation for #594279? The scaling translation support was created with simple use case with single policy with or without triggers. This one seems a advance one with multiple policies. Thanks!

[Sahdev Zala (spzala)]

@Kanagaraj, also if your schedule allows and you can contribute to code in heat-translator that would be very welcome too! The project will be benefited with your expertise!

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/418114

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418114
Committed: https://git.openstack.org/cgit/openstack/heat-translator/commit/?id=40485206458722833f94995e1950a855eb5a995e
Submitter: Jenkins
Branch: master

commit 40485206458722833f94995e1950a855eb5a995e
Author: sahdev zala <email address hidden>
Date: Mon Jan 9 16:07:17 2017 -0500

    Remove invalid char ' from translated nested autoscaling template

    The translation of autoscaling template requires to produce nested templates,
    which currently injecting invalid char in some properties.

    Change-Id: I60a770694c39cc364d3b7a8507ee791ebbdf3507
    Related-Bug: 1652928

[Kanagaraj Manickam (kanagaraj-manickam)]

@sahdev, thank you for quick fix on removing the invalid char.

I tired to test the http://paste.openstack.org/show/594413/ and it has two scaling policy and it seems HT is failing. its mandatory for tacker in ocata release for supporting selective vdu scaling.
can you please plan HT release to support tacker req.

To support your recent version deliverable, i hope below strategy would help:
currently we have HT version in tacker as heat-translator>=0.4.0
And if your new version 0.7.0 breaks then we could push one patch to exclude this version.

Thank you for actively supporting.

[Sahdev Zala (spzala)]

@Kanagaraj, you are welcome and thank you too for flexibility. We will give priority of making needed enhancements to support your Tacker use cases and make another release (0.7.1 or 0.8.0) during the Ocata cycle so that Tacker can consume in Ocata release. The 0.7.0 should be out this week.

On a side, I sent email to you and Sridhar this morning on current limited support for policies in parser and HT with some detail on it of why so. I believe the 0.6.0 of HT doesn't support this use case either, so Tacker should be able to use 0.7.0 with the current limited support (which will be better than 0.6.0 as we now added nested template support and APIs etc) without skipping the planned 0.7.0 release, unless it's needed to to skip it. Thanks again!

[Sahdev Zala (spzala)]

@Kanagaraj, can you please provide ASAP an expected HOT for http://paste.openstack.org/show/594413/ template you have mentioned above? That will help with HT coding. Thanks!