ceph integration's failure to parse invalid JSON in NodeDataLookup error message should be more helpful

Bug #1784967 reported by John Fulton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Low
John Fulton

Bug Description

When using NodeDataLookup [1], JSON is pasted into THT and it's possible that the user might supply invalid JSON [2]. In this scenario, the error handling of the tripleo-common workbook ceph-ansible.yaml could provide a more helpful error message. When investigating the Mistral task which failed, only a message about being unable to evaluate a YAQL expression is seen. Because this happens when JSON from node data look up is parsed, it would be more helpful to have an on-error statement in the parse_node_data_lookup task and report that the JSON from NodeDataLookup was not valid as it couldn't be parsed. This would then direct the user to update the JSON under NodeDataLookup in the THT.

[stack@b08-h02-r620 workbooks]$ mistral task-get 18d0daf2-5191-4948-93bb-9cc0b91e9f73
...
| Name | parse_node_data_lookup |
| State info | Failed to handle action completion [error=Can not evaluate YAQL expression [expression=json_parse($.node_data_lookup), error=Expecting property name enclosed in double quotes: line 1 column 3 (char 2), data={}], wf=tripleo.storage.v1.ceph-install, task=parse_node_data_lookup, action=std.noop]: |
| | YaqlEvaluationException: Can not evaluate YAQL expression [expression=json_parse($.node_data_lookup), error=Expecting property name enclosed in double quotes: line 1 column 3 (char 2), data={}] |
...

[1] https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/node_specific_hieradata.html

[2]
parameter_defaults:
  NodeDataLookup: |
    { # c05-h01-6048r is missing scsi-0:2:35:0 <--------- this comment makes the JSON invalid
    "00000000-0000-0000-0000-0CC47A6EFD0C": {"devices": [
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:1:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:32:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:2:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:3:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:4:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:5:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:6:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:33:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:7:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:8:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:34:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:9:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:10:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:11:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:12:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:13:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:14:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:15:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:16:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:17:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:18:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:19:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:20:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:21:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:22:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:23:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:24:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:25:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:26:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:27:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:28:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:29:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:30:0",
    "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:31:0"
        ]
      }
    }

tags: added: queens-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (master)

Fix proposed to branch: master
Review: https://review.openstack.org/588109

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/588112

Revision history for this message
Alex Schultz (alex-schultz) wrote :
Revision history for this message
Alex Schultz (alex-schultz) wrote :

https://docs.openstack.org/heat/latest/template_guide/hot_spec.html#parameters-section json is a valid type so we would have the validation earlier on in the deployment

Revision history for this message
John Fulton (jfulton-org) wrote :

Thanks Alex. Next steps:

- When https://review.openstack.org/#/c/588109 is working, land it and backport it to queens
- Submit new change in THT docker/services/ceph-ansible/ceph-base.yaml switching the type to JSON but don't backport for backwards compatibility reasons

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/588640

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/588654

Changed in tripleo:
milestone: rocky-rc1 → stein-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.openstack.org/588109
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=5b1c4c00eea2242b82bd68498b45b7f65adbf811
Submitter: Zuul
Branch: master

commit 5b1c4c00eea2242b82bd68498b45b7f65adbf811
Author: John Fulton <email address hidden>
Date: Thu Aug 2 00:23:57 2018 +0000

    Add on-error task to parse_node_data_lookup

    If the ceph-ansible workbook fails to parse node_data_lookup
    because it does not contain valid JSON, then fail the task
    with a message that the NodeDataLookup JSON isn't valid.

    Change-Id: I4b05933bd8b24119ed2d1040af60b4b3a2e3140f
    Closes-Bug: #1784967

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/607695

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/607840

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/queens)

Reviewed: https://review.openstack.org/607695
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=814bd2cf4b4bb393cb7af1ea8587c9573c5edcef
Submitter: Zuul
Branch: stable/queens

commit 814bd2cf4b4bb393cb7af1ea8587c9573c5edcef
Author: John Fulton <email address hidden>
Date: Thu Aug 2 00:23:57 2018 +0000

    Add on-error task to parse_node_data_lookup

    If the ceph-ansible workbook fails to parse node_data_lookup
    because it does not contain valid JSON, then fail the task
    with a message that the NodeDataLookup JSON isn't valid.

    Change-Id: I4b05933bd8b24119ed2d1040af60b4b3a2e3140f
    Closes-Bug: #1784967
    (cherry picked from commit 5b1c4c00eea2242b82bd68498b45b7f65adbf811)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/rocky)

Reviewed: https://review.openstack.org/607840
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=d709699b4cace989007344b69c67fe59d57c935b
Submitter: Zuul
Branch: stable/rocky

commit d709699b4cace989007344b69c67fe59d57c935b
Author: John Fulton <email address hidden>
Date: Thu Aug 2 00:23:57 2018 +0000

    Add on-error task to parse_node_data_lookup

    If the ceph-ansible workbook fails to parse node_data_lookup
    because it does not contain valid JSON, then fail the task
    with a message that the NodeDataLookup JSON isn't valid.

    Change-Id: I4b05933bd8b24119ed2d1040af60b4b3a2e3140f
    Closes-Bug: #1784967
    (cherry picked from commit 5b1c4c00eea2242b82bd68498b45b7f65adbf811)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 10.0.0

This issue was fixed in the openstack/tripleo-common 10.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (stable/queens)

Change abandoned by John Fulton (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/588654
Reason: was just a test for a depends-on review which now merged

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by John Fulton (<email address hidden>) on branch: master
Review: https://review.openstack.org/588640
Reason: can't just switch this to json

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by John Fulton (<email address hidden>) on branch: master
Review: https://review.openstack.org/588112
Reason: was a DNM patch for testing depends-on patch which is now merged.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/588640
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=fe2fda491b1ed97dcc77b3bf3f2df1df7b94493b
Submitter: Zuul
Branch: master

commit fe2fda491b1ed97dcc77b3bf3f2df1df7b94493b
Author: John Fulton <email address hidden>
Date: Fri Aug 3 14:43:10 2018 -0400

    Change NodeDataLookup type from string to json

    The NodeDataLookup parameter should be valid JSON and we should
    help the user by adding type checking for this early in the
    deployment so that the user doesn't experience the related bug.

    Change-Id: Id9d2208f1cbaba9234d7657420cd7efcad3507a0
    Related-Bug: #1784967
    Related-Bug: #1814070

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/634978

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.openstack.org/634978
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=65111909cc2370b77a31fe163d82c2315108b4b4
Submitter: Zuul
Branch: stable/rocky

commit 65111909cc2370b77a31fe163d82c2315108b4b4
Author: John Fulton <email address hidden>
Date: Fri Aug 3 14:43:10 2018 -0400

    Change NodeDataLookup type from string to json

    The NodeDataLookup parameter should be valid JSON and we should
    help the user by adding type checking for this early in the
    deployment so that the user doesn't experience the related bug.

    Change-Id: Id9d2208f1cbaba9234d7657420cd7efcad3507a0
    Related-Bug: #1784967
    Related-Bug: #1814070
    (cherry picked from commit fe2fda491b1ed97dcc77b3bf3f2df1df7b94493b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 8.6.7

This issue was fixed in the openstack/tripleo-common 8.6.7 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 9.5.0

This issue was fixed in the openstack/tripleo-common 9.5.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.