customise pipeline.yaml further

[Arif Ali]

Hi,

As of mitaka the pipeline-yaml configuration option does not work, and therefore users are not able to specify their own customised pipeline.

A new option was added to add remote-sink, which allowed to add multiple remote sinks to the configuration.

I am proposing 2 things

1. Update the config.yaml to update the pipeline-yaml config option, so that it says that this option is only supported pre-mitaka

2. The user in question was trying to use pipeline-yaml to add a different sink other than remote or event sinks, i.e. meter_sink. So I am suggesting we add a new configuration option i.e. meter-sink which would add something very similar to what remote-sink does.

Below is an excerpt from what the user was trying to attempt with the pipeline-yaml configuration option

        sources:
          - name: 'meter source'
            meters:
             - '*'
            sinks:
             - 'meter_sink'
        sinks:
          - name: 'meter_sink'
            transformers:
            publishers:
              - http://X.X.X.X:8080/telegraf?timeout=10&max_retries=2&batch=False

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceilometer/+/804127

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceilometer/+/804127
Committed: https://opendev.org/openstack/charm-ceilometer/commit/6ed691807de73b49f057a813b11ebb67bb517782
Submitter: "Zuul (22348)"
Branch: master

commit 6ed691807de73b49f057a813b11ebb67bb517782
Author: Arif Ali <email address hidden>
Date: Tue Aug 10 19:21:05 2021 +0100

    Add meter-sink as an option

    If the user wants to add a meter-sink to the pipeline.yaml
    at the moment this cannot be done. Previously, pipeline-yaml
    config option was allowed and could replace the whole config.

    This will now allow to also add meter-sink as an option to
    the event_pipeline.yaml, and allow a new source of collecting
    all meters, and pushing to the value of meter-sink URLs.

    Closes-Bug: 1938807
    Change-Id: If6cd82e4153b29c1d8fdb6df150d0d4bb11d4e49

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceilometer/+/816150

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceilometer/+/816150
Committed: https://opendev.org/openstack/charm-ceilometer/commit/f4d772e6a3af21d7917aec60e8ab8b93eaba3688
Submitter: "Zuul (22348)"
Branch: master

commit f4d772e6a3af21d7917aec60e8ab8b93eaba3688
Author: Arif Ali <email address hidden>
Date: Mon Nov 1 08:05:32 2021 +0000

    Enable pipeline_yaml for mitaka and above

    This configuration for pipeline.yaml is still valid for mitaka and
    above, so adding this in.

    The ceilometer-collector package is not available in newer openstack
    releases, so basing this on QUEENS_SERVICES.

    This should solve the custom requirement of pipeline moving forward.

    Other fixes:
    * Update test-requirements.txt for cffi and related pins.
    * Update focal-xena bundle to fix memcached series bionic requirement.

    Closes-Bug: 1938807
    Change-Id: I1946b6f172cdae6f444d07999d5e54d4e11a450e

[Aurelien Lourot (aurelien-lourot)]

The first (incorrect) patch 'Add meter-sink as an option' is now getting reverted everywhere (`master` and `stable/21.10`):
https://review.opendev.org/c/openstack/charm-ceilometer/+/816124

The new patch 'Enable pipeline_yaml for mitaka and above' https://review.opendev.org/c/openstack/charm-ceilometer/+/816150 will probably not be backported to `stable/21.10`

[Aurelien Lourot (aurelien-lourot)]

Backport of the revert: https://review.opendev.org/c/openstack/charm-ceilometer/+/816874

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-ceilometer/+/886683

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-ceilometer/+/886684

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-ceilometer/+/886685

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-ceilometer/+/886686