config.generator could not handle split configs

[Zhi Yan Liu]

Current config.generator could not handle split configs for the different services within a project, all configurations be collected and save to a single large template file. But for most project, the code repo contains more then one service, like nova repo/project contains nova-api, nova-compute. So the single template file is hard to be used/maintained for separated service. IMO, it will be cool if config.generator could generate separated configs based on service instead of project.

Example of "split configs" is glance-api.conf versus glance-registry.conf

[Ben Nemec (bnemec)]

This is blocking Glance's adoption of the config generator, so we need to figure out some solution here.

[Doug Hellmann (doug-hellmann)]

There are a couple of ways to handle this.

The config generator takes as argument the files and directories to scan. Glance could call it multiple times, passing the right set of files each time.

The generator also has a new feature that lets it load entry points that declare sets of options. This was intended for use in libraries, but could be repurposed for having multiple sets of options registered to different apps. See oslo.messaging for an example of the entry points.

Along with this, Mark asked about how to ensure a service only uses the options it declares it is using. The best way to do that is to register the options at runtime, instead of import time. Doing that ensures that a module can't use an option just because it is being imported as a side-effect of importing another module.

[Doug Hellmann (doug-hellmann)]

Would the -l option to the generator help with this? Glance could define different entry points for each service and then run the generator only specifying those entry points and not using the code-scanning mode. There's an example of setting up the entry points in http://git.openstack.org/cgit/openstack/oslo.messaging/tree/oslo/messaging/opts.py#n56 and http://git.openstack.org/cgit/openstack/oslo.messaging/tree/setup.cfg#n53

[Doug Hellmann (doug-hellmann)]

Is this still an issue for glance?

[Zhi Yan Liu (lzy-dev)]

Hi Doung, yes, I'm taking this for glance and will try -l option approach.

[Zhi Yan Liu (lzy-dev)]

Hello Doung, I tried -l approach with code practice, but it seems it can only controls generate_sample.sh tool whether involving extra library options for discovery, those options which in code base can be generated in any case still. Any suggests?

[Doug Hellmann (doug-hellmann)]

If you want to generate separate config files with only some of the options, you will need to use *only* the -l option and not have the generator scan the code, or at least have it ignore some of the files when doing the scanning. Heat does something similar, so you can look at their tools/config/oslo.config.generator.rc for an example.

[Zhi Yan Liu (lzy-dev)]

Adding oslo-config-generator: https://review.openstack.org/#/c/100946/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/104476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/104476
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=819f28a0b8863bd18f8a14491b5966c8b2723432
Submitter: Jenkins
Branch: master

commit 819f28a0b8863bd18f8a14491b5966c8b2723432
Author: Zhi Yan Liu <email address hidden>
Date: Thu Mar 27 17:39:14 2014 +0800

    Enabling separated sample config file generation

    As a common approach most projects used now and Oslo preferred, this
    change enabled sample configuration file generation mechanism for
    each Glance services.

    This change, as an enhancement, allows generating separated sample
    configuration files for each Glance major services, e.g.:

        etc/glance-api.conf.sample
        etc/glance-cache.conf.sample
        etc/glance-manage.conf.sample
        etc/glance-registry.conf.sample
        etc/glance-scrubber.conf.sample

    It is different than I94d486d6686815c45705a7a9b00fb26062e1eb63
    which only supports generating an unified sample configuration
    file to including all Glance available options.

    This mechanism not only can help auditing by packager, milestone
    maintainer or developer as a function (testenv) of tox, but also
    those separated configuration files could make deployment be easy.
    And it helps keeping sample configuration files be update with
    code change.

    The change added "genconfigs" function as a tox testenv (-egenconfigs),
    and it dependes on oslo-config-generator function of oslo.config.

    The change doesn't introduce those sample files Glance repo, so
    next step is to investigate if we can generate them in gate
    automatically when a change was merged.

    Related-Change-Id: I15686708fc9460948a58cfea3d18dae40ba1fda9
    Related-Change-Id: Iae31856d5886ee78786972d80c7c103c3460a2b3
    Related-Change-Id: I76043b08e2872867e5af2a5ac902e4d092fda5c8
    Closes-Bug: #1300546
    Closes-Bug: #1361963

    Change-Id: Ibe03a3fe80b96ca32acb1a6bea7e38e6075951bb
    Signed-off-by: Zhi Yan Liu <email address hidden>