Using galera_my_cnf_overrides brakes cluster config

[Daniel Marks]

TL;DR: When using galera_my_cnf_overrides the includedir line in my.cnf is treated as a comment and gets removed by the config_template module.

I discovered that while troubleshooting a galera setup problem, where I used galera_my_cnf_overrides in my user_variables.

galera_my_cnf_overrides:
  sst:
    sst-initial-timeout: 600

Only to find that it made the problem even worse. After some further troubleshooting I realized that something removed a quiet important line from my my.cnf in the galera lxc container. The line that is missing is:

!includedir /etc/mysql/conf.d/

Which will include other configuration files including those used by wsrep. Bootstrapping a new galera cluster obviously will not work if this line is missing, I have no idea what that change would do to an existing cluster...

The corresponding task can be found in /etc/ansible/roles/galera_server/tasks/galera_post_install.yml

- name: Drop mariadb config(s)
  config_template:
    src: "{{ item.src }}"
    dest: "{{ item.dest }}"
    owner: "root"
    group: "root"
    mode: "0644"
    config_overrides: "{{ item.config_overrides }}"
    config_type: "{{ item.config_type }}"
  with_items:
    - src: my.cnf.j2
      dest: /etc/mysql/my.cnf
      config_overrides: "{{ galera_my_cnf_overrides }}"
      config_type: "ini"
    - src: cluster.cnf.j2
      dest: /etc/mysql/conf.d/cluster.cnf
      config_overrides: "{{ galera_cluster_cnf_overrides }}"
      config_type: "ini"
    - src: debian.cnf.j2
      dest: /etc/mysql/debian.cnf
      config_overrides: "{{ galera_debian_cnf_overrides }}"
      config_type: "ini"
  notify: Restart mysql
  tags:
    - galera-config
    - galera-client-user-config

For me it looks like the "!includedir " line is treated as an comment and gets removed by the config_template module.

[Jean-Philippe Evrard (jean-philippe-evrard)]

There is indeed a gap in our testing, which doesn't include a test of "!".
It should work by default, but we'll confirm it when we include this testing.

Marking as confirmed for that reason.

[Miguel Alejandro Cantu (miguel-cantu)]

There is no indication that ConfigParser omits lines starting with an exclamation point. According to the ConfigParser docs[1], "Lines beginning with '#' or ';' are ignored and may be used to provide comments.". I'm also looking at the ConfigParser.RawConfigParser subclass in config_template[2], and there is no logic written it to ignore lines starting with an exclamation point.

Perhaps this line is being ignored by some other code in the config_template module. Maybe it's happening during the merging process. I'll keep digging.

[1] https://docs.python.org/2/library/configparser.html
[2] https://github.com/openstack/openstack-ansible-plugins/blob/master/action/_v2_config_template.py#L111

[Miguel Alejandro Cantu (miguel-cantu)]

Turns out ConfigParser does indeed ignore the lines that are not section headers or key/value options.[1]

A possible solution would be to add some logic into our custom _read method to preserve these other lines.

[1] https://github.com/python/cpython/blob/2.7/Lib/ConfigParser.py#L471-L472

[Miguel Alejandro Cantu (miguel-cantu)]

I did some more digging, the ConfigTemplateParser subclass is what ends up stripping these lines away. Allow me to explain. When ConfigTemplateParser is run, it processes the rendered template file 'my.cnf' line-by-line. When it reaches the '!includedir /etc/mysql/conf.d/' statement, it processes the entire string(with the space) as an option name with the value set to None.
Since the value of this option is None, unfortunately it gets stripped away in the _write method here:
https://github.com/openstack/openstack-ansible-plugins/blob/master/action/_v2_config_template.py#L158-L160

It's possible to write in some logic here so it doesn't strip these options away, but I would like @cloudnull's and @odyssey4me's opinion before I move forward.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/477010

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/477010
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-plugins/commit/?id=76d5f02a32465028b47347048dde96533595036b
Submitter: Jenkins
Branch: master

commit 76d5f02a32465028b47347048dde96533595036b
Author: Miguel Alex Cantu <email address hidden>
Date: Fri Jun 23 19:32:44 2017 +0000

    Add ignore_none_type to config template

    It is sometimes useful to tell ConfigTemplateParser to write
    out options that are valueless and not suffixed with '=' or ':',
    such as when overriding a my.cnf.

    This commit adds the 'ignore_none_type' attribute to the config_template
    module. If this attribute is set to false, then valueless options will be
    written out to the resultant INI file as-is, without the '=' or ':' suffix.

    Change-Id: I5c88b2019c01b44193a5d0df9299ecce6de52f01
    Partial-Bug: #1693234

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/478207
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-galera_server/commit/?id=34f52a673eee1c635030fab135b36f75867c47d2
Submitter: Jenkins
Branch: master

commit 34f52a673eee1c635030fab135b36f75867c47d2
Author: Miguel Alex Cantu <email address hidden>
Date: Tue Jun 27 15:59:31 2017 +0000

    Add ignore_none_type to config_template

    The .cnf files can have valueless options present.
    This commit ensures that these valueless options are not ignored

    Addtionally, my.cnf overrides are added to the variable overrides
    test file to ensure that the my.cnf file can run through the
    config_template engine properly.

    Change-Id: I8b09c2520c84bb43353b4c56bac157259f71b041
    Closes-Bug: #1693234

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-galera_server 16.0.0.0b3 development milestone.

[Daniel Marks (d3n14l)]

Thank you, Miguel!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/495938

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/495942

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/495938
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-plugins/commit/?id=b4a5996c0e163bf39e399045d372e5877339a72b
Submitter: Jenkins
Branch: stable/ocata

commit b4a5996c0e163bf39e399045d372e5877339a72b
Author: Miguel Alex Cantu <email address hidden>
Date: Fri Jun 23 19:32:44 2017 +0000

    Add ignore_none_type to config template

    It is sometimes useful to tell ConfigTemplateParser to write
    out options that are valueless and not suffixed with '=' or ':',
    such as when overriding a my.cnf.

    This commit adds the 'ignore_none_type' attribute to the config_template
    module. If this attribute is set to false, then valueless options will be
    written out to the resultant INI file as-is, without the '=' or ':' suffix.

    Conflicts:
     tests/test-config_template.yml

    Change-Id: I5c88b2019c01b44193a5d0df9299ecce6de52f01
    Partial-Bug: #1693234
    (cherry picked from commit 76d5f02a32465028b47347048dde96533595036b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/495942
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-galera_server/commit/?id=0211193d6f70d7778cfea13b2fcfad4a44c08a02
Submitter: Jenkins
Branch: stable/ocata

commit 0211193d6f70d7778cfea13b2fcfad4a44c08a02
Author: Miguel Alex Cantu <email address hidden>
Date: Tue Jun 27 15:59:31 2017 +0000

    Add ignore_none_type to config_template

    The .cnf files can have valueless options present.
    This commit ensures that these valueless options are not ignored

    Addtionally, my.cnf overrides are added to the variable overrides
    test file to ensure that the my.cnf file can run through the
    config_template engine properly.

    Conflicts:
     tests/galera_server-overrides.yml

    Change-Id: I8b09c2520c84bb43353b4c56bac157259f71b041
    Closes-Bug: #1693234
    Depends-On: I5c88b2019c01b44193a5d0df9299ecce6de52f01
    (cherry picked from commit 34f52a673eee1c635030fab135b36f75867c47d2)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-galera_server 15.1.9 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/533789

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/533790

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kevin Carter (cloudnull) (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/533789

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kevin Carter (cloudnull) (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/533790