Update custom-scripts configuration for change-escalator script

[Alexander Lomski]

Need to update custom-scripts server in Fuel Infra for new change-escalator script.

Script task: https://bugs.launchpad.net/fuel/+bug/1528350
Script gerrit: https://review.fuel-infra.org/16196

Script-related Jenkins job update: https://review.fuel-infra.org/16416

Requirements:

1. Script needs to connect to review.openstack.com Gerrit server via SSH. As such, it needs a gerrit user, private key and known_hosts file.
2. Script needs to send escalation emails to Gerrit core reviewers. As such, SMTP server needs to be running where the script can connect to and send letters.
3. All required files should be added to custom-scripts repository. Some of them might already be present since they were required by other scripts.

[Alexander Lomski (aliaksandr-lomski)]

Updated information.

Script has been renamed to `escalate-changes`.

It's Jenkins job expects to see the following structure:

Folder /etc/custom_scripts/escalate-changes/ containing files:

    * review.openstack.org - private key for Gerrit access
    * escalate-changes.conf - modified escalate-changes.conf.example file with all relevant settings configured properly (including SMTP access).

If SMTP access will require key and/or certificate files, they should be put into the same folder and linked from escalate-changes.conf.

[Andrey Nikitin (heos)]

I want to have more additional information about your questions:

1. What the permission for the user do you want to have on review.openstack.org?
2. I suggest to you the following method to send email-notifications: you shouldn't use any smtp-server to send email, but you can use email-notification from a console. For example, you can use 'mail' or 'mailx' tools. You could create special files with information for each request from the job in a directory, which will be available from docker container and jenkins slave is used to run that container. So, when docker container is finished work, job can send the email notifications by using those files with data. What do you think about my suggestion?

[Alexander Lomski (aliaksandr-lomski)]

1. User account needs read-only access to all "openstack/fuel-*" projects, except "openstack/fuel-plugin-*". It needs to retrieve patchset details, revision history and files list.

2. It is possible to fix the script to use mail/mailx, but:

  a. Will it be able to send email to Google Mail addresses like those at mirantis.com? As online guides say: "If you try to send mails from your local computer to a gmail address, your mail would most likely be rejected, so don't try doing that."

  b. Doesn't it still use the SMTP in the end?

[Andrey Nikitin (heos)]

Yes, it will. We're using such scheme to send notifications from our servers in any cases, that's why I have no obstacles here. If you want, we can use any sender-address in 'fuel-infra.org' domain to avoid the problem.

[Alexander Lomski (aliaksandr-lomski)]

Okay. I can fix the script to fallback to mail command if SMTP settings are not defined.

But won't setting SMTP settings to "localhost" without a password do essentially the same thing? Because Python has tools to work with SMTP, but creating a bunch of files and spamming tons of console-imitating subprocesses running mailx with command-line syntax feels like an extremely complicated and useless trick.

Plus, I'm ill-equipped to test this trick even if I manage to implement it.

[Andrey Nikitin (heos)]

Okay, I'm going to create the user to you.

[Alexander Lomski (aliaksandr-lomski)]

What about mail/mailx? Should I implement it or SMTP'ing to local sendmail/postfix is fine?

[Andrey Nikitin (heos)]

I've created the user and credentials to use it on Launchpad.

[Andrey Nikitin (heos)]

Please let me know, when you made a decisions about a method to send email-notifications and assign the current bug on Fuel Devops team again.

[Alexander Lomski (aliaksandr-lomski)]

After some research it's been decided to leave mail sending as SMTP only, at least for now: interfacing with mail/mailx is much more complicated in Python than it is in PHP, for little visible gain.

Both changesets are ready for merging and adding to custom-scripts:

https://review.fuel-infra.org/#/c/16196/
https://review.fuel-infra.org/#/c/16416/

Required credentials for the script:

1. SMTP account to send email from. Local SMTP is acceptable.
2. Script configuration file: /etc/custom_scripts/escalate-reviews/escalate-reviews.conf (use escalate-reviews.conf.example from Gerrit changeset as the base).
3. Configuration file needs SMTP credentials (host, login, password, FROM address). If a certificate and/or private key are required for SMTP server, their paths are specified in configuration file.
4. Private key for Gerrit access to review.openstack.org. Script expects to find it in /etc/custom_scripts/escalate-reviews/review.openstack.org but other path can be specified in configuration file.

[Andrey Nikitin (heos)]

The configs are created on the custom scripts slave.

[Andrey Nikitin (heos)]

The script doesn't work, related the link on a log: https://custom-scripts.infra.mirantis.net/job/escalate-reviews/9/console.

The following files are presented:

root@custom-scripts-slave01:~# ls -lh /etc/custom_scripts/escalate-reviews/
total 8.0K
-rw------- 1 root root 914 Mar 15 12:54 escalate-reviews.conf
-rw------- 1 root root 3.2K Mar 15 09:49 review.openstack.org

The body of 'escalate-reviews.conf' file:

[smtp]
smtp-host = smtp.gmail.com
smtp-port = 587
smtp-user = <email address hidden>
smtp-pass = <hidden_password>
smtp-from = <email address hidden>
smtp-mode = STARTTLS
email-subject = Openstack commit escalations: some changes are in need of review!
email-body-plain = /opt/escalate-reviews/emails/escalate.txt
mail-on-weekends = False

[openstack]
gerrit-host = review.openstack.org
gerrit-port = 29418
gerrit-user = fuel-devops-change-escalator
keyfile = /etc/custom_scripts/escalate-reviews/review.openstack.org
query = project:^openstack/fuel-.* -project:^openstack/fuel-plugin-.* status:open branch:master -label:Workflow<0 -label:Workflow>0 -label:Verified<0 -label:Code-Review<0
escalation-delay = 2
escalation-delay-serious = 4
managers = <email address hidden>
core-team-file = https://raw.githubusercontent.com/{project}/master/MAINTAINERS
guess-core-team = False
guess-core-period = 4w

If you want to test this script with proper credentials, please contact me, I'll send it by a private message.

[Andrey Nikitin (heos)]

Correct link on the log: https://custom-scripts.infra.mirantis.net/job/escalate-reviews/8/console.

[Andrey Nikitin (heos)]

The script is implemented on Custom Scripts infrastructure. The job is working fine: https://custom-scripts.infra.mirantis.net/job/escalate-reviews/14/.

The problem with SMTP authentication was fixed by enabling less secure apps here: https://www.google.com/settings/security/lesssecureapps.

[Andrey Nikitin (heos)]

The fix was found here: http://serverfault.com/questions/635139/how-to-fix-send-mail-authorization-failed-534-5-7-14.