'ntpd' on controllers is configured to access itself as a remote peer

[Dennis Dmitriev]

'ntpd' on all controllers is configured to access the vip__management IP as a remote peer.
Thus, the time on controllers and other remote nodes is never synchronized to external clock source.

root@node-1:~# ip netns exec vrouter ip r | grep vr-mgmt
10.109.12.0/24 dev vr-mgmt proto kernel scope link src 10.109.12.2

root@node-1:~# fgrep 'server' /etc/ntp.conf
server 10.109.12.2 iburst minpoll 3

Reproduced on swarm tests: http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.ubuntu.cic_maintenance_mode/50/

ISO version: {u'build_id': u'2015-08-03_22-04-28', u'build_number': u'132', u'auth_required': True, u'fuel-ostf_sha': u'53109a99d923cccdf88c5cf5aba0af8050df47e3', u'fuel-library_sha': u'1cfd80a833ed27c777c950006a8d4e4080f81616', u'nailgun_sha': u'd1536c3a57459e205e39bc4d86d2b4efc6141c4e', u'openstack_version': u'2015.1.0-7.0', u'fuel-nailgun-agent_sha': u'1512b9af6b41cc95c4d891c593aeebe0faca5a63', u'fuel-agent_sha': u'1fe47720ba554818a0be707f2e16281791492d50', u'api': u'1.0', u'python-fuelclient_sha': u'4fe70fb5c0ce8905ae5908f63d45b45e89a99340', u'astute_sha': u'6d09f3fc7f69ac558095299211ebfd081fa54b8f', u'fuelmain_sha': u'7a374fbd1f5ebde943cb391a4f71b94888ce4a15', u'feature_groups': [u'mirantis'], u'release': u'7.0', u'release_versions': {u'2015.1.0-7.0': {u'VERSION': {u'build_id': u'2015-08-03_22-04-28', u'build_number': u'132', u'fuel-library_sha': u'1cfd80a833ed27c777c950006a8d4e4080f81616', u'nailgun_sha': u'd1536c3a57459e205e39bc4d86d2b4efc6141c4e', u'fuel-ostf_sha': u'53109a99d923cccdf88c5cf5aba0af8050df47e3', u'fuel-nailgun-agent_sha': u'1512b9af6b41cc95c4d891c593aeebe0faca5a63', u'fuel-agent_sha': u'1fe47720ba554818a0be707f2e16281791492d50', u'api': u'1.0', u'python-fuelclient_sha': u'4fe70fb5c0ce8905ae5908f63d45b45e89a99340', u'astute_sha': u'6d09f3fc7f69ac558095299211ebfd081fa54b8f', u'fuelmain_sha': u'7a374fbd1f5ebde943cb391a4f71b94888ce4a15', u'feature_groups': [u'mirantis'], u'release': u'7.0', u'openstack_version': u'2015.1.0-7.0', u'production': u'docker'}}}, u'production': u'docker'}

[Oleksiy Molchanov (omolchanov)]

Strange, the config was changed by task ntp-client.conf that run in post-deploy. But it is not supposed to be executed on controller nodes. Python team, please take a look.

[Andrey Shestakov (ashestakov)]

Works on build #140

node-1 - compute
node-2 - controller

root@node-1:~# fgrep 'server' /etc/ntp.conf
# Specify one or more NTP servers.
# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
# Use Ubuntu's ntp server as a fallback.
# Note that "restrict" applies to both servers and clients, so a configuration
# up blocking replies from your own upstream servers.
# Local users may interrogate the ntp server more closely.
server 10.20.0.2 burst iburst

root@node-2:~# fgrep 'server' /etc/ntp.conf
# Set up servers for ntpd with next options:
# server - IP address or DNS name of upstream NTP server
# prefer - select preferrable server
server 0.fuel.pool.ntp.org iburst minpoll 3
server 1.fuel.pool.ntp.org iburst minpoll 3
server 2.fuel.pool.ntp.org iburst minpoll 3

[Serge Kovaleff (serge-kovaleff)]

@Fuel QA Team Please verify

What is the correct status for it if we didn't fix it?

[Andrey Shestakov (ashestakov)]

looks like fixed at
https://github.com/stackforge/fuel-library/commit/54c510d97db11b216fe2a91884c5c8e3ce02c24a

[Dennis Dmitriev (ddmitriev)]

Still reproduced on ISO with build_number: '139' build_id: 2015-08-05_21-24-26. Let's wait for tests on more fresh ISO.

[Sergey Yudin (tsipa740)]

I have verifyied this bug on

VERSION:
  feature_groups:
    - experimental
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "135"
  build_id: "2015-08-06_06-45-26"
  nailgun_sha: "c842770d56680d0e0cdf3573d329ce36d4fb2c64"
  python-fuelclient_sha: "28ddd022fd230fde0e88f2f3e4a6a6dddcb58abe"
  fuel-agent_sha: "1fe47720ba554818a0be707f2e16281791492d50"
  fuel-nailgun-agent_sha: "1512b9af6b41cc95c4d891c593aeebe0faca5a63"
  astute_sha: "e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc"
  fuel-library_sha: "347d51f391a6f2c621006c00e3416718140889ac"
  fuel-ostf_sha: "5cb828f6200e5ae0da63dd39d11fbf910237db95"
  fuelmain_sha: "858a4ee8336789c10c5c2189c3718f9a605d0eeb"

and it was fixed controller:
Last login: Fri Aug 7 17:08:29 2015 from 10.109.5.2
root@node-6:~# cat /etc/ntp.conf | grep server
server 10.109.6.1 iburst minpoll 3
root@node-6:~# cat /etc/astute.yaml | grep '10.109.6.1$'
        gateway: 10.109.6.1
  dns_list: 10.109.6.1
      gateway: 10.109.6.1
  ntp_list: 10.109.6.1

This bug was fixed by Dmitry Ilyin in https://review.openstack.org/#/c/197629/14/deployment/puppet/osnailyfacter/modular/ntp/tasks.yaml

[Dennis Dmitriev (ddmitriev)]

Confirmed on ISO #150.

Reproduced on CI job: http://jenkins-product.srt.mirantis.net:8080/job/7.0.system_test.ubuntu.thread_3/54/console

root@node-1:~# cat /etc/astute.yaml |grep 'vrouter:' -A4
    vrouter:
      network_role: mgmt/vip
      ipaddr: 10.109.4.2
      node_roles:
      - controller
--
vrouter:
  network_role: mgmt/vip
  ipaddr: 10.109.4.2
  node_roles:
  - controller

root@node-1:~# cat /etc/ntp.conf | grep 'server'
# Set up servers for ntpd with next options:
# server - IP address or DNS name of upstream NTP server
# prefer - select preferrable server
server 10.109.4.2 iburst minpoll 3

root@node-2:~# cat /etc/ntp.conf | grep 'server'
# Set up servers for ntpd with next options:
# server - IP address or DNS name of upstream NTP server
# prefer - select preferrable server
server 10.109.4.2 iburst minpoll 3

root@node-3:~# cat /etc/ntp.conf | grep 'server'
# Set up servers for ntpd with next options:
# server - IP address or DNS name of upstream NTP server
# prefer - select preferrable server
server 10.109.4.2 iburst minpoll 3

[Andrey Sledzinskiy (asledzinskiy)]

Reproduced on swarm, logs are attached

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/211092

[Bogdan Dobrelya (bogdando)]

https://review.openstack.org/211092

[Bogdan Dobrelya (bogdando)]

I believe the https://review.openstack.org/#/c/197629/14/deployment/puppet/osnailyfacter/modular/ntp/tasks.yaml didn't fix the issue as multiple roles still may be applied on controllers. And this executes ntp-client task at controllers.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/211092
Reason: This would have made all nodes to rely on only hiera provided ntp servers, which are normally external ones. While on compute nodes we may want to not rely on external ntp servers

[Timur Nurlygayanov (tnurlygayanov)]

Not reproduced on my environment with MOS 7.0 #132:

root@node-1:~# ip netns exec vrouter ip r | grep vr-mgmt
192.168.0.0/24 dev vr-mgmt proto kernel scope link src 192.168.0.1
root@node-1:~# fgrep 'server' /etc/ntp.conf
server 0.fuel.pool.ntp.org iburst minpoll 3
server 1.fuel.pool.ntp.org iburst minpoll 3
server 2.fuel.pool.ntp.org iburst minpoll 3

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/211194

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/211194
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a4b5e9bb4510563d5bc52ca356a92bb6bab44dfc
Submitter: Jenkins
Branch: master

commit a4b5e9bb4510563d5bc52ca356a92bb6bab44dfc
Author: Oleksiy Molchanov <email address hidden>
Date: Mon Aug 10 17:24:25 2015 +0300

    Restrict ntp-client on controller nodes

    NTP-client task shouldn't be run on controller multirole nodes

    Change-Id: I54defe421ded251ad13b665b6fd1843600a517a7
    Closes-Bug: 1481627

[Oleksiy Molchanov (omolchanov)]

Should be merged to 6.1 with https://bugs.launchpad.net/fuel/6.1.x/+bug/1466090

[Dennis Dmitriev (ddmitriev)]

Released on ISO #160:

[root@nailgun ~]# fuel node| grep 'controller'
3 | ready | slave-03_controller_ceph-osd | 1 | 10.109.0.5 | 64:c8:0d:de:55:23 | ceph-osd, controller | | True | 1
1 | ready | slave-02_controller_ceph-osd | 1 | 10.109.0.4 | 64:03:54:18:2f:d4 | ceph-osd, controller | | True | 1
2 | ready | slave-01_controller_ceph-osd | 1 | 10.109.0.3 | 64:8d:1c:9e:6b:44 | ceph-osd, controller | | True | 1

[root@nailgun ~]# for i in {1..3}; do ssh node-$i cat /etc/ntp.conf|grep server|grep -v "^#"; done
Warning: Permanently added 'node-1' (RSA) to the list of known hosts.
server 10.109.1.1 iburst minpoll 3
Warning: Permanently added 'node-2' (RSA) to the list of known hosts.
server 10.109.1.1 iburst minpoll 3
Warning: Permanently added 'node-3' (RSA) to the list of known hosts.
server 10.109.1.1 iburst minpoll 3

root@node-1:~# cat /etc/astute.yaml |grep ntp
external_ntp:
  ntp_list: 10.109.1.1

[Dennis Dmitriev (ddmitriev)]

api: '1.0'
astute_sha: e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc
auth_required: true
build_id: 2015-08-11_17-24-26
build_number: '160'
feature_groups:
- mirantis
fuel-agent_sha: 57145b1d8804389304cd04322ba0fb3dc9d30327
fuel-library_sha: ecdfcc0b23c4fbeea193e222e6eb5fb1f4b4e68f
fuel-nailgun-agent_sha: e01693992d7a0304d926b922b43f3b747c35964c
fuel-ostf_sha: c7f745431aa3c147f2491c865e029e0ffea91c47
fuelmain_sha: af2d875a47c5d08bb62943bed5ef1ae7d7ea3329
nailgun_sha: ca95dd5d79282d4613da4275205b67b6da408e2e
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: b297cee0c54b4ff787ca0ac518a54348b838d342
release: '7.0'
release_versions:
  2015.1.0-7.0:
    VERSION:
      api: '1.0'
      astute_sha: e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc
      build_id: 2015-08-11_17-24-26
      build_number: '160'
      feature_groups:
      - mirantis
      fuel-agent_sha: 57145b1d8804389304cd04322ba0fb3dc9d30327
      fuel-library_sha: ecdfcc0b23c4fbeea193e222e6eb5fb1f4b4e68f
      fuel-nailgun-agent_sha: e01693992d7a0304d926b922b43f3b747c35964c
      fuel-ostf_sha: c7f745431aa3c147f2491c865e029e0ffea91c47
      fuelmain_sha: af2d875a47c5d08bb62943bed5ef1ae7d7ea3329
      nailgun_sha: ca95dd5d79282d4613da4275205b67b6da408e2e
      openstack_version: 2015.1.0-7.0
      production: docker
      python-fuelclient_sha: b297cee0c54b4ff787ca0ac518a54348b838d342
      release: '7.0'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/310440

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/310440
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4cb075120678e67507f7d4aa246b614a644df7aa
Submitter: Jenkins
Branch: stable/6.1

commit 4cb075120678e67507f7d4aa246b614a644df7aa
Author: Oleksiy Molchanov <email address hidden>
Date: Mon Aug 10 17:24:25 2015 +0300

    Restrict ntp-client on controller nodes

    NTP-client task shouldn't be run on controller multirole nodes

    Change-Id: I54defe421ded251ad13b665b6fd1843600a517a7
    Closes-Bug: #1481627