Bug #1487966 “[2.20-Build 64] Analytics node failed to purge sto...” : Bugs : Juniper Openstack

Daisuke Nakajima (dnakajima) on 2015-08-24

information type:

Proprietary → Public

Revision history for this message

Daisuke Nakajima (dnakajima) wrote on 2015-08-24:

#1

This issue is seen on 2.20-64.

Nobuhiko Nagataki (nnagatak) on 2015-08-26

tags:

added: customer

Nobuhiko Nagataki (nnagatak) on 2015-09-14

tags:

added: bms

Revision history for this message

Raj Reddy (rajreddy) wrote on 2015-09-14: Re: [Bug 1487966] [2.20-Build 64] Analytics node failed to purge stored data

#2

Download full text (3.3 KiB)

Hi Nobuhiko,

Can you let us know what’s your configuration wrt space and scale? How many db nodes, how much
space on each node and how many vrouters — can you also provide contrail-collector.conf to see
the TTLs that are set.

thanks

> On Sep 13, 2015, at 6:17 PM, Nobuhiko Nagataki <email address hidden> wrote:
>
> ** Tags added: bms
>
> --
> You received this bug notification because you are a member of Contrail
> Systems engineering, which is subscribed to Juniper Openstack.
> https://bugs.launchpad.net/bugs/1487966
>
> Title:
> [2.20-Build 64] Analytics node failed to purge stored data
>
> Status in Juniper Openstack:
> New
> Status in Juniper Openstack r2.20 series:
> New
> Status in Juniper Openstack trunk series:
> New
>
> Bug description:
> Analytics node failes to purge stored data even though the data
> exceeds it threshold.
>
>
> 07/30/2015 09:35:26 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: Database usage of 71 on ot2-opsm00037 exceeds threshold
> 07/30/2015 09:35:26 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: start times:OrderedDict([('SystemObjectFlowStartTime', 1436772521864673), ('SystemObjectMsgStartTime', 1436772521864673), ('SystemObjectStartTime', 1436772521864671), ('SystemObjectStatStartTime', 1436772521864674)])
> 07/30/2015 09:36:34 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: Exception: Purge_id:def5d1ee-3652-11e5-a6d0-70e2840e0ba2 table:MessageTableTimestamp error: 'int' object has no attribute '__getitem__'
> 08/10/2015 04:18:47 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: Database usage of 72 on ot2-opsm00037 exceeds threshold
> 08/10/2015 04:18:47 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: start times:OrderedDict([('SystemObjectFlowStartTime', 1436772521864673), ('SystemObjectMsgStartTime', 1436772521864673), ('SystemObjectStartTime', 1436772521864671), ('SystemObjectStatStartTime', 1436772521864674)])
> 08/10/2015 04:21:36 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: Exception: Purge_id:75a94654-3ecb-11e5-a6d0-70e2840e0ba2 table:MessageTableTimestamp error: 'int' object has no attribute '__getitem__'
> 08/10/2015 05:12:21 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: Database usage of 71 on ot2-opsm00037 exceeds threshold
> 08/10/2015 05:12:21 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: start times:OrderedDict([('SystemObjectFlowStartTime', 1436772521864673), ('SystemObjectMsgStartTime', 1436772521864673), ('SystemObjectStartTime', 1436772521864671), ('SystemObjectStatStartTime', 1436772521864674)])
> 08/10/2015 05:15:09 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: Exception: Purge_id:f16b2b02-3ed2-11e5-a6d0-70e2840e0ba2 table:MessageTableTimestamp error: 'int' object has no attribute '__getitem__'
> 08/10/2015 06:05:49 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: Database usage of 71 on ot2-opsm00037 exceeds threshold
> 08/10/2015 06:05:49 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: start times:OrderedDict([('SystemObjectFlowStartTime', 1436772521864673), ('SystemObjectMsgStartTime', 1436772521864673), ('SystemObjectStartTime', 143...

Hi Nobuhiko,

Can you let us know what’s your configuration wrt space and scale? How many db nodes, how much
space on each node and how many vrouters — can you also provide contrail-collector.conf to see
the TTLs that are set.

thanks

> On Sep 13, 2015, at 6:17 PM, Nobuhiko Nagataki <nnagatak@juniper.net> wrote:
> 
> ** Tags added: bms
> 
> -- 
> You received this bug notification because you are a member of Contrail
> Systems engineering, which is subscribed to Juniper Openstack.
> https://bugs.launchpad.net/bugs/1487966
> 
> Title:
>  [2.20-Build 64] Analytics node failed to purge stored data
> 
> Status in Juniper Openstack:
>  New
> Status in Juniper Openstack r2.20 series:
>  New
> Status in Juniper Openstack trunk series:
>  New
> 
> Bug description:
>  Analytics node failes to purge stored data even though the data
>  exceeds it threshold.
> 
> 
>  07/30/2015 09:35:26 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: Database usage of 71 on ot2-opsm00037 exceeds threshold
>  07/30/2015 09:35:26 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: start times:OrderedDict([('SystemObjectFlowStartTime', 1436772521864673), ('SystemObjectMsgStartTime', 1436772521864673), ('SystemObjectStartTime', 1436772521864671), ('SystemObjectStatStartTime', 1436772521864674)])
>  07/30/2015 09:36:34 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: Exception: Purge_id:def5d1ee-3652-11e5-a6d0-70e2840e0ba2 table:MessageTableTimestamp error: 'int' object has no attribute '__getitem__'
>  08/10/2015 04:18:47 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: Database usage of 72 on ot2-opsm00037 exceeds threshold
>  08/10/2015 04:18:47 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: start times:OrderedDict([('SystemObjectFlowStartTime', 1436772521864673), ('SystemObjectMsgStartTime', 1436772521864673), ('SystemObjectStartTime', 1436772521864671), ('SystemObjectStatStartTime', 1436772521864674)])
>  08/10/2015 04:21:36 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: Exception: Purge_id:75a94654-3ecb-11e5-a6d0-70e2840e0ba2 table:MessageTableTimestamp error: 'int' object has no attribute '__getitem__'
>  08/10/2015 05:12:21 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: Database usage of 71 on ot2-opsm00037 exceeds threshold
>  08/10/2015 05:12:21 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: start times:OrderedDict([('SystemObjectFlowStartTime', 1436772521864673), ('SystemObjectMsgStartTime', 1436772521864673), ('SystemObjectStartTime', 1436772521864671), ('SystemObjectStatStartTime', 1436772521864674)])
>  08/10/2015 05:15:09 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: Exception: Purge_id:f16b2b02-3ed2-11e5-a6d0-70e2840e0ba2 table:MessageTableTimestamp error: 'int' object has no attribute '__getitem__'
>  08/10/2015 06:05:49 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: Database usage of 71 on ot2-opsm00037 exceeds threshold
>  08/10/2015 06:05:49 AM [ot2-opsm00036:contrail-analytics-api:Analytics:0]: start times:OrderedDict([('SystemObjectFlowStartTime', 1436772521864673), ('SystemObjectMsgStartTime', 1436772521864673), ('SystemObjectStartTime', 1436772521864671), ('SystemObjectStatStartTime', 1436772521864674)])
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juniperopenstack/+bug/1487966/+subscriptions

Revision history for this message

Nobuhiko Nagataki (nnagatak) wrote on 2015-09-14:

#3

Download full text (12.1 KiB)

Hi Raj,

Here is the info.

root@ot2-opsm00036:~# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda6 75495536 5354932 66282532 8% /
none 4 0 4 0% /sys/fs/cgroup
udev 98866152 4 98866148 1% /dev
tmpfs 19775392 1940 19773452 1% /run
none 5120 0 5120 0% /run/lock
none 98876956 0 98876956 0% /run/shm
none 102400 0 102400 0% /run/user
/dev/sda1 49082176 6592620 39973232 15% /var/log

The below is testbed.py and contrail-collector.conf.
There are 3 DB and 4 Compute.

root@ot1-fab3:~# cat /opt/contrail/utils/fabfile/testbeds/testbed.py

from fabric.api import env

# 2015/07/13

#Management ip addresses of hosts in the cluster
host_os = 'root@10.11.10.24'
host_cmp1 = 'root@10.11.11.2'
host_cmp2 = 'root@10.11.11.3'
host_cmp3 = 'root@10.11.11.4'
host_cmp4 = 'root@10.11.11.5'
host_contrail1 = 'root@10.11.10.15'
host_contrail2 = 'root@10.11.10.16'
host_contrail3 = 'root@10.11.10.17'

#External routers if any
#for eg.
#ext_routers = [('mx1', '10.204.216.253')]
ext_routers = [('mx-vip', '10.11.111.1')]

#Autonomous system number
router_asn = 64512

#Host from which the fab commands are triggered to install and provision
host_build = 'root@10.10.10.40'

#Role definition of the hosts.
env.roledefs = {
    'all': [host_os, host_cmp1, host_cmp2, host_cmp3, host_cmp4,
host_contrail1, host_contrail2, host_contrail3],
    'cfgm': [host_contrail1, host_contrail2, host_contrail3],
    'openstack': [host_os],
    'control': [host_contrail1, host_contrail2, host_contrail3],
    'compute': [host_cmp1, host_cmp2, host_cmp3, host_cmp4],
    'collector': [host_contrail1, host_contrail2, host_contrail3],
    'webui': [host_contrail1, host_contrail2, host_contrail3],
    'database': [host_contrail1, host_contrail2, host_contrail3],
    'build': [host_build],
}

env.hostnames = {
'all': ['ot2-opct1', 'ot2-opsh00012', 'ot2-opsh00013',
'ot2-opsh00014', 'ot2-opsh00015', 'ot2-opsm00035', 'ot2-opsm00036',
'ot2-opsm00037']
}

#Openstack admin password
env.openstack_admin_password = 'Ncinci01'

env.keystone = {
    'keystone_ip' : '10.11.10.118',
    'auth_protocol' : 'http',
    'auth_port' : '30357',
    'admin_token' : 'ADMIN',
    'admin_user' : 'usrAdmin',
    'admin_password': 'Ncinci01',
    'service_tenant': 'tntService',
    'admin_tenant' : 'tntAdmin',
    'region_name' : 'ote1',
    'insecure' :'True',
    'manage_neutron': 'no',
    'nova_password': 'Ncinci01',
    'neutron_password': 'Ncinci01',
}

#Passwords of each host
env.passwords = {
    host_os: 'Ncinci01',
    host_cmp1: 'Ncinci01',
    host_cmp2: 'Ncinci01',
    host_cmp3: 'Ncinci01',
    host_cmp4: 'Ncinci01',
    host_contrail1: 'Ncinci01',
    host_contrail2: 'Ncinci01',
    host_contrail3: 'Ncinci01',
    host_build: 'Ncinci01',
}

#For reimage purpose
env.ostypes = {
    host_os: 'ubuntu',
    host_cmp1: 'ubuntu',
    host_cmp2: 'ubuntu',
    host_cmp3: 'ubuntu',
    host_cmp4: 'ubuntu',
    host_contrail1: 'ubuntu',
    host_contrail2: 'ubuntu',
    host_contrail3: 'ubuntu',
}
env.orchestrator = 'opens...

Hi Raj,

Here is the info.

root@ot2-opsm00036:~# df -k
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/sda6       75495536 5354932  66282532   8% /
none                   4       0         4   0% /sys/fs/cgroup
udev            98866152       4  98866148   1% /dev
tmpfs           19775392    1940  19773452   1% /run
none                5120       0      5120   0% /run/lock
none            98876956       0  98876956   0% /run/shm
none              102400       0    102400   0% /run/user
/dev/sda1       49082176 6592620  39973232  15% /var/log

The below is testbed.py and contrail-collector.conf.
There are 3 DB and 4 Compute.

root@ot1-fab3:~# cat /opt/contrail/utils/fabfile/testbeds/testbed.py

from fabric.api import env

# 2015/07/13

#Management ip addresses of hosts in the cluster
host_os = 'root@10.11.10.24'
host_cmp1 = 'root@10.11.11.2'
host_cmp2 = 'root@10.11.11.3'
host_cmp3 = 'root@10.11.11.4'
host_cmp4 = 'root@10.11.11.5'
host_contrail1 = 'root@10.11.10.15'
host_contrail2 = 'root@10.11.10.16'
host_contrail3 = 'root@10.11.10.17'

#External routers if any
#for eg.
#ext_routers = [('mx1', '10.204.216.253')]
ext_routers = [('mx-vip', '10.11.111.1')]

#Autonomous system number
router_asn = 64512

#Host from which the fab commands are triggered to install and provision
host_build = 'root@10.10.10.40'

#Role definition of the hosts.
env.roledefs = {
    'all': [host_os, host_cmp1, host_cmp2, host_cmp3, host_cmp4,
host_contrail1, host_contrail2, host_contrail3],
    'cfgm': [host_contrail1, host_contrail2, host_contrail3],
    'openstack': [host_os],
    'control': [host_contrail1, host_contrail2, host_contrail3],
    'compute': [host_cmp1, host_cmp2, host_cmp3, host_cmp4],
    'collector': [host_contrail1, host_contrail2, host_contrail3],
    'webui': [host_contrail1, host_contrail2, host_contrail3],
    'database': [host_contrail1, host_contrail2, host_contrail3],
    'build': [host_build],
}

env.hostnames = {
    'all': ['ot2-opct1', 'ot2-opsh00012', 'ot2-opsh00013',
'ot2-opsh00014', 'ot2-opsh00015', 'ot2-opsm00035', 'ot2-opsm00036',
'ot2-opsm00037']
}

#Openstack admin password
env.openstack_admin_password = 'Ncinci01'

env.keystone = {
    'keystone_ip' : '10.11.10.118',
    'auth_protocol' : 'http',
    'auth_port' : '30357',
    'admin_token' : 'ADMIN',
    'admin_user' : 'usrAdmin',
    'admin_password': 'Ncinci01',
    'service_tenant': 'tntService',
    'admin_tenant' : 'tntAdmin',
    'region_name' : 'ote1',
    'insecure' :'True',
    'manage_neutron': 'no',
    'nova_password': 'Ncinci01',
    'neutron_password': 'Ncinci01',
}

#Passwords of each host
env.passwords = {
    host_os: 'Ncinci01',
    host_cmp1: 'Ncinci01',
    host_cmp2: 'Ncinci01',
    host_cmp3: 'Ncinci01',
    host_cmp4: 'Ncinci01',
    host_contrail1: 'Ncinci01',
    host_contrail2: 'Ncinci01',
    host_contrail3: 'Ncinci01',
    host_build: 'Ncinci01',
}

#For reimage purpose
env.ostypes = {
    host_os: 'ubuntu',
    host_cmp1: 'ubuntu',
    host_cmp2: 'ubuntu',
    host_cmp3: 'ubuntu',
    host_cmp4: 'ubuntu',
    host_contrail1: 'ubuntu',
    host_contrail2: 'ubuntu',
    host_contrail3: 'ubuntu',
}
env.orchestrator = 'openstack' #other values are 'vcenter'
default:openstack

#OPTIONAL SEPARATION OF MANAGEMENT AND CONTROL + DATA and OPTIONAL VLAN
INFORMATION
#==========================================================================
========
control_data = {
    host_os : { 'ip': '192.168.22.24/24', 'gw' : '192.168.22.1',
'device':'eth3' },
    host_cmp1 : { 'ip': '10.11.111.2/25', 'gw' : '10.11.111.1',
'device':'bond1.57', 'vlan': '57' },
    host_cmp2 : { 'ip': '10.11.111.3/25', 'gw' : '10.11.111.1',
'device':'bond1.57', 'vlan': '57' },
    host_cmp3 : { 'ip': '10.11.111.4/25', 'gw' : '10.11.111.1',
'device':'bond1.57', 'vlan': '57' },
    host_cmp4 : { 'ip': '10.11.111.5/25', 'gw' : '10.11.111.1',
'device':'bond1.57', 'vlan': '57' },
    host_contrail1 : { 'ip': '192.168.22.15/24', 'gw' : '192.168.22.1',
'device':'bond1.42', 'vlan': '42' },
    host_contrail2 : { 'ip': '192.168.22.16/24', 'gw' : '192.168.22.1',
'device':'bond1.42', 'vlan': '42' },
    host_contrail3 : { 'ip': '192.168.22.17/24', 'gw' : '192.168.22.1',
'device':'bond1.42', 'vlan': '42' },
}

# In Openstack or Contrail High Availability setups.
# internal_vip          : Virtual IP of the openstack HA Nodes in the
data/control(internal) nerwork,
#                         all the Openstack services behind this VIP are
accessed using this VIP.
# external_vip          : Virtual IP of the Openstack HA Nodes in the
management(external) nerwork,
#                         Openstack dashboard and novncproxy  services
behind this VIP are accessed using this VIP.
# contrail_internal_vip : Virtual IP of the Contrail HA Nodes in the
data/control(internal) nerwork,
#                         all the Contrail services behind this VIP is
accessed using this VIP.
# contrail_external_vip : Virtual IP of the Contrail HA Nodes in the
management(external) nerwork,
#                         Contrail introspects are are accessed using this
VIP.
# nfs_server            : NFS server to be used to store the glance images.
# nfs_glance_path       : NFS server image path, which will be mounted on
the Openstack Nodes and
#                         the glance images will be placed/accesed in/from
this location.
env.ha = {
    'contrail_internal_vip' : '192.168.22.244',
    'contrail_external_vip' : '10.11.10.115',
}

# In environments where openstack services are deployed independently
# from contrail, you can use the below options
# service_token : Common service token for for all services like nova,
#                 neutron, glance, cinder etc
# amqp_host     : IP of AMQP Server to be used in openstack
# manage_amqp   : Default = 'no', if set to 'yes' provision's amqp in
openstack nodes and
#                 openstack services uses the amqp in openstack nodes
instead of config nodes.
#                 amqp_host is neglected if manage_amqp is set
#
env.openstack = {
    'service_token': 'ADMIN',
    'amqp_host' : '10.11.10.116',
}

#OPTIONAL ANALYTICS CONFIGURATION
minimum_diskGB = 40

root@ot2-opsm00036:~# cat /etc/contrail/contrail-collector.conf
#
# Copyright (c) 2014 Juniper Networks, Inc. All rights reserved.
#
# Collector configuration options
#

[DEFAULT]
# Everything in this section is optional

# Time-to-live in hours of the various data stored by collector into
# cassandra
# analytics_config_audit_ttl, if not set (or set to -1), defaults to
analytics_data_ttl
# analytics_statistics_ttl, if not set (or set to -1), defaults to
analytics_data_ttl
# analytics_flow_ttl, if not set (or set to -1), defaults to
analytics_statsdata_ttl
analytics_data_ttl=48
analytics_config_audit_ttl=-1
analytics_statistics_ttl=-1
analytics_flow_ttl=-1

# IP address and port to be used to connect to cassandra.
# Multiple IP:port strings separated by space can be provided
cassandra_server_list=192.168.22.16:9160 192.168.22.15:9160
192.168.22.17:9160

# IP address and port to be used to connect to kafka.
# Multiple IP:port strings separated by space can be provided
kafka_broker_list=

# IP address of analytics node. Resolved IP of 'hostname'
hostip=192.168.22.16

# Hostname of analytics node. If this is not configured value from
`hostname`
# will be taken
# hostname=

# Http server port for inspecting collector state (useful for debugging)
http_server_port=8089

# Category for logging. Default value is '*'
# log_category=

# Local log file name
log_file=/var/log/contrail/contrail-collector.log

# Maximum log file rollover index
# log_files_count=10

# Maximum log file size
# log_file_size=1048576 # 1MB

# Log severity levels. Possible values are SYS_EMERG, SYS_ALERT, SYS_CRIT,
# SYS_ERR, SYS_WARN, SYS_NOTICE, SYS_INFO and SYS_DEBUG. Default is
SYS_DEBUG
log_level=SYS_NOTICE

# Enable/Disable local file logging. Possible values are 0 (disable) and
# 1 (enable)
log_local=1

# TCP and UDP ports to listen on for receiving syslog messages. -1 to
disable.
syslog_port=-1

# UDP port to listen on for receiving sFlow messages. -1 to disable.
# sflow_port=6343

# UDP port to listen on for receiving ipfix messages. -1 to disable.
# ipfix_port=4739

[COLLECTOR]
# Everything in this section is optional

# Port to listen on for receiving Sandesh messages
port=8086

# IP address to bind to for listening
# server=0.0.0.0

# UDP port to listen on for receiving Google Protocol Buffer messages
# protobuf_port=3333

[DISCOVERY]
# Port to connect to for communicating with discovery server
# port=5998

# IP address of discovery server
server=192.168.22.244

[REDIS]
# Port to connect to for communicating with redis-server
port=6379

# IP address of redis-server
server=127.0.0.1

Regards,
Nobu

On 9/14/15, 1:21 PM, "Raj Reddy" <rajreddy@juniper.net> wrote:

>Hi Nobuhiko,
>
>Can you let us know what¹s your configuration wrt space and scale? How
>many db nodes, how much
>space on each node and how many vrouters ‹ can you also provide
>contrail-collector.conf to see
>the TTLs that are set.
>
>thanks
>
>> On Sep 13, 2015, at 6:17 PM, Nobuhiko Nagataki <nnagatak@juniper.net>
>>wrote:
>> 
>> ** Tags added: bms
>> 
>> -- 
>> You received this bug notification because you are a member of Contrail
>> Systems engineering, which is subscribed to Juniper Openstack.
>> https://bugs.launchpad.net/bugs/1487966
>> 
>> Title:
>>  [2.20-Build 64] Analytics node failed to purge stored data
>> 
>> Status in Juniper Openstack:
>>  New
>> Status in Juniper Openstack r2.20 series:
>>  New
>> Status in Juniper Openstack trunk series:
>>  New
>> 
>> Bug description:
>>  Analytics node failes to purge stored data even though the data
>>  exceeds it threshold.
>> 
>> 
>>  07/30/2015 09:35:26 AM
>>[ot2-opsm00036:contrail-analytics-api:Analytics:0]: Database usage of 71
>>on ot2-opsm00037 exceeds threshold
>>  07/30/2015 09:35:26 AM
>>[ot2-opsm00036:contrail-analytics-api:Analytics:0]: start
>>times:OrderedDict([('SystemObjectFlowStartTime', 1436772521864673),
>>('SystemObjectMsgStartTime', 1436772521864673),
>>('SystemObjectStartTime', 1436772521864671),
>>('SystemObjectStatStartTime', 1436772521864674)])
>>  07/30/2015 09:36:34 AM
>>[ot2-opsm00036:contrail-analytics-api:Analytics:0]: Exception:
>>Purge_id:def5d1ee-3652-11e5-a6d0-70e2840e0ba2
>>table:MessageTableTimestamp error: 'int' object has no attribute
>>'__getitem__'
>>  08/10/2015 04:18:47 AM
>>[ot2-opsm00036:contrail-analytics-api:Analytics:0]: Database usage of 72
>>on ot2-opsm00037 exceeds threshold
>>  08/10/2015 04:18:47 AM
>>[ot2-opsm00036:contrail-analytics-api:Analytics:0]: start
>>times:OrderedDict([('SystemObjectFlowStartTime', 1436772521864673),
>>('SystemObjectMsgStartTime', 1436772521864673),
>>('SystemObjectStartTime', 1436772521864671),
>>('SystemObjectStatStartTime', 1436772521864674)])
>>  08/10/2015 04:21:36 AM
>>[ot2-opsm00036:contrail-analytics-api:Analytics:0]: Exception:
>>Purge_id:75a94654-3ecb-11e5-a6d0-70e2840e0ba2
>>table:MessageTableTimestamp error: 'int' object has no attribute
>>'__getitem__'
>>  08/10/2015 05:12:21 AM
>>[ot2-opsm00036:contrail-analytics-api:Analytics:0]: Database usage of 71
>>on ot2-opsm00037 exceeds threshold
>>  08/10/2015 05:12:21 AM
>>[ot2-opsm00036:contrail-analytics-api:Analytics:0]: start
>>times:OrderedDict([('SystemObjectFlowStartTime', 1436772521864673),
>>('SystemObjectMsgStartTime', 1436772521864673),
>>('SystemObjectStartTime', 1436772521864671),
>>('SystemObjectStatStartTime', 1436772521864674)])
>>  08/10/2015 05:15:09 AM
>>[ot2-opsm00036:contrail-analytics-api:Analytics:0]: Exception:
>>Purge_id:f16b2b02-3ed2-11e5-a6d0-70e2840e0ba2
>>table:MessageTableTimestamp error: 'int' object has no attribute
>>'__getitem__'
>>  08/10/2015 06:05:49 AM
>>[ot2-opsm00036:contrail-analytics-api:Analytics:0]: Database usage of 71
>>on ot2-opsm00037 exceeds threshold
>>  08/10/2015 06:05:49 AM
>>[ot2-opsm00036:contrail-analytics-api:Analytics:0]: start
>>times:OrderedDict([('SystemObjectFlowStartTime', 1436772521864673),
>>('SystemObjectMsgStartTime', 1436772521864673),
>>('SystemObjectStartTime', 1436772521864671),
>>('SystemObjectStatStartTime', 1436772521864674)])
>> 
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/juniperopenstack/+bug/1487966/+subscriptions
>

Revision history for this message

Raj Reddy (rajreddy) wrote on 2015-09-14:

#4

Download full text (12.5 KiB)

Hi Nobuhiko,

Deleting and recovering disk space should be last resort, ideally you should size the disk
and configure *_ttl such that we don’t overrun the space.. The following are the recommended
ttl values

#analytics_data_ttl=48
#analytics_config_audit_ttl=2160
#analytics_statistics_ttl=168
#analytics_flow_ttl=2

thanks,

On Sep 13, 2015, at 9:47 PM, Nobuhiko Nagataki <<email address hidden><mailto:<email address hidden>>> wrote:

Hi Raj,

Here is the info.

root@ot2-opsm00036:~# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda6 75495536 5354932 66282532 8% /
none 4 0 4 0% /sys/fs/cgroup
udev 98866152 4 98866148 1% /dev
tmpfs 19775392 1940 19773452 1% /run
none 5120 0 5120 0% /run/lock
none 98876956 0 98876956 0% /run/shm
none 102400 0 102400 0% /run/user
/dev/sda1 49082176 6592620 39973232 15% /var/log

The below is testbed.py and contrail-collector.conf.
There are 3 DB and 4 Compute.

root@ot1-fab3:~# cat /opt/contrail/utils/fabfile/testbeds/testbed.py

from fabric.api import env

# 2015/07/13

#Management ip addresses of hosts in the cluster
host_os = 'root@10.11.10.24<mailto:root@10.11.10.24>'
host_cmp1 = 'root@10.11.11.2<mailto:root@10.11.11.2>'
host_cmp2 = 'root@10.11.11.3<mailto:root@10.11.11.3>'
host_cmp3 = 'root@10.11.11.4<mailto:root@10.11.11.4>'
host_cmp4 = 'root@10.11.11.5<mailto:root@10.11.11.5>'
host_contrail1 = 'root@10.11.10.15<mailto:root@10.11.10.15>'
host_contrail2 = 'root@10.11.10.16<mailto:root@10.11.10.16>'
host_contrail3 = 'root@10.11.10.17<mailto:root@10.11.10.17>'

#External routers if any
#for eg.
#ext_routers = [('mx1', '10.204.216.253')]
ext_routers = [('mx-vip', '10.11.111.1')]

#Autonomous system number
router_asn = 64512

#Host from which the fab commands are triggered to install and provision
host_build = 'root@10.10.10.40<mailto:root@10.10.10.40>'

#Role definition of the hosts.
env.roledefs = {
   'all': [host_os, host_cmp1, host_cmp2, host_cmp3, host_cmp4,
host_contrail1, host_contrail2, host_contrail3],
   'cfgm': [host_contrail1, host_contrail2, host_contrail3],
   'openstack': [host_os],
   'control': [host_contrail1, host_contrail2, host_contrail3],
   'compute': [host_cmp1, host_cmp2, host_cmp3, host_cmp4],
   'collector': [host_contrail1, host_contrail2, host_contrail3],
   'webui': [host_contrail1, host_contrail2, host_contrail3],
   'database': [host_contrail1, host_contrail2, host_contrail3],
   'build': [host_build],
}

env.hostnames = {
'all': ['ot2-opct1', 'ot2-opsh00012', 'ot2-opsh00013',
'ot2-opsh00014', 'ot2-opsh00015', 'ot2-opsm00035', 'ot2-opsm00036',
'ot2-opsm00037']
}

#Openstack admin password
env.openstack_admin_password = 'Ncinci01'

env.keystone = {
   'keystone_ip' : '10.11.10.118',
   'auth_protocol' : 'http',
   'auth_port' : '30357',
   'admin_token' : 'ADMIN',
   'admin_user' : 'usrAdmin',
   'admin_password': 'Ncinci01',
   'service_tenant': 'tntService',
   'admin_tenant' : 'tntAdmin',
   'region_name' : 'ote1',
   'insecure' :'True',
   'manage_neutron': 'no',
   'nova_password'...

Hi Nobuhiko,

Deleting and recovering disk space should be last resort, ideally you should size the disk
and configure *_ttl such that we don’t overrun the space.. The following are the recommended
ttl values

#analytics_data_ttl=48
#analytics_config_audit_ttl=2160
#analytics_statistics_ttl=168
#analytics_flow_ttl=2

thanks,

On Sep 13, 2015, at 9:47 PM, Nobuhiko Nagataki <nnagatak@juniper.net<mailto:nnagatak@juniper.net>> wrote:

Hi Raj,

Here is the info.

root@ot2-opsm00036:~# df -k
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/sda6       75495536 5354932  66282532   8% /
none                   4       0         4   0% /sys/fs/cgroup
udev            98866152       4  98866148   1% /dev
tmpfs           19775392    1940  19773452   1% /run
none                5120       0      5120   0% /run/lock
none            98876956       0  98876956   0% /run/shm
none              102400       0    102400   0% /run/user
/dev/sda1       49082176 6592620  39973232  15% /var/log

The below is testbed.py and contrail-collector.conf.
There are 3 DB and 4 Compute.

root@ot1-fab3:~# cat /opt/contrail/utils/fabfile/testbeds/testbed.py

from fabric.api import env

# 2015/07/13

#Management ip addresses of hosts in the cluster
host_os = 'root@10.11.10.24<mailto:root@10.11.10.24>'
host_cmp1 = 'root@10.11.11.2<mailto:root@10.11.11.2>'
host_cmp2 = 'root@10.11.11.3<mailto:root@10.11.11.3>'
host_cmp3 = 'root@10.11.11.4<mailto:root@10.11.11.4>'
host_cmp4 = 'root@10.11.11.5<mailto:root@10.11.11.5>'
host_contrail1 = 'root@10.11.10.15<mailto:root@10.11.10.15>'
host_contrail2 = 'root@10.11.10.16<mailto:root@10.11.10.16>'
host_contrail3 = 'root@10.11.10.17<mailto:root@10.11.10.17>'

#External routers if any
#for eg.
#ext_routers = [('mx1', '10.204.216.253')]
ext_routers = [('mx-vip', '10.11.111.1')]

#Autonomous system number
router_asn = 64512

#Host from which the fab commands are triggered to install and provision
host_build = 'root@10.10.10.40<mailto:root@10.10.10.40>'

#Role definition of the hosts.
env.roledefs = {
   'all': [host_os, host_cmp1, host_cmp2, host_cmp3, host_cmp4,
host_contrail1, host_contrail2, host_contrail3],
   'cfgm': [host_contrail1, host_contrail2, host_contrail3],
   'openstack': [host_os],
   'control': [host_contrail1, host_contrail2, host_contrail3],
   'compute': [host_cmp1, host_cmp2, host_cmp3, host_cmp4],
   'collector': [host_contrail1, host_contrail2, host_contrail3],
   'webui': [host_contrail1, host_contrail2, host_contrail3],
   'database': [host_contrail1, host_contrail2, host_contrail3],
   'build': [host_build],
}

env.hostnames = {
   'all': ['ot2-opct1', 'ot2-opsh00012', 'ot2-opsh00013',
'ot2-opsh00014', 'ot2-opsh00015', 'ot2-opsm00035', 'ot2-opsm00036',
'ot2-opsm00037']
}

#Openstack admin password
env.openstack_admin_password = 'Ncinci01'

env.keystone = {
   'keystone_ip' : '10.11.10.118',
   'auth_protocol' : 'http',
   'auth_port' : '30357',
   'admin_token' : 'ADMIN',
   'admin_user' : 'usrAdmin',
   'admin_password': 'Ncinci01',
   'service_tenant': 'tntService',
   'admin_tenant' : 'tntAdmin',
   'region_name' : 'ote1',
   'insecure' :'True',
   'manage_neutron': 'no',
   'nova_password': 'Ncinci01',
   'neutron_password': 'Ncinci01',
}

#Passwords of each host
env.passwords = {
   host_os: 'Ncinci01',
   host_cmp1: 'Ncinci01',
   host_cmp2: 'Ncinci01',
   host_cmp3: 'Ncinci01',
   host_cmp4: 'Ncinci01',
   host_contrail1: 'Ncinci01',
   host_contrail2: 'Ncinci01',
   host_contrail3: 'Ncinci01',
   host_build: 'Ncinci01',
}

#For reimage purpose
env.ostypes = {
   host_os: 'ubuntu',
   host_cmp1: 'ubuntu',
   host_cmp2: 'ubuntu',
   host_cmp3: 'ubuntu',
   host_cmp4: 'ubuntu',
   host_contrail1: 'ubuntu',
   host_contrail2: 'ubuntu',
   host_contrail3: 'ubuntu',
}
env.orchestrator = 'openstack' #other values are 'vcenter'
default:openstack

#OPTIONAL SEPARATION OF MANAGEMENT AND CONTROL + DATA and OPTIONAL VLAN
INFORMATION
#==========================================================================
========
control_data = {
   host_os : { 'ip': '192.168.22.24/24', 'gw' : '192.168.22.1',
'device':'eth3' },
   host_cmp1 : { 'ip': '10.11.111.2/25', 'gw' : '10.11.111.1',
'device':'bond1.57', 'vlan': '57' },
   host_cmp2 : { 'ip': '10.11.111.3/25', 'gw' : '10.11.111.1',
'device':'bond1.57', 'vlan': '57' },
   host_cmp3 : { 'ip': '10.11.111.4/25', 'gw' : '10.11.111.1',
'device':'bond1.57', 'vlan': '57' },
   host_cmp4 : { 'ip': '10.11.111.5/25', 'gw' : '10.11.111.1',
'device':'bond1.57', 'vlan': '57' },
   host_contrail1 : { 'ip': '192.168.22.15/24', 'gw' : '192.168.22.1',
'device':'bond1.42', 'vlan': '42' },
   host_contrail2 : { 'ip': '192.168.22.16/24', 'gw' : '192.168.22.1',
'device':'bond1.42', 'vlan': '42' },
   host_contrail3 : { 'ip': '192.168.22.17/24', 'gw' : '192.168.22.1',
'device':'bond1.42', 'vlan': '42' },
}

# In Openstack or Contrail High Availability setups.
# internal_vip          : Virtual IP of the openstack HA Nodes in the
data/control(internal) nerwork,
#                         all the Openstack services behind this VIP are
accessed using this VIP.
# external_vip          : Virtual IP of the Openstack HA Nodes in the
management(external) nerwork,
#                         Openstack dashboard and novncproxy  services
behind this VIP are accessed using this VIP.
# contrail_internal_vip : Virtual IP of the Contrail HA Nodes in the
data/control(internal) nerwork,
#                         all the Contrail services behind this VIP is
accessed using this VIP.
# contrail_external_vip : Virtual IP of the Contrail HA Nodes in the
management(external) nerwork,
#                         Contrail introspects are are accessed using this
VIP.
# nfs_server            : NFS server to be used to store the glance images.
# nfs_glance_path       : NFS server image path, which will be mounted on
the Openstack Nodes and
#                         the glance images will be placed/accesed in/from
this location.
env.ha = {
   'contrail_internal_vip' : '192.168.22.244',
   'contrail_external_vip' : '10.11.10.115',
}

# In environments where openstack services are deployed independently
# from contrail, you can use the below options
# service_token : Common service token for for all services like nova,
#                 neutron, glance, cinder etc
# amqp_host     : IP of AMQP Server to be used in openstack
# manage_amqp   : Default = 'no', if set to 'yes' provision's amqp in
openstack nodes and
#                 openstack services uses the amqp in openstack nodes
instead of config nodes.
#                 amqp_host is neglected if manage_amqp is set
#
env.openstack = {
   'service_token': 'ADMIN',
   'amqp_host' : '10.11.10.116',
}