keystone-startup.log is partially broken

[Boris Bobrov]

/var/log/keystone/keystone-startup.log contains a huge number of zeros in the beginning of the file. "less" warns that the file is binary. Though the latter part of it is text, which can be checked with "tail -n1".

env-10, was noticed by me on node 21 and by Ilya SHakhat on node 91 (controllers)

[Boris Bobrov (bbobrov)]

keystone-startup.log has something to do with /usr/share/keystone/daemon_notify.sh.

[Ilya Shakhat (shakhat)]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.0"
  api: "1.0"
  build_number: "39"
  build_id: "2014-12-04_18-22-07"
  astute_sha: "16b252d93be6aaa73030b8100cf8c5ca6a970a91"
  fuellib_sha: "605ac6deffd601c1da809eb114cd315f670029d8"
  ostf_sha: "a9afb68710d809570460c29d6c3293219d3624d4"
  nailgun_sha: "0df508c3247a06f8a636ded8fb04733d5aed7732"
  fuelmain_sha: "7ab330b4958ab20955372e85de05e8732e8f6df2"

DEPLOYMENT:
  HA + CentOS + Neutron GRE

[Ilya Shakhat (shakhat)]

[root@node-91 ~]# ls -al /var/log/keystone/
total 48340
drwxr-x--- 2 keystone keystone 4096 Dec 9 10:41 .
drwxr-xr-x. 21 root root 4096 Dec 9 08:30 ..
-rw-r--r-- 1 keystone keystone 3120779 Dec 9 10:41 keystone-all.log
-rw-r--r-- 1 keystone keystone 7309671 Dec 9 10:40 keystone-all.log.1.gz
-rw-r--r-- 1 keystone keystone 8005602 Dec 9 09:30 keystone-all.log.2.gz
-rw-r--r-- 1 keystone keystone 7893459 Dec 9 06:50 keystone-all.log.3.gz
-rw-r--r-- 1 keystone keystone 8000368 Dec 9 05:10 keystone-all.log.4.gz
-rw-r--r-- 1 keystone keystone 31787 Dec 8 20:14 keystone-manage.log
-rw-r--r-- 1 keystone keystone 2424563364 Dec 9 10:41 keystone-startup.log
-rw-r--r-- 1 keystone keystone 2763852 Dec 9 10:40 keystone-startup.log.1.gz
-rw-r--r-- 1 keystone keystone 2703294 Dec 9 10:30 keystone-startup.log.2.gz
-rw-r--r-- 1 keystone keystone 3510086 Dec 9 10:20 keystone-startup.log.3.gz
-rw-r--r-- 1 keystone keystone 3376604 Dec 9 10:10 keystone-startup.log.4.gz

[Ilya Shakhat (shakhat)]

[root@node-91 ~]# hexdump /var/log/keystone/keystone-startup.log | head -n 10
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
9059d9b0 0000 3032 3431 312d 2d32 3930 3120 3a30
9059d9c0 3034 333a 2e32 3930 2037 3933 3031 4420
9059d9d0 4245 4755 6b20 7965 7473 6e6f 2e65 6f63
9059d9e0 6d6d 6e6f 632e 6361 6568 5f2e 656d 636d
9059d9f0 6361 6568 705f 6f6f 206c 2d5b 205d 654d
9059da00 636d 6361 6568 2064 6f70 6c6f 3620 3331
9059da10 3137 3937 2c32 7420 7268 6165 2064 3431
9059da20 3630 3339 3230 3734 3936 3937 3a32 4120

[Dmitry Mescheryakov (dmitrymex)]

Raising to medium as broken log is not nice

[Vladimir Kuklin (vkuklin)]

What FUEL team should do with this? I cannot understand how it is related to deployment - we do not alter how keystone writes to keystone-startup.log

[Boris Bobrov (bbobrov)]

Folks, I am not sure that the issue is still there. It seems to be a one-time glitch. I could run into it again neither on labs, nor locally. I suggest to mark it as Incomplete.

[Boris Bobrov (bbobrov)]

OK, I'm setting it as incomplete since it never occured any more

[Dmitry Mescheryakov (dmitrymex)]

The issue is hanging for more than a month in incomplete state, hence moving to invalid.