HBASE regionserver failure causes DB to become inconsistent.
Bug #1412487 reported by
Guy Groulx
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Trafodion |
Fix Committed
|
Critical
|
Oliver Bucaojit |
Bug Description
Running version of ORDERENTRY on our system with high concurrency.
Because of memory issue, a regionserver eventually failed.
When we recovered the system, our OE table started returning "UNIQUE CONSTRAINT" errors telling us that our DB is now inconsistent.
Changed in trafodion: | |
status: | New → In Progress |
To post a comment you must log in.
This may have all the information, except for the hbase logs --
In addition to the info currently capture in sqcollectlogs, I’ve also captured the following:
a) Latest Hbase .out files for each node (placed in /home/squser4/ logs/sqinfo. 20150118. 0543/hbase/ directory) logs/sqinfo. 20150118. 0543/logs/ directory – search for Trafodion. dtm.n__ __.log files) logs/sqinfo. 20150118. 0543/logs/ directory – search for gc.log- 20151117* .n____ files)
b) Current Trafodion.dtm.log files for each node (placed in /home/squser4/
c) Latest gc file for each regionserver in each node (placed in /home/squser4/
/home/squser4/ trafodion_ tools> sqcollectlogs -a squser4/ git150117/ sql/scripts/ mon.env' : No such file or directory squser4/ git150117/ sql/scripts/ shell.env' : No such file or directory logs/sqinfo. 20150118_ 0543
Collection in progress...
cp: cannot stat `/opt/hp/
cp: cannot stat `/opt/hp/
Collecting dp2 pstacks
[1] 29914
Collecting monitor pstacks
[2] 29915
Collecting mxosrvr pstacks
[3] 29916
Collecting tdm_arkcmp pstacks
[4] 29917
Collecting tdm_arkesp pstacks
[5] 29918
Collecting tm pstacks
[6] 29919
[1] Done $SQPDSHA "sqnpstack dp2 $PWD/dp2" 2> /dev/null
[4] Done $SQPDSHA "sqnpstack tdm_arkcmp $PWD/tdm_arkcmp" 2> /dev/null
[5]- Done $SQPDSHA "sqnpstack tdm_arkesp $PWD/tdm_arkesp" 2> /dev/null
[2] Done $SQPDSHA "sqnpstack monitor $PWD/monitor" 2> /dev/null
[3]- Done $SQPDSHA "sqnpstack mxosrvr $PWD/mxosrvr" 2> /dev/null
[6]+ Done $SQPDSHA "sqnpstack tm $PWD/tm" 2> /dev/null
Logs and pstacks collected in /home/squser4/