queries stop working after ETL process

[Jessie Evangelista]

We are using Calpont InfiniDB 2.2.11 Community on Ubuntu Server 12.04LTS
Hardware: 2 x Quad Core Xeon, 48GB mem, LSI RAID card 512MB w/ BBU in writeback mode, 8TB RAID10
we processed a 315MB sales data file containing several days of data
we first split the file by date and process each one in turn:

119M sales_transaction_20130601.csv
134M sales_transaction_20130602.csv
30M sales_transaction_20130603.csv
33M sales_transaction_20130531.csv
29K sales_transaction_20130530.csv
13K sales_transaction_20130527.csv
1.6K sales_transaction_20130526.csv
12K sales_transaction_20130525.csv
104 sales_transaction_20130516.csv
20K sales_transaction_20130523.csv
46 sales_transaction_20130524.csv
471K sales_transaction_20130528.csv
1.2M sales_transaction_20130529.csv
4.8K sales_transaction_20130522.csv
2.3K sales_transaction_20130515.csv
33K sales_transaction_20130510.csv

we check if there is existing data for the particular date, if none we simple load the data using cpimport
if data already exists, we load the data into an adjustment table and detect duplicates between the target fact table and the adjustment table
we delete the duplicate data in the fact table and load the adjustments into the fact table
we then generate aggregates on the loaded fact table which encountered the error below
we then restarted infinidb and tried running the same query, no more error

#idbmysql -BN suppliers_portal <<ENDSQL
select store_id,product_id,sum(net_sales),sum(qty),sum(total_cost)
from sales_transaction_20130601
group by store_id,product_id
limit 21
ENDSQL
ERROR 122 (HY000) at line 1: An unexpected condition within the query caused an internal processing error within InfiniDB. Please check the log files for more details. Additional Information: error in BatchPrimitiveProcessor projectResult. Please check crit.log for more details.

#last 21 lines from /usr/local/Calpont/crit.log
Aug 1 15:11:54 node150 PrimProc[11430]: 54.238113 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 7430328 lbid 2433024: input rids 7742, output rids 3706#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.238300 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 0 lbid 2434560: input rids 8192, output rids 0#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.238491 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 0 lbid 2432000: input rids 7858, output rids 3644#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.238542 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 0 lbid 2432768: input rids 7812, output rids 3992#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.238832 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 0 lbid 2434304: input rids 8192, output rids 441#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.239136 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 0 lbid 2431752: input rids 7775, output rids 3481#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.242645 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 0 lbid 2434568: input rids 5120, output rids 0#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.242669 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 6379 lbid 2430976: input rids 5385, output rids 3220#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.244982 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 0 lbid 2433800: input rids 7504, output rids 1493#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.245472 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 0 lbid 2432776: input rids 7801, output rids 3913#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.245628 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 0 lbid 2432008: input rids 7848, output rids 3626#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.245790 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 0 lbid 2431760: input rids 7760, output rids 3456#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.246118 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 0 lbid 2434312: input rids 8192, output rids 467#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.246953 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 2 lbid 2431488: input rids 7783, output rids 3976#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.247241 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 0 lbid 2432256: input rids 7849, output rids 3850#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.249014 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 0 lbid 2433536: input rids 7695, output rids 2411#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.245507 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 7430328 lbid 2433032: input rids 7755, output rids 3734#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.250233 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 7430328 lbid 2432512: input rids 7796, output rids 3898#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.251424 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 0 lbid 2433808: input rids 7494, output rids 1538#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.251960 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 7430328 lbid 2431232: input rids 7863, output rids 4179#012#012
Aug 1 15:11:54 node150 PrimProc[11430]: 54.255229 |0|0|0| C 28 CAL0000: rtscommand.cpp (token column) error on projection for oid 7430328 lbid 2433280: input rids 7611, output rids 2787#012#012

#service infinidb restart
Shutting down Calpont InfiniDB MySQL:
Shutting down MySQL
. *
Shutting down Calpont InfiniDB Database Platform:

Starting Calpont InfiniDB Database Platform: ....... DONE
Starting Calpont InfiniDB MySQL:
Starting MySQL
. *
Validate InfiniDB System Catalog:
 Validation Succesfully Completed
Perform Functionality test:
 InfiniDB Logging check: DONE
 Platform Process check: DONE
 MySQL Daemon check: * MySQL running (338)
 Database check: DONE

#idbmysql -BN suppliers_portal <<ENDSQL
select store_id,product_id,sum(net_sales),sum(qty),sum(total_cost)
from sales_transaction_20130601
group by store_id,product_id
limit 21
ENDSQL

157 18195 74.10 3.00 63.03
157 18045 99.00 1.00 94.50
157 49179 25.50 1.00 23.47
157 159260 401.00 10.00 356.34
157 416277 214.00 1.00 203.80
720 66657 506.60 1.00 438.22
131 329421 59.00 1.00 52.91
723 334488 697.50 1.00 681.66
139 278551 219.00 1.00 199.17
832 363178 220.00 1.00 186.00
124 341351 377.00 13.00 341.25
124 313248 544.00 1.00 527.87
184 408198 136.00 2.00 122.36
302 18496 64.70 1.00 52.65
134 113135 44.75 1.00 33.56
127 14542 59.00 1.00 53.21
127 382633 24.95 1.00 20.74
125 12260 13.50 1.00 12.57
151 35001 13.55 1.00 12.60
127 12553 66.50 1.00 61.91
118 371290 55.00 1.00 50.00