Index 82 out of bounds for (SIMPLE-ARRAY ... (64)), should be nonnegative and <64. ...

[james anderson]

this error has occurred several times over the past day in an application which has been in service, unchanged, for weeks.
one possible difference is that the it may have been presented with queries which have a higher operator count and thereby involve more threads, but it is not yet possible to quantify that difference.

$ sbcl --version
SBCL 1.0.46
$ uname -a
Linux nxp-qa 3.5.0-48-generic #72-Ubuntu SMP Mon Mar 10 23:18:29 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

we have yet to move from 1.0.46 as there has been no response to #1292673

$ sbcl --core sbcl-spocq.core
This is SBCL 1.0.46, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
* *features*

(:BORDEAUX-THREADS :WILBUR2 :WILBUR :CL-PPCRE :SB-BSD-SOCKETS-ADDRINFO
 :CLOSER-MOP :DE.SETF.UTILITY.TEST CFFI-FEATURES:FLAT-NAMESPACE
 CFFI-FEATURES:X86-64 CFFI-FEATURES:UNIX :CFFI CFFI-SYS::FLAT-NAMESPACE
 :LITTLE-ENDIAN :THREAD-SUPPORT :WILBUR-TRIPLES-AS-CLASSES
 :RDFCACHE-NATIVE-INTEGERS :ASDF.HIERARCHICAL-NAMES :DE.SETF.UTILITY :ASDF2
 :ASDF :ANSI-CL :COMMON-LISP :SBCL :SB-DOC :SB-TEST :SB-LDB :SB-PACKAGE-LOCKS
 :SB-UNICODE :SB-EVAL :SB-SOURCE-LOCATIONS :IEEE-FLOATING-POINT :X86-64 :UNIX
 :ELF :LINUX :SB-THREAD :LARGEFILE :GENCGC :STACK-GROWS-DOWNWARD-NOT-UPWARD
 :C-STACK-IS-CONTROL-STACK :LINKAGE-TABLE :COMPARE-AND-SWAP-VOPS
 :UNWIND-TO-FRAME-AND-CALL-VOP :RAW-INSTANCE-INIT-VOPS
 :STACK-ALLOCATABLE-CLOSURES :STACK-ALLOCATABLE-VECTORS
 :STACK-ALLOCATABLE-LISTS :STACK-ALLOCATABLE-FIXED-OBJECTS :ALIEN-CALLBACKS
 :CYCLE-COUNTER :COMPLEX-FLOAT-VOPS :FLOAT-EQL-VOPS :INLINE-CONSTANTS
 :MEMORY-BARRIER-VOPS :OS-PROVIDES-DLOPEN :OS-PROVIDES-DLADDR
 :OS-PROVIDES-PUTWC :OS-PROVIDES-SUSECONDS-T :OS-PROVIDES-GETPROTOBY-R
 :OS-PROVIDES-POLL)
*

stacktrace :

0: (SB-DEBUG::MAP-BACKTRACE #<CLOSURE (LAMBDA (ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::FRAME)) {101F1A1999}> :START 0 :COUNT 1152921504606846975)
1: (ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::PRINT-STACKTRACE #<SB-SYS:FD-STREAM for "file /var/tmp/logs/spocq.log" {101EF50001}> :START 0 :COUNT 1152921504606846975 :VERBOSITY 1 :PRINT-ARRAY
NIL :PRINT-PRETTY NIL)
2: ((FLET SB-THREAD::WITH-MUTEX-THUNK))
3: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-MUTEX]300))
4: (SB-THREAD::CALL-WITH-MUTEX #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK) {7FFF9FDAC5C9}> #S(SB-THREAD:MUTEX :NAME "trace lock" :%OWNER #<SB-THREAD:THREAD "algebra-dynamic-369178"
RUNNING {101EB470A1}> :STATE 1) #<SB-THREAD:THREAD "algebra-dynamic-369178" RUNNING {101EB470A1}> T)
5: ((FLET ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::CALL-WRITE-LOG))
6: ((FLET ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::HANDLE-ERROR) #<SB-INT:INVALID-ARRAY-INDEX-ERROR expected-type: (INTEGER 0 (64)) datum: 82>)
7: ((FLET #:FORM-FUN-[GUARDED-HANDLER]628))
8: ((FLET ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::GUARDED-HANDLER) #<SB-INT:INVALID-ARRAY-INDEX-ERROR expected-type: (INTEGER 0 (64)) datum: 82>)
9: (SIGNAL #<SB-INT:INVALID-ARRAY-INDEX-ERROR expected-type: (INTEGER 0 (64)) datum: 82>)
10: (ERROR SB-INT:INVALID-ARRAY-INDEX-ERROR :ARRAY #<(SIMPLE-ARRAY (UNSIGNED-BYTE 64) (64)) {10009954EF}> :AXIS NIL :DATUM 82 :EXPECTED-TYPE (INTEGER 0 (64)))
11: (SB-INT:INVALID-ARRAY-INDEX-ERROR #<(SIMPLE-ARRAY (UNSIGNED-BYTE 64) (64)) {10009954EF}> 82 64 NIL)
12: (SB-KERNEL::INVALID-ARRAY-INDEX-ERROR-HANDLER #<unavailable argument> #.(SB-SYS:INT-SAP #X7FFF9FDAD580) #<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP #X7FFF9FDACF40 :TYPE (* (STRUCT
SB-VM::OS-CONTEXT-T-STRUCT))> (533 148 84))
13: (SB-KERNEL:INTERNAL-ERROR #.(SB-SYS:INT-SAP #X7FFF9FDACF40) #<unavailable argument>)
14: ("foreign function: call_into_lisp")
15: ("foreign function: funcall2")
16: ("foreign function: interrupt_internal_error")
17: ("foreign function: #x41107A")
18: ((FLET #:BODY-FUN-[%PUTHASH]316))
19: (SB-KERNEL:%PUTHASH #<SB-PCL::WRAPPER #<STANDARD-CLASS ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::PAGE-MAILBOX> {101EC57601}> #<HASH-TABLE :TEST EQL :COUNT 63 {10004DA851}> ((:FLUSH
#<SB-PCL::WRAPPER #<STANDARD-CLASS ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::PAGE-MAILBOX> {101EC57601}>) (:FLUSH #<SB-PCL::WRAPPER #<STANDARD-CLASS
ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::PAGE-MAILBOX> {101EC57601}>) (:FLUSH #<SB-PCL::WRAPPER #<STANDARD-CLASS ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::PAGE-MAILBOX> {101EC57601}>) (:FLUSH
#<SB-PCL::WRAPPER #<STANDARD-CLASS ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::PAGE-MAILBOX> {101EC57601}>) (:FLUSH #<SB-PCL::WRAPPER #<STANDARD-CLASS
ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::PAGE-MAILBOX> {101EC57601}>) (:FLUSH #<SB-PCL::WRAPPER #<STANDARD-CLASS ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::PAGE-MAILBOX> {101EC57601}>) (:FLUSH
#<SB-PCL::WRAPPER #<STANDARD-CLASS ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::PAGE-MAILBOX> {101EC57601}>) (:FLUSH #<SB-PCL::WRAPPER #<STANDARD-CLASS
ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::PAGE-MAILBOX> {101EC57601}>) (:FLUSH #<SB-PCL::WRAPPER #<STANDARD-CLASS ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::PAGE-MAILBOX> {101EC57601}>)))
20: ((FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK))
21: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-RECURSIVE-LOCK]324))
22: (SB-THREAD::CALL-WITH-RECURSIVE-LOCK #<CLOSURE (FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK) {7FFF9FDAD899}> #S(SB-THREAD:MUTEX :NAME "World Lock" :%OWNER #<SB-THREAD:THREAD
"algebra-dynamic-369178" RUNNING {101EB470A1}> :STATE 2))
23: (SB-PCL::CHECK-WRAPPER-VALIDITY #<ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::PAGE-MAILBOX [[64:0/0/]x0.0@IO: (pt ptgraph ptName)] (ORG.DATAGRAPH.SPOCQ.ALGEBRA:|join| .
"59ff9eb4-faea-4b6a-a53d-72805d352c83") {1013FD1E71}>)

[Stas Boukarev (stassats)]

Are you modifying hash-tables concurrently?

[james anderson (james-anderson-0)]

good morning;

On 2014-08-04, at 08:46, Stas Boukarev <email address hidden> wrote:

when i modify a hash table concurrently, i lock it.

as far as i can discern the problematic operation from the stack (note please levels 19–23), the error is in pcl code, not application code.
from the entry at level 20, it would appear that this includes logic to lock resources.

> Are you modifying hash-tables concurrently?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1352156
>
> Title:
> Index 82 out of bounds for (SIMPLE-ARRAY ... (64)), should be
> nonnegative and <64. ...
>
> Status in Steel Bank Common Lisp:
> New
>
> Bug description:
> this error has occurred several times over the past day in an application which has been in service, unchanged, for weeks.
> one possible difference is that the it may have been presented with queries which have a higher operator count and thereby involve more threads, but it is not yet possible to quantify that difference.
>
> $ sbcl --version
> SBCL 1.0.46
> $ uname -a
> Linux nxp-qa 3.5.0-48-generic #72-Ubuntu SMP Mon Mar 10 23:18:29 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>
> we have yet to move from 1.0.46 as there has been no response to
> #1292673
>
> $ sbcl --core sbcl-spocq.core
> This is SBCL 1.0.46, an implementation of ANSI Common Lisp.
> More information about SBCL is available at <http://www.sbcl.org/>.
>
> SBCL is free software, provided as is, with absolutely no warranty.
> It is mostly in the public domain; some portions are provided under
> BSD-style licenses. See the CREDITS and COPYING files in the
> distribution for more information.
> * *features*
>
> (:BORDEAUX-THREADS :WILBUR2 :WILBUR :CL-PPCRE :SB-BSD-SOCKETS-ADDRINFO
> :CLOSER-MOP :DE.SETF.UTILITY.TEST CFFI-FEATURES:FLAT-NAMESPACE
> CFFI-FEATURES:X86-64 CFFI-FEATURES:UNIX :CFFI CFFI-SYS::FLAT-NAMESPACE
> :LITTLE-ENDIAN :THREAD-SUPPORT :WILBUR-TRIPLES-AS-CLASSES
> :RDFCACHE-NATIVE-INTEGERS :ASDF.HIERARCHICAL-NAMES :DE.SETF.UTILITY :ASDF2
> :ASDF :ANSI-CL :COMMON-LISP :SBCL :SB-DOC :SB-TEST :SB-LDB :SB-PACKAGE-LOCKS
> :SB-UNICODE :SB-EVAL :SB-SOURCE-LOCATIONS :IEEE-FLOATING-POINT :X86-64 :UNIX
> :ELF :LINUX :SB-THREAD :LARGEFILE :GENCGC :STACK-GROWS-DOWNWARD-NOT-UPWARD
> :C-STACK-IS-CONTROL-STACK :LINKAGE-TABLE :COMPARE-AND-SWAP-VOPS
> :UNWIND-TO-FRAME-AND-CALL-VOP :RAW-INSTANCE-INIT-VOPS
> :STACK-ALLOCATABLE-CLOSURES :STACK-ALLOCATABLE-VECTORS
> :STACK-ALLOCATABLE-LISTS :STACK-ALLOCATABLE-FIXED-OBJECTS :ALIEN-CALLBACKS
> :CYCLE-COUNTER :COMPLEX-FLOAT-VOPS :FLOAT-EQL-VOPS :INLINE-CONSTANTS
> :MEMORY-BARRIER-VOPS :OS-PROVIDES-DLOPEN :OS-PROVIDES-DLADDR
> :OS-PROVIDES-PUTWC :OS-PROVIDES-SUSECONDS-T :OS-PROVIDES-GETPROTOBY-R
> :OS-PROVIDES-POLL)
> *
>
>
> stacktrace :
>
> 0: (SB-DEBUG::MAP-BACKTRACE #<CLOSURE (LAMBDA (ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::FRAME)) {101F1A1999}> :START 0 :COUNT 1152921504606846975)
> 1: (ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::PRINT-STACKTRACE #<SB-SYS:FD-STREAM for "file /var/tmp/logs/spocq.log" {101EF50001}> :START 0 :COUNT 1152921504606846975 :VERBOSITY 1 :PRINT-ARRAY
> NIL...

[Stas Boukarev (stassats)]

How do you lock it? SBCL has a :synchronized keyword for make-hash-table for that purpose.

[james anderson (james-anderson-0)]

i noted earlier, the question does not appear to be, how i might or might not lock it.
the access is not in application code, it is in pcl.

stack level 23 reads as if the problem occurs in SB-PCL::CHECK-WRAPPER-VALIDITY

levels 20 — 21 read as if that code does not lock the hash table, but rather, as if the SB-PCL::WITH-WORLD-LOCK in SB-PCL::CHECK-WRAPPER-VALIDITY may be the only guard, with the presumption that all access to that hash table follow the same practice.

On 2014-08-05, at 11:31, Stas Boukarev <email address hidden> wrote:

> How do you lock it? SBCL has a :synchronized keyword for make-hash-table
> for that purpose.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1352156
>
> Title:
> Index 82 out of bounds for (SIMPLE-ARRAY ... (64)), should be
> nonnegative and <64. ...
>
> Status in Steel Bank Common Lisp:
> New
>
> Bug description:
> this error has occurred several times over the past day in an application which has been in service, unchanged, for weeks.
> one possible difference is that the it may have been presented with queries which have a higher operator count and thereby involve more threads, but it is not yet possible to quantify that difference.
>
> $ sbcl --version
> SBCL 1.0.46
> $ uname -a
> Linux nxp-qa 3.5.0-48-generic #72-Ubuntu SMP Mon Mar 10 23:18:29 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>
> we have yet to move from 1.0.46 as there has been no response to
> #1292673
>
> $ sbcl --core sbcl-spocq.core
> This is SBCL 1.0.46, an implementation of ANSI Common Lisp.
> More information about SBCL is available at <http://www.sbcl.org/>.
>
> SBCL is free software, provided as is, with absolutely no warranty.
> It is mostly in the public domain; some portions are provided under
> BSD-style licenses. See the CREDITS and COPYING files in the
> distribution for more information.
> * *features*
>
> (:BORDEAUX-THREADS :WILBUR2 :WILBUR :CL-PPCRE :SB-BSD-SOCKETS-ADDRINFO
> :CLOSER-MOP :DE.SETF.UTILITY.TEST CFFI-FEATURES:FLAT-NAMESPACE
> CFFI-FEATURES:X86-64 CFFI-FEATURES:UNIX :CFFI CFFI-SYS::FLAT-NAMESPACE
> :LITTLE-ENDIAN :THREAD-SUPPORT :WILBUR-TRIPLES-AS-CLASSES
> :RDFCACHE-NATIVE-INTEGERS :ASDF.HIERARCHICAL-NAMES :DE.SETF.UTILITY :ASDF2
> :ASDF :ANSI-CL :COMMON-LISP :SBCL :SB-DOC :SB-TEST :SB-LDB :SB-PACKAGE-LOCKS
> :SB-UNICODE :SB-EVAL :SB-SOURCE-LOCATIONS :IEEE-FLOATING-POINT :X86-64 :UNIX
> :ELF :LINUX :SB-THREAD :LARGEFILE :GENCGC :STACK-GROWS-DOWNWARD-NOT-UPWARD
> :C-STACK-IS-CONTROL-STACK :LINKAGE-TABLE :COMPARE-AND-SWAP-VOPS
> :UNWIND-TO-FRAME-AND-CALL-VOP :RAW-INSTANCE-INIT-VOPS
> :STACK-ALLOCATABLE-CLOSURES :STACK-ALLOCATABLE-VECTORS
> :STACK-ALLOCATABLE-LISTS :STACK-ALLOCATABLE-FIXED-OBJECTS :ALIEN-CALLBACKS
> :CYCLE-COUNTER :COMPLEX-FLOAT-VOPS :FLOAT-EQL-VOPS :INLINE-CONSTANTS
> :MEMORY-BARRIER-VOPS :OS-PROVIDES-DLOPEN :OS-PROVIDES-DLADDR
> :OS-PROVIDES-PUTWC :OS-PROVIDES-SUSECONDS-T :OS-PROVIDES-GETPROTOBY-R
> :OS-PROVIDES-POLL)
> *
>
>
> stacktrace :
>
> 0: (SB-DEBUG::MAP-BACKTRACE #<CLOSURE (LAMBDA (ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::FRAME)) {101F1A1999}> :START...

[Stas Boukarev (stassats)]

I didn't notice that the version is 1.0.46, that's too old.

[james anderson (james-anderson-0)]

ok.

we will try the current release and hope that something in the interim has corrected the issue raised in bug 1292673, to permit us to upgrade our version.
when we had attempted to advance to 1.1.8, it was not possible.
as there was no further response to that report, even after the issue was narrowed to some change between 1.0.51 and 1.0.52, we have hesitated to try again.

On 2014-08-05, at 13:04, Stas Boukarev <email address hidden> wrote:

> I didn't notice that the version is 1.0.46, that's too old.
>
> ** Changed in: sbcl
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1352156
>
> Title:
> Index 82 out of bounds for (SIMPLE-ARRAY ... (64)), should be
> nonnegative and <64. ...
>
> Status in Steel Bank Common Lisp:
> Incomplete
>
> Bug description:
> this error has occurred several times over the past day in an application which has been in service, unchanged, for weeks.
> one possible difference is that the it may have been presented with queries which have a higher operator count and thereby involve more threads, but it is not yet possible to quantify that difference.
>
> $ sbcl --version
> SBCL 1.0.46
> $ uname -a
> Linux nxp-qa 3.5.0-48-generic #72-Ubuntu SMP Mon Mar 10 23:18:29 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>
> we have yet to move from 1.0.46 as there has been no response to
> #1292673
>
> $ sbcl --core sbcl-spocq.core
> This is SBCL 1.0.46, an implementation of ANSI Common Lisp.
> More information about SBCL is available at <http://www.sbcl.org/>.
>
> SBCL is free software, provided as is, with absolutely no warranty.
> It is mostly in the public domain; some portions are provided under
> BSD-style licenses. See the CREDITS and COPYING files in the
> distribution for more information.
> * *features*
>
> (:BORDEAUX-THREADS :WILBUR2 :WILBUR :CL-PPCRE :SB-BSD-SOCKETS-ADDRINFO
> :CLOSER-MOP :DE.SETF.UTILITY.TEST CFFI-FEATURES:FLAT-NAMESPACE
> CFFI-FEATURES:X86-64 CFFI-FEATURES:UNIX :CFFI CFFI-SYS::FLAT-NAMESPACE
> :LITTLE-ENDIAN :THREAD-SUPPORT :WILBUR-TRIPLES-AS-CLASSES
> :RDFCACHE-NATIVE-INTEGERS :ASDF.HIERARCHICAL-NAMES :DE.SETF.UTILITY :ASDF2
> :ASDF :ANSI-CL :COMMON-LISP :SBCL :SB-DOC :SB-TEST :SB-LDB :SB-PACKAGE-LOCKS
> :SB-UNICODE :SB-EVAL :SB-SOURCE-LOCATIONS :IEEE-FLOATING-POINT :X86-64 :UNIX
> :ELF :LINUX :SB-THREAD :LARGEFILE :GENCGC :STACK-GROWS-DOWNWARD-NOT-UPWARD
> :C-STACK-IS-CONTROL-STACK :LINKAGE-TABLE :COMPARE-AND-SWAP-VOPS
> :UNWIND-TO-FRAME-AND-CALL-VOP :RAW-INSTANCE-INIT-VOPS
> :STACK-ALLOCATABLE-CLOSURES :STACK-ALLOCATABLE-VECTORS
> :STACK-ALLOCATABLE-LISTS :STACK-ALLOCATABLE-FIXED-OBJECTS :ALIEN-CALLBACKS
> :CYCLE-COUNTER :COMPLEX-FLOAT-VOPS :FLOAT-EQL-VOPS :INLINE-CONSTANTS
> :MEMORY-BARRIER-VOPS :OS-PROVIDES-DLOPEN :OS-PROVIDES-DLADDR
> :OS-PROVIDES-PUTWC :OS-PROVIDES-SUSECONDS-T :OS-PROVIDES-GETPROTOBY-R
> :OS-PROVIDES-POLL)
> *
>
>
> stacktrace :
>
> 0: (SB-DEBUG::MAP-BACKTRACE #<CLOSURE (LAMBDA (ORG.DATAGRAPH.SPOCQ.IMPLEMENTATION::FRAME)) {101F1A1999}> :START 0 :COUNT 1152921504606846975)
> 1: (ORG.DATAGRAPH.SPOCQ.I...