I did some more debugging of this this morning, and here is a summary of what I found:

- js::Vector<long, 8ul, js::TempAllocPolicy>::calculateNewCapacity() doesn't actually appear to be inlined by the compiler in both the working / non-working cases.

- Using JS_NEVER_INLINE for js::Vector<T,N,AP>::calculateNewCapacity() makes the problem go away.

- In the non-working case with gcc-4.4, js::Vector<long, 8ul, js::TempAllocPolicy>::calculateNewCapacity() appears to be completely optimized for the append() case (ie, it completely ignores lengthInc, and just assumes this is always 1). This can be seen by looking at the generated code:

0000000000da3b9a <js::Vector<long, 8ul, js::TempAllocPolicy>::growStorageBy(unsigned long)>:
  da3b9a:    41 54                    push   %r12
  da3b9c:    48 8d 47 20              lea    0x20(%rdi),%rax
  da3ba0:    55                       push   %rbp
  da3ba1:    53                       push   %rbx
  da3ba2:    48 89 fb                 mov    %rdi,%rbx
  da3ba5:    48 83 ec 10              sub    $0x10,%rsp
  da3ba9:    48 39 47 08              cmp    %rax,0x8(%rdi)

Up until here, %rdi contains our instance pointer, and %rsi contains "incr". 0x8(%rdi) is the pointer to "mBegin" and 0x20(%rdi) is "storage". The last instruction is usingInlineStorage().

  da3bad:    48 8b 77 10              mov    0x10(%rdi),%rsi

Y'ouch. This overwrites "incr" with "mLength", which then gets passed to calculateNewCapacity() here:

  da3bb1:    48 8d 54 24 08           lea    0x8(%rsp),%rdx
  da3bb6:    75 68                    jne    da3c20 <js::Vector<long, 8ul, js::TempAllocPolicy>::growStorageBy(unsigned long)+0x86>
  da3bb8:    e8 85 ff ff ff           callq  da3b42 <T.2961>

%rdx now contains the address of where calculateNewCapacity() will store "newCap"

I'm not sure what this is doing, but it seems to pretty much boil down to "move 0x1 to (%rdx):

0000000000da3b42 <T.2961>:
  da3b42:    48 89 f0                 mov    %rsi,%rax
  da3b45:    48 83 ec 08              sub    $0x8,%rsp
  da3b49:    48 83 c0 01              add    $0x1,%rax
  da3b4d:    72 42                    jb     da3b91 <T.2961+0x4f>
  da3b4f:    48 b9 00 00 00 00 00     movabs $0xf000000000000000,%rcx
  da3b56:    00 00 f0
  da3b59:    48 85 c8                 test   %rcx,%rax
  da3b5c:    75 33                    jne    da3b91 <T.2961+0x4f>
  da3b5e:    48 83 f8 01              cmp    $0x1,%rax
  da3b62:    41 b8 01 00 00 00        mov    $0x1,%r8d
  da3b68:    76 13                    jbe    da3b7d <T.2961+0x3b>
  da3b6a:    48 0f bd f6              bsr    %rsi,%rsi
  da3b6e:    b9 3f 00 00 00           mov    $0x3f,%ecx
  da3b73:    83 f6 3f                 xor    $0x3f,%esi
  da3b76:    29 f1                    sub    %esi,%ecx
  da3b78:    ff c1                    inc    %ecx
  da3b7a:    49 d3 e0                 shl    %cl,%r8
  da3b7d:    4c 89 02                 mov    %r8,(%rdx)
  da3b80:    48 ba 00 00 00 00 00     movabs $0xf000000000000000,%rdx
  da3b87:    00 00 f0
  da3b8a:    b0 01                    mov    $0x1,%al
  da3b8c:    49 85 d0                 test   %rdx,%r8
  da3b8f:    74 07                    je     da3b98 <T.2961+0x56>
  da3b91:    e8 aa 01 87 ff           callq  613d40 <js::TempAllocPolicy::reportAllocOverflow() const@plt>
  da3b96:    31 c0                    xor    %eax,%eax
  da3b98:    5a                       pop    %rdx
  da3b99:    c3                       retq

Now, we call malloc with 0x1 * 8:

  da3bbd:    84 c0                    test   %al,%al
  da3bbf:    0f 84 a0 00 00 00        je     da3c65 <js::Vector<long, 8ul, js::TempAllocPolicy>::growStorageBy(unsigned long)+0xcb>
  da3bc5:    48 8b 6c 24 08           mov    0x8(%rsp),%rbp
  da3bca:    48 c1 e5 03              shl    $0x3,%rbp
  da3bce:    48 89 ef                 mov    %rbp,%rdi
  da3bd1:    e8 aa 6f 87 ff           callq  61ab80 <malloc@plt>

And so, growStorageBy() appears to succeed, yet it has allocated a lot less than the caller thinks. mLength gets incremented by the value of "incr", and then MakeRangeGCSafe() (from js::AutoVectorRooter<long>::growBy()) tramples all over the end of the buffer.