The problem comes from the macro mod() used in computation.
The code in fbComposite() from fbpict.c reads like this:
if (srcRepeat)
{
y_src = mod (y_src - pSrc->pDrawable->y, pSrc->pDrawable->height);
if (h_this > pSrc->pDrawable->height - y_src)
h_this = pSrc->pDrawable->height - y_src;
y_src += pSrc->pDrawable->y;
}
While inspecting the values, we see that initially, y_src=871, pSrc->pDrawable->y=1024, pSrc->pDrawable->height=500
After computation of mod() y_src=895 (which is wrong) so that h_this = pSrc->pDrawable->height - y_src = -395
Passing a negative value to a CARD16 in mmx function will cause the crash. But the real problem is that the value returned by mod() is actually greater than the values passed which is not possible, so there should be no way that y_src is greater than pSrc->pDrawable->height and therefore h_this should/could not be negative.
mod() is defined as follow (earlier in that code):
Created attachment 403292
Proposed patch
The problem comes from the macro mod() used in computation.
The code in fbComposite() from fbpict.c reads like this:
if (srcRepeat) ->height) ; ->height - y_src) ->height - y_src;
{
y_src = mod (y_src - pSrc->pDrawable->y, pSrc->pDrawable
if (h_this > pSrc->pDrawable
h_this = pSrc->pDrawable
y_src += pSrc->pDrawable->y;
}
While inspecting the values, we see that initially, y_src=871, pSrc->pDrawable ->y=1024, pSrc->pDrawable ->height= 500
After computation of mod() y_src=895 (which is wrong) so that h_this = pSrc->pDrawable ->height - y_src = -395
Passing a negative value to a CARD16 in mmx function will cause the crash. But the real problem is that the value returned by mod() is actually greater than the values passed which is not possible, so there should be no way that y_src is greater than pSrc->pDrawable ->height and therefore h_this should/could not be negative.
mod() is defined as follow (earlier in that code):
# define mod(a,b) ((b) == 1 ? 0 : (a) >= 0 ? (a) % (b) : (b) - (-a) % (b))
Problem is that (-a) gets expanded as "-871 - 1024" (and *not* "- (871 - 1024)" as expected) and therefore "(b) - (-a) % (b)" = 500 - (-871 - 1024) = 895
TI think the following would be more appropriate:
# define mod(a,b) ((b) == 1 ? 0 : (a) >= 0 ? (a) % (b) : (b) - (-(a)) % (b))
That seems to fix the crash and produces the correct output.