wanted: by-value structures in SB-ALIEN

[Nikodemus Siivola]

Passing and returning structures by value is not supported by SB-ALIEN.

As of 07.06.2019, there's a bounty on this feature: https://www.bountysource.com/issues/75202399-wanted-by-value-structures-in-sb-alien

[Ekaterina Vaartis (vaartis)]

There's now a bunty on this: https://www.bountysource.com/issues/75202399-wanted-by-value-structures-in-sb-alien.

[Drew Crampsie (drewc)]

Now? 02 January 2009 is not "now", it's over 10 years ago. Why do you point attention to it now? Has any progress been made over the last 10 years 5 months and 5 days?

If not, does inflation matter? $83.58 would be what my local bank says.

Also, "bunty" is probably not what you meant. :)

[Stas Boukarev (stassats)]

What are you talking about?

[Ekaterina Vaartis (vaartis)]

The bounty has been created just today. I posted this bounty here for anyone who would find this here and would want to claim the bounty.

[Drew Crampsie (drewc)]

That's so very confusing, as the bounty page says :

"wanted: by-value structures in SB-ALIEN
 02 January 2009 Posted by Nikodemus Siivola
Passing and returning structures by value is not supported by SB-ALIEN."

Can you please update it to say that you are offering the money, and this is not 2009 but 2019?

Can you see where people may get confused?

[Ekaterina Vaartis (vaartis)]

Bountysource is a site where you offer money for solving bugs, this would be the only purpose for the bug to be there. The page there reflects the state of the bug on launchpad, i cannot alter neither the bug description, nor the date, as far as i know. The fact remains that the bounty is unclaimed and the feature is not implemented, the date does not mean much in the context, only the fact that it hasn't been implemented for ten years.

[Ekaterina Vaartis (vaartis)]

Actually, i found a way to edit the bug description, so i will add the bounty note to the description.

[Drew Crampsie (drewc)]

Thank you for explaining that to me. Hopefully your editing will make people less confused about who is offering what and when. Many blessings.

[g_o (go0)]

Hey, already started working on this; I got the return value partially working. It seems like a lot of other code on sb-alien is guess-work, should i also guess optimiztions or should i require some flag when compiling?

Thanks in advance

[g_o (go0)]

Update here: I'm almost finished, problem is I need to call a function i made that fixes something on alien-funcall. I made prints but somehow they won't show when i run a test lisp program? i might need some help

[Ekaterina Vaartis (vaartis)]

Did you try printing to a different stream with (print obj :output-stream stream)? Maybe stdout isn't set up there or something. Also, does anything that might show if it works work in there? Something like raising a condition?

[g_o (go0)]

I tried printing to different stream (stderr) to no avail; not sure what you mean by "raising a condition" but i can't think of a simple condition to conclude - i practically have 1 input in my control - the args, tried to do length mismatch but it's caught beforehand; the only hypothesis i have is that it might be optimized by a transform from aliencomp or something.. though i really am not knowledgeable in sbcl's optimizations to be honest.

[g_o (go0)]

so somehow the print shows on some contrib tests ... [visible confusion]

[g_o (go0)]

Update: somehow using defvar to extern-alien fixes the problem. I'll try to get rid of the cause root by the time I send the patch

[g_o (go0)]

Semester started so progress will be a bit slow, though it really is almost done

[g_o (go0)]

so i guess i'll just update that i got back to the issue and it partially works. what i have left is to deal with default alignment (not packing/inner padding), i.e: initial offset of the struct when passed. can anyone give me some pointers as to how to guess that offset on the sbcl side?

[g_o (go0)]

nevermind it seems like the optimization of passing args by registers is a little different for structs; and so i don't think it was really an offset problem. i'll try to fix it this week, hopefully it goes well and i'll finally have a patch

[g_o (go0)]

alright, so it works but now i get back to the problem i discussed with @vaartis where if i use dynamic binding (i.e: defvar/defparameter) for my alien function my changes show up, but if i use lexical (i.e: let) - as if i never changed anything (e.g: if i inserted a print it wouldn't show). changes to aliencomp though still show. i'm still pretty lost here - is this a host/target compilation thing?

[Ekaterina Vaartis (vaartis)]

Could you perhaps upload your version somewhere with a simple test?

[g_o (go0)]

@vaartis i can send you my patch but the problem is that it's incredibly important to fix it because define-alien-routine is using lexical variables so if you want to both pass and return struct you'd have to make an almost clone of it on the user side which is really clunky..

[g_o (go0)]

here's what i have so far.

[g_o (go0)]

these are external tests i have but didn't integrate yet

[g_o (go0)]

anyway what i sent should suffice for pass and return, if you want to do both i can send an example file of how it'd be done under this patch from user standpoint. it's not a good solution though, so i really prefer an explanation as to why alien-funcall problematic for lexical vars and if it's a separate issue - open another bug.

to be clear: it doesn't matter *how* i change alien-funcall as far as i could tell, it just uses whatever i had previously. this last edge case allegedly can be solved simply by changing a let to a defvar in define-alien-routine; but even if we settle on this horrible solution - it crashes compilation itself. i really am not sure what i can do here

[g_o (go0)]

alright i think i fixed most of the problems

[g_o (go0)]

umm @vaartis mind giving some sort of feedback on the latest patch i sent?

[Ekaterina Vaartis (vaartis)]

Sorry for not responding before. I'll try your patch tomorrow with the use case I had in mind initially and get back to you.

[Ekaterina Vaartis (vaartis)]

When building latest SBCL (d0243a9f9961f0afdc09b555821b88edb2488be9 on github) with your patch applied I get the following signal (which pauses compilation):

[314/314] src/code/last-file

; file: /home/vaartis/Code/sbcl/src/code/target-alieneval.lisp
; in: DEFUN SB-ALIEN:ALIEN-FUNCALL
; (T (ERROR "~S is not an alien function." SB-ALIEN:ALIEN))
;
; caught STYLE-WARNING:
; undefined function: COMMON-LISP:T
;
; compilation unit finished
; Undefined function:
; T
; caught 1 STYLE-WARNING condition
; printed 4 notes
While evaluating the form starting at line 23, column 0
  of #P"/home/vaartis/Code/sbcl/make-host-2.lisp":

However, afterwards the rest builds fine.

When trying the code you have and following the way you call the function in the tests, it seems like alien-funcall'ing the same externed function only works once, because afterwards the externed function value is modified to have an additional struct parameter (i'm trying a functon that returns a struct and has no arguments).

The tests do pass, but when I tried writing a function of my own and calling it, the data in the struct was all messed up. Not sure if i'm doing something wrong. The test lisp file is attached to this post. I'll also attach a C file to the following post since it seems like only one attachment is allowed.

[g_o (go0)]

about the compilation pause - i did not experience it so that's weird, i'll see if i can replicate that.
about multiple calls - it should inject arguments to a copy but i probably forgot to change it in the one of the steps, i'll see what's up
about your own test - my first impression is that it's an edge case i didn't notice. i think i know what it is and it shouldn't take too long to fix

stay tuned i guess

[g_o (go0)]

i think this should do the job

[g_o (go0)]

cleaned up a bit and changed something tiny. i think it's done

[Stas Boukarev (stassats)]

(defun alien-void-unparsed-type-p (type)
  (string-equal "(VALUES)" (princ-to-string type)))

doesn't look good...

[Ekaterina Vaartis (vaartis)]

With the latest patch, the first value in the passed truct did work as expected. However, in the test file I attached before there were two values and the second one (y) is now always zero when passed into C. In fact, every second field is equal to zero, and the next field after them has their value. Seems like an issue with offsets, perhaps. When I added a third field, it had the value that the third field was supposed to have. This is all despite unsigned int and unsigned-int (in lisp) both being 32bits. Attached is a modified test lisp file. I will also attach the modifed C test file in the next post.

Another thing i've noticed is when I try to create a boolean field in lisp, i always get

debugger invoked on a TYPE-ERROR in thread
#<THREAD "main thread" RUNNING {1004AE0993}>:
  The value
    T
  is not of type
    CHARACTER
  when binding CHAR

when declaring it

[Ekaterina Vaartis (vaartis)]

> third field was supposed to have
* the second field

[g_o (go0)]

@stassats oops forgot about that "^^, that hack should be easily replaceable though
@vaartis about the boolean - i'm not responsible for the typing system. i think i saw some
         sbcl kludges that said that alien system doesn't really support less than byte types
         (in case you're hoping for bitfield); i'm pretty sure i tested char,
         try using it instead. in any case i think it might be out of the scope of the issue.
         (i made no real distinction between types on my end anyways).
         hmm about 32-bit types on 64 bit machine i'm not sure if there's an elegant way to solve
         this; see, i let sbcl's existing argument packer deal with padding and offsetting.
         i prefer being verbose as a workaround using prefered offsetting/alignment attributes
         on gcc's end or on the lisp alien type end.

         tbh at this point i had my fun with this issue and while it's nice to have i
         don't care all that much about the beer money. so if anyone wanna improve this thing or try
         something different he/she is very welcome, hopefully we'll share both.
         i'm just really out of time for this issue, as another semester is coming my way
         and with things being crazy as they are right now.