How can I modify the stack with nasm, x86_64, linux functions (using `ret` keyword)?

Question

TL;DR

How can I modify the stack while using ret or achieving similar effect while using something else?

Hello world,

I am trying to make a compiler for my language, currently everything is inlined and it makes the compilation slow for some steps so today I decided to try to optimise it using functions, though it keeps segfaulting, then I realised

This seems to not work:

;; main.s

BITS 64
segment .text

global _start

exit:
    mov rax, 60  ;; Linux syscall number for exit
    pop rdi      ;; Exit code
    syscall
    ret

write:
    mov rax, 1  ;; Linux syscall number for write
    mov rdi, 1  ;; File descriptor (1 = stdout)
    pop rsi     ;; Pointer to string
    pop rdx     ;; String length
    syscall
    ret

_start:
    mov rax, msg_len
    push rax

    mov rax, msg
    push rax

    call write

    mov rax, 0
    push rax

    call exit


segment .data

msg: db "Hello, world!", 10
msg_len: equ $-msg

My output for this is.... questionable:

$ nasm -felf64 main.s
$ ld -o main main.s
$ ./main
PHello, world!
@       @ @$@ @+ @2 @main.sexitwritemsgmsg_len__bss_start_edata_end.symtab.strtab.shstrtab.text.data9! @  !77!'Segmentation fault

• $? (exit code) is 139 (segfault)

While all inlined all works:

;; main1.s

BITS 64
segment .text

global _start

_start:
    mov rax, msg_len
    push rax

    mov rax, msg
    push rax

    mov rax, 1  ;; Linux syscall number for write
    mov rdi, 1  ;; File descriptor (1 = stdout)
    pop rsi     ;; Pointer to string
    pop rdx     ;; String length
    syscall

    mov rax, 0
    push rax

    mov rax, 60  ;; Linux syscall number for exit
    pop rdi      ;; Exit code
    syscall

segment .data

msg: db "Hello, world!", 10
msg_len: equ $-msg

My output is completely normal:

$ nasm -felf64 main1.s
$ ld -o main1 main1.o
$ ./main1
Hello, world!

• $? (exit code) is 0 (as specified in assembly, meaning success)

So now I'm here confused as I am a newbie at assembly what to do, even though I found related solutions like

• NASM push before ret

I am still confused how to take that in... Is there a way I can do it or am I stuck with inlining? Should I maybe switch assemblers all together from nasm to something else?

Thanks in advance

Answer 1

tl;dr

Remember that call is technically a push rip, and ret is technically a pop rip, so you pretty much messed up your stack in your example because you inadvertently pop it in the wrong spot.

More of an answer

Although you should probably properly learn how calling conventions work, I'm going to attempt an answer to briefly "soften" the idea, and for the fun of learning.

Abstractly speaking, in order to have functions, you must have something called stack frames, or else you'd have a pretty hard time managing local variables and getting ret to work. On x86_64, a stack frame is pretty much composed of a few things, in order.

• The function arguments, if there are any⁰,
  • If some arguments were passed in registers, this may be omitted.
• the return address,
  • The call instruction will push this onto the stack.
  • It's on you to make sure the ret instruction will pop this off the stack.
• optionally a frame pointer,
  • If your stack grows by a dynamic amount, this can keep track of the start of the frame.
  • Otherwise, if you know the stack size ahead of time, it's optional.
• and then your local state on the stack.

As long as execution stays within your little assembly space, you are technically free to pass arguments however you want¹ as long as you are aware of how instructions like call and ret manipulate the stack. The simplest way, in my opinion, is to make it sort of stack-based, so that your compiler would not need to worry about register allocation as much².

To keep things simple, I'd suggest using something like the x86 convention but applied to x86_64, as you seem to be using 64-bit code. That is to say, the caller function would push all of its arguments onto the stack (usually in reverse order), and then call the callee function. For example, for a 3-argument function, your stack would end up looking something like this (beware that the top of the stack is actually on the bottom).

+----------------+
| argument 2     |
+----------------+
| argument 1     |
+----------------+
| argument 0     |
+----------------+
| return address |
+----------------+
| local state    |
| ...            |
+----------------+

Also, I noticed that you never really made use of the rsp register. Depending on the design of your compiler, you technically could get away with this. Stack machines like the JVM rely solely on pushes and pops, anyway, I believe. As long as your pushes and pops match (especially call and ret, which act as a special push and pop), you should be fine.

---

⁰ Windows actually allocates at least an extra 32 bytes here for argument spilling, but you can probably ignore that in this case.

¹ There are specific calling conventions that dictate how parameters are passed from caller to callee and back. Beyond your programming exercise, I highly recommend reading about how they work, so that your compiler can output code that can easily be called by and easily call functions that weren't emitted by your compiler, or go the Forth way as Nate mentioned.

²goto 1