amd64 kmod symbol relocation

[Peter Wemm]

On Wednesday 06 June 2007 16:24, Dixit, Amol wrote:
> Hi,
> I am having issues loading a test module on amd64 platform, ONLY when
> its loaded below 2gb (0x80000000).
> The module is trying to access a function 'printf()' in a shared
> library which is beyond the 2gb boundary but the runtime loader is
> trying to relocate the symbol to some other offset in its .text
> section.
>
> (gdb) disass alloc_loader
> Dump of assembler code for function alloc_loader:
> 0xffffff000b3a2000 <alloc_loader+0>:    push   %rbp
> 0xffffff000b3a2001 <alloc_loader+1>:    mov    $0xb3a2026,%rdi
> 0xffffff000b3a2008 <alloc_loader+8>:    xor    %eax,%eax
> 0xffffff000b3a200a <alloc_loader+10>:   mov    %rsp,%rbp
> 0xffffff000b3a200d <alloc_loader+13>:   callq  0x8024b730
> <---- call printf()
> 0xffffff000b3a2012 <alloc_loader+18>:   leaveq
> 0xffffff000b3a2013 <alloc_loader+19>:   xor    %eax,%eax
> 0xffffff000b3a2015 <alloc_loader+21>:   retq
>
> (gdb) p/x frame.tf_rip
> $10 = 0xffffff008024b730                  <--- panic "privileged
> instruction fault" address
> (gdb) info sym 0xffffff008024b730
> No symbol matches 0xffffff008024b730.      <--- nothing here, hence
> the panic!
> (gdb) info sym 0xffffffff8024b730
> printf in section .text                           <--- here it is!
> beyond 2gb (sign-extended)
>
> The module is compiled with kernel memory model (mcmodel=kernel). Any
> ideas why the relocation is failing in this manner? How do we force
> kldload to use addresses above 2gb?
>
> Object code looks like this:
>
> alloc_loader():
>    0:   55                      push   %rbp
>    1:   48 c7 c7 00 00 00 00    mov    $0x0,%rdi
>                         4: R_X86_64_32S .rodata.str1.1+0x6
>    8:   31 c0                   xor    %eax,%eax
>    a:   48 89 e5                mov    %rsp,%rbp
>    d:   e8 00 00 00 00          callq  12 <alloc_loader+0x12>
>                         e: R_X86_64_PC32       
> printf+0xfffffffffffffffc 12:   c9                      leaveq
>   13:   31 c0                   xor    %eax,%eax
>   15:   c3                      retq
>
> I believe relocation types R_X86_64_32S and R_X86_64_PC32 require top
> 33 bits 1 or 0 and in case of 'printf' 0xffffff008024b730 doesn't
> obey it...I guess!
>
> Thanks,
> Amol

OK, you're running into the same problems we had at work.

It sounds like you're doing some evil things that are not in standard 
freebsd.  I'm guessing that you've extended kvm considerably so that it 
is now no longer restricted to 2GB.

The problem is that -mcmodel=kernel generates code that only allows for 
immediate asm offsets of +/- 31 bits.  Consider the instruction in the 
trace above for "callq alloc_loader".  Note that it is e8 00 00 00 00 - 
4 bytes only!  Not 64 bit.

If you are going to have modules loaded outside of that 2gb of space, 
then you will need to compile with -fpic.  You need a tiny tweak to the 
loader to allow this, but it works.

-fpic causes the compiler to make all external function calls and data 
references via indirect lookups, and then relocations are generated 
against a 64 bit jump (PLT) and offset table (GOT).  That allows the 
kld to make immediate mode jumps and data references to something that 
is more than 2GB away.

There is a small performance penalty for using -fpic mode (but NOT as 
much of a penalty as it is in i386 mode!), but it still isn't free.  
Your only alternative is to come up with a way of giving the loader a 
way to allocate space in that top 2gb of kvm.

We use -fpic kld modules at work FWIW.
-- 
Peter Wemm - peter at wemm.org; peter at FreeBSD.org; peter at yahoo-inc.com
"All of this is for nothing if we don't go to the stars" - JMS/B5