sys64738
/
airs-notes
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
airs-notes/maskray-4.md

20 KiB
Raw Blame History

LLD and GNU linker incompatibilities

Subtitle: Is LLD a drop-in replacement for GNU ld?

The motivation for this article was someone challenging the "drop-in replacement" claim on LLD's website (the discussion was about Linux-like ELF toolchain):

LLD is a linker from the LLVM project that is a drop-in replacement for system linkers and runs much faster than them. It also provides features that are useful for toolchain developers.

99.9% pieces of software work with LLD without a change. Some linker script applications may need an adaption (such adaption is oftentimes due to brittle assumptions: asking too much from GNU ld's behavior which should be fixed anyway). So I defended for this claim.

Piotr Kubaj said that this is a probably more of a marketing term than a technical term, the term tries to lure existing users into thinking "it's the same you know, but better!". I think that this is fair in some senses: for many applications LLD has achieved much faster speed and much lower memory usage than GNU ld. A more important thing is that LLD adds a third choice to the spectrum. It brings competitive pressure to both sides, gives incentive for improvement, and makes for more standardized future features/extensions. One reason that I am subscribed to the binutils mailing list is I want to participate in its design processes (I am proud to say that I have managed to find some early issues of various new things).

Anyway, I thought documenting the compatibility problems between the ELF ports of LLD and GNU ld is useful, not only to others but also to my future self, hence this article. I will try to describe GNU gold behaviors as well.

So here is the long list. Please keep in mind that many compatibility issues do not really matter and a user may never run into such an issue. Many of them just serve as educational purposes and my personal reference. There some some user perceivable differences but quite a lot are WONTFIX on both GNU ld and LLD. LLD, as a newer linker, has less legacy compatibility burden and can make good default choices in some cases and say no to some unneeded features/behaviors. A large number of features are duplicated in GNU ld's various ports. It is also common that one thing behaves this way in port A and another way in port B.

  • GNU ld reports gc-sections requires either an entry or an undefined symbol in a -r --gc-section link. LLD doesn't error (https://reviews.llvm.org/D84131#2162411). I am unsure whether such a diagnostic will be useful (an uncommon use case where the GC roots are more than the explict linker options).
  • The default image base for -no-pie links is different. For example, on x86-64, GNU ld defaults to 0x400000 while LLD defaults to 0x200000.
  • GNU ld synthesizes a STT_FILE symbol when copying non-STT_SECTION STB_LOCAL symbols. LLD doesn't.
  • Text relocations.
    • In GNU ld, -z notext/-z text/unspecified are a tri-state. For -z notext/unspecified, the dynamic tags DT_TEXTREL and DF_TEXTREL are added on demand. If unspecified and GNU ld is configured with --enable-textrel-check=warning, a warning will be issued.
    • LLD has two states and add DT_TEXTREL and DF_TEXTREL if -z notext is specified.
    • GNU ld supports more relocation types as text relocations.
  • Default library paths.
  • GNU ld supports grouped short options. This can sometimes cause surprising behaviors with misspelled or unimplemented options, e.g. -no-pie means -n -o -pie because GNU ld as of 2.35 has not implemented -no-pie. Nick Clifton committed Update the BFD linker so that it deprecates grouped short options. to deprecated the GNU ld feature. LLD never supports grouped short options.
  • Mixed SHF_LINK_ORDER and non-SHF_LINK_ORDER input sections in an output section.
  • LLD defaults to -z relro by default. This is probably not a good default but it is difficult to change now. I have a comment https://bugs.llvm.org/show_bug.cgi?id=48549. GNU ld warns for -z relro and -z norelro for non Linux/FreeBSD BFD emulations (e.g. -m aarch64elf).
  • Different archive member extraction semantics. See http://lld.llvm.org/ELF/warn_backrefs.html for details.
  • LLD --warn-backrefs warns for def.a ref.o def.so if def.a cannot satisfy previous unresolved symbols. LLD resolves the definition to def.a while GNU linkers resolve the definition to def.so.
  • GNU ld -static has traditionally been a synonym to -Bstatic. Recently on x86 it has been changed to behave a bit similar to gold -static, which disallows linking against shared objects. LLD -static is still a synonym to -Bstatic.
  • GNU linkers have a default --dynamic-linker. LLD doesn't.
  • GNU linkers warn for .gnu.warning.* sections. LLD doesn't. It is unclear the feature is useful. https://bugs.llvm.org/show_bug.cgi?id=42008
  • GNU ld has architecture-specific rules for relocations referencing undefined weak symbols. I don't think the GNU ld behaviors can be summarized (even by maintainers!). LLD's are consistent.
  • The conditions to create .interp are different. I believe GNU ld's is quite difficult to describe.
  • --no-allow-shlib-undefined and --rpath-link
    • GNU ld traces all shared objects (transitive DT_NEEDED dependencies) and emulates the bheavior of a dynamic loader to warn more cases.
    • gold and LLD implement a simplified version. They warn for shared objects whose DT_NEEDED dependencies are all seen as input files.
  • --fatal-warnings
    • GNU ld still reports warning: ....
    • LLD switches to error: ....
  • --no-relax
  • LLD places .rodata (among other SHF_ALLOC and non-SHF_WRITE-non-SHF_EXECINSTR sections) before .text (among other SHF_ALLOC and SHF_EXECINSTR sections).
  • .symtab/.shstrtab/.strtab in a linker script.
    • Ignored by GNU ld, therefore --orphan-handling= does not warn/error.
    • Respected by LLD
  • Whether ADDR(.foo) in a linker script can retain an empty output section.
    • GNU ld: no. Symbol assignments relative to such empty sections may have strange st_shndx.
    • LLD: yes.
  • If an undefined symbol is referenced by both R_X86_64_JUMP_SLOT (lazy) and R_X86_64_GLOB_DAT (non-lazy)
    • GNU ld generates .plt.got with R_X86_64_GLOB_DAT relocations. R_X86_64_JUMP_SLOT can thus be omitted to decrease the number of dynamic relocations.
    • LLD does not implement this saving. This naturally requires more than one pass scanning relocations which LLD doesn't do at present. https://bugs.llvm.org/show_bug.cgi?id=32938
  • GNU ld relaxes R_X86_64_GOTPCREL relocations with some forms (e.g. movq foo@GOTPCREL(%rip), %reg -> leaq foo(%rip), %reg). LLD never relaxes R_X86_64_GOTPCREL relocations.
  • GNU linkers give .gnu.linkonce* sections COMDAT section semantics. LLD simply ignores such sections. https://bugs.llvm.org/show_bug.cgi?id=31586 tracks when the hack can be removed.
  • GNU ld adds PT_PHDR and PT_INTERP together. A shared object usually does not have two program headers. In LLD, PT_PHDR is always added unless the address assignment makes is unsuitable to place program headers at all.
  • The conditions to create the dynamic symbol table .dynsym.
    • LLD: there is an input shared object, -pie/-shared, or --export-dynamic.
    • GNU ld's is quite complex. --export-dynamic is not special, though.
  • --export-dynamic-symbol
    • gold's implies -u.
    • GNU ld (from 2.35 onwards) and LLD's do not imply -u.
  • In GNU ld, a defined foo@v can suppress the extraction of an archive member defining foo@@v1. LLD treats them two separate symbols and thus the archive member extraction still happens. This can hardly matter. See All about symbol versioning for details.
  • Default program headers.
    • With traditional -z noseparate-code, GNU ld defaults to a RX/R/RW program header layout. With -z separate-code (default on Linux/x86 from binutils 2.31 onwards), GNU ld defaults to a R/RX/R/RW program header layout.
    • LLD defaults to R/RX/RW(RELRO)/RW(non-RELRO). With --rosegment, LLD uses RX/RW(RELRO)/RW(non-RELRO).
    • Placing all R before RX is preferable because it can save one program header and reduce alignment costs.
    • LLD's split of RW saves one maxpagesize alignment and can make the linked image smaller.
    • This breaks some assumptions that the (so-called) "text segment" precedes the (so-called) "data segment".
    • For example, certain programs expect .text is the first section of the text segment and specify -Ttext=0 to place the PF_R|PF_X program header at p_vaddr=0. This is a brittle assumption and should be avoided. If PT_PHDR is needed, --image-base=0 is a replacement. If PT_PHDR is not needed, .text 0 : { *(.text .text.*) } is a replacement.
  • GNU ld and gold define __rela_iplt_start in -no-pie mode, but not in -pie mode. glibc csu/libc-start.c needs it when statically linked, but not in the static pie mode. LLD does not distinguish -no-pie, -pie and -shared. https://bugs.llvm.org/show_bug.cgi?id=48674
  • LLD uses --no-apply-dynamic-relocs by default. GNU ld and gold fill in the GOT entries with link-time values. GNU ld only supports --no-apply-dynamic-relocs for aarch64 https://sourceware.org/bugzilla/show_bug.cgi?id=25891.
  • When relaxing R_X86_64_REX_GOTPCRELX, GNU ld suppresses the relaxation if it would cause relocation overflow. LLD does not perform the check.
  • GNU ld and gold allow --exclude-libs=b to hide b.a. LLD requires --exclude=libs=b.a.
  • Whether to use executable stack if neither -z execstack nor -z noexecstack is specified. GNU ld and gold check whether an object file does not have .note.GNU-stack. LLD ignores .note.GNU-stack and defaults to -z noexecstack.

Semantics of --wrap

GNU ld and LLD have slightly different --wrap semantics. I use "slightly" because in most use cases users will not observe a difference.

In GNU ld, --wrap only applies to undefined symbols. In LLD, --wrap happens after all other symbol resolution steps. The implementation is to mangle the symbol table of each object file (foo -> __wrap_foo; __real_foo -> foo) so that all relocations to foo or __real_foo will be redirected.

The LLD semantics have the advantage that non-LTO, LTO and relocatable link behaviors are consistent. I filed https://sourceware.org/bugzilla/show_bug.cgi?id=26358 for GNU ld.

# GNU ld: call bar
# LLD: call __wrap_bar
  call bar
.globl bar
bar:

Relocation referencing a local relative to a discarded input section

  • How to resolve a relocation referencing a STT_SECTION symbol associated to a discarded .debug_* input section.
    • GNU ld and gold have logic resolving the relocation to the prevailing section symbol.
    • LLD does not have the logic. LLD 11 defines some tombstone values.

A symbol table entry with STB_LOCAL binding that is defined relative to one of a group's sections, and that is contained in a symbol table section that is not part of the group, must be discarded if the group members are discarded. References to this symbol table entry from outside the group are not allowed.

ld.bfd/gold/lld error if the section containing the relocation is SHF_ALLOC. .debug* do not have the SHF_ALLOC flag and those relocations are allowed.

lld resolves such relocations to 0. ld.bfd and gold, however, have some CB_PRETEND/PRETEND logic to resolve relocations to the definitions in the prevailing comdat groups. The code is hacky and may not suit lld.

https://bugs.llvm.org/show_bug.cgi?id=42030

Canonical PLT entry for ifunc

How to handle a direct access relocation referencing a STT_GNU_IFUNC?

c.f. GNU indirect function.

__rela_iplt_start

GNU ld and gold define __rela_iplt_start in -no-pie mode, but not in -pie mode. LLD defines __rela_iplt_start regardless of -no-pie, -pie or -shared.

Static pie and static no-pie relocation processing is very different in glibc.

  • Static no-pie uses special code to process a magic array delimitered by __rela_iplt_start/__rela_iplt_end.
  • Static pie uses self-relocation to take care of R_*_IRELATIVE. The above magic array code is executed as well. If __rela_iplt_start/__rela_iplt_end are defined (like what LLD does), we will get 0 < __rela_iplt_start < __rela_iplt_end in csu/libc-start.c. ARCH_SETUP_IREL will crash when resolving the first relocation which has been processed.

nsz has a glibc patch that moves the self-relocation later so everything is set up for ifunc resolvers.

Linker scripts

  • Some linker script commands are unimplemented in LLD, e.g. BLOCK() as a compatibility alias for ALIGN(). BLOCK is documented in GNU ld as a compatibility alias and it is not widely used, so there is no reason to keep the kludge in LLD.
  • Some syntax is not recognized by LLD, e.g. LLD recognizes *(EXCLUDE_FILE(a.o) .text) but not EXCLUDE_FILE(a.o) *(.text) (https://bugs.llvm.org/show_bug.cgi?id=45764)
    • To me the unrecognized syntax is misleading.
    • If we support one way doing something, and the thing has several alternative syntax, we may not consider the alternative syntax just for the sake of completeness.
  • Different orphan section placement. GNU ld has very complex rules and certain section names have special semantics. LLD adopted some of its core ideas but made a lot of simplication:
  • For an error detected when processing a linker script, LLD may report it multiple times (e.g. ASSERT failure). GNU ld has such issues, too, but probably much rarer.
  • SORT commands
  • In LLD, AT(lma) forces creation of a new PT_LOAD program header. GNU ld can reuse the previous PT_LOAD program header if LMA addresses are contiguous. lma-offset.s
  • In LLD, non-SHF_ALLOC sections always get 0 sh_addr. In GNU ld you can have non-zero sh_addr but STT_SECTION relocations referencing such sections are not really meaningful.
  • Dot assignment (e.g. . = 4;) in an output section description.
    • GNU ld: dot advances to 4 relative to the start. If you consider . on the right hand side and ABSOLUTE(.), I don't think the behaviors are consistent.
    • LLD: move dot to address 0x4, which will usually trigger an unable to move location counter backward error. https://bugs.llvm.org/show_bug.cgi?id=41169

I'll also mention some LLD release notes which can demonstrate some GNU incompatibility in previous versions. (For example, if one thing is supported in version N, then the implication is that it is unsupported in previous versions. Well, it could be that it worked in older versions but regressed at some version. However, I don't know the existence of such things.)

LLD 12.0.0

  • -r --gc-sections is supported.
  • The archive member extraction semantics of COMMON symbols is by default (--fortran-common) compatible with GNU ld. You may want to read Semantics of a common definition in an archive for details. This is unfortunate.
  • .rel[a].plt and .rel[a].dyn get the SHF_INFO_LINK flag. https://reviews.llvm.org/D89828

LLD 11.0.0

LLD 10.0.0

LLD 9.0.0

  • The DF_STATIC_TLS flag is set for i386 and x86-64 when initial-exec TLS models are used.
  • Many configurations of the Linux kernel's arm32_7, arm64, powerpc64le and x86_64 ports can be linked by LLD.

LLD 8.0.0

In the LLD 7.0.0 era, https://reviews.llvm.org/D44264 was my first meaningful (albeit trivial) patch to LLD. Next I made contribution to --warn-backrefs. Then I started to fix tricky issues like copy relocations of a versioned symbol, duplicate --wrap, and section ranks. I have learned a lot from these code reviews. In the 8.0.0, 9.0.0 and 10.0.0 era, I have fixed a number of tricky issues and improved a dozen of other things and am confident to say that other than MIPS ;-) and certain other ISA specific things I am familiar with every corner of the code base. These are still challenges such as integration of RISC-V style linker relaxation and post-link optimization, improvement to some aspects of the linker script, but otherwise LLD is a stable and finished part of the toolchain.

A few random notes:

  • Symbol resolution can take 10%~20% time. Parallelization can theoretically improve the process but it is hard to overstate the challenge (if you additionally take into account determinism).
  • Be wary of feature creep. I have learned a lot from ELF design discussions on generic-abi and from Solaris "linker aliens" in particular. I am sorry to say so but some development on LLD indeed belongs to such categories. Sometimes it is difficult to draw a line between unsupported legacy and legacy we have to support.
  • LLD's adoption is now so large that sometimes a decision (like a default value for an option) cannot make everyone happy.