Ancillary Objects: Separate Debug ELF Files For Solaris
- by Ali Bahrami
We introduced a new object ELF object type in Solaris 11 Update 1
called the Ancillary Object. This posting describes them,
using material originally written during their development, the PSARC
arc case, and the Solaris Linker and Libraries Manual.
ELF objects contain allocable sections, which are mapped into memory
at runtime, and non-allocable sections, which are present in the file
for use by debuggers and observability tools, but which are not mapped or
used at runtime. Typically, all of these sections exist within a single
object file. Ancillary objects allow them to instead go into a separate
file.
There are different reasons given for wanting such a feature. One
can debate whether the added complexity is worth the benefit, and in
most cases it is not. However, one important case stands out
— customers
with very large 32-bit objects who are not ready or able to make the
transition to 64-bits.
We have customers who build extremely large 32-bit objects. Historically,
the debug sections in these objects have used the stabs format, which is
limited, but relatively compact. In recent years, the industry has
transitioned to the powerful but verbose DWARF standard. In some cases,
the size of these debug sections is large enough to push the total
object file size past the fundamental 4GB limit for 32-bit ELF object files.
The best, and ultimately only, solution to overly large objects is to
transition to 64-bits. However, consider environments where:
Hundreds of users may be executing the code on large shared
systems. (32-bits use less memory and bus bandwidth, and on sparc
runs just as fast as 64-bit code otherwise).
Complex finely tuned code, where the original authors may no
longer be available.
Critical production code, that was expensive to qualify and
bring online, and which is otherwise serving its intended purpose
without issue.
Users in these risk adverse and/or high scale categories have good reasons to
push 32-bits objects to the limit before moving on. Ancillary
objects offer these users a longer runway.
Design
The design of ancillary objects is intended to be simple, both to
help human understanding when examining elfdump output, and to lower
the bar for debuggers such as dbx to support them.
The primary and ancillary objects have the same set of
section headers, with the same names, in the same order
(i.e. each section has the same index in both files).
A single added section of type SHT_SUNW_ANCILLARY is added to
both objects, containing information that allows a debugger to
identify and validate both files relative to each other. Given one
of these files, the ancillary section allows you to identify the other.
Allocable sections go in the primary object, and non-allocable
ones go into the ancillary object. A small set of non-allocable
objects, notably the symbol table, are copied into both objects.
As noted above, most sections are only written to one of the
two objects, but both objects have the same section header
array. The section header in the file that does not contain
the section data is tagged with the SHF_SUNW_ABSENT section
header flag to indicate its placeholder status.
Compiler writers and others who produce objects can set the
SUNW_SHF_PRIMARY section header flag to mark non-allocable sections
that should go to the primary object rather than the ancillary.
If you don't request an ancillary object, the Solaris ELF format is
unchanged. Users who don't use ancillary objects do not pay for
the feature. This is important, because they exist to serve a small
subset of our users, and must not complicate the common case.
If you do request an ancillary object, the runtime behavior of the
primary object will be the same as that of a normal object. There
is no added runtime cost.
The primary and ancillary object together represent a logical single
object. This is facilitated by the use of a single set of section
headers. One can easily imagine a tool that can merge a primary and
ancillary object into a single file, or the reverse. (Note that although
this is an interesting intellectual exercise, we don't actually supply
such a tool because there's little practical benefit above and beyond
using ld to create the files).
Among the benefits of this approach are:
There is no need for per-file symbol tables to reflect the
contents of each file. The same symbol table that would be
produced for a standard object can be used.
The section contents are identical in either case
— there is no
need to alter data to accommodate multiple files.
It is very easy for a debugger to adapt to these new files, and
the processing involved can be encapsulated in input/output routines.
Most of the existing debugger implementation applies without
modification.
The limit of a 4GB 32-bit output object is now raised to 4GB of code,
and 4GB of debug data. There is also the future possibility
(not currently supported) to support multiple ancillary objects,
each of which could contain up to 4GB of additional debug data.
It must be noted however that the 32-bit DWARF debug format is
itself inherently 32-bit limited, as it uses 32-bit offsets
between debug sections, so the ability to employ multiple ancillary
object files may not turn out to be useful.
Using Ancillary Objects (From the Solaris Linker and Libraries Guide)
By default, objects contain both allocable and non-allocable sections. Allocable sections are
the sections that contain executable code and the data needed by that code
at runtime. Non-allocable sections contain supplemental information that is not required to execute an
object at runtime. These sections support the operation of debuggers and other observability
tools. The non-allocable sections in an object are not loaded into memory at
runtime by the operating system, and so, they have no impact on memory
use or other aspects of runtime performance no matter their size.
For convenience, both allocable and non-allocable sections are normally maintained in the same
file. However, there are situations in which it can be useful to separate
these sections.
To reduce the size of objects in order to improve the speed at which they can be copied across wide area networks.
To support fine grained debugging of highly optimized code
requires considerable debug data. In modern systems, the debugging data
can easily be larger than the code it describes. The size of a 32-bit
object is limited to 4 Gbytes. In very large 32-bit objects, the debug
data can cause this limit to be exceeded and prevent the creation of the
object.
To limit the exposure of internal implementation details.
Traditionally, objects have been stripped of non-allocable sections in order to address these
issues. Stripping is effective, but destroys data that might be needed later. The
Solaris link-editor can instead write non-allocable sections to an ancillary object. This feature is enabled
with the -z ancillary command line option.
$ ld ... -z ancillary[=outfile] ...By default, the ancillary file is given the same name as the
primary output object, with a .anc file extension. However, a different name can be
provided by providing an outfile value to the -z ancillary option.
When -z ancillary is specified, the link-editor performs the following actions.
All allocable sections are written to the primary object. In
addition, all non-allocable sections containing one or more input
sections that have the SHF_SUNW_PRIMARY section header flag set are written to the primary object.
All remaining non-allocable sections are written to the ancillary object.
The following non-allocable sections are written to both the primary object and ancillary object.
.shstrtab
The section name string table.
.symtab
The full non-dynamic symbol table.
.symtab_shndx
The symbol table extended index section associated with .symtab.
.strtab
The non-dynamic string table associated with .symtab.
.SUNW_ancillary
Contains the information required to identify the primary and ancillary objects, and to identify the object being examined.
The primary object and all ancillary objects contain the same
array of sections headers. Each section has the same section index in
every file.
Although the primary and ancillary objects all define the same
section headers, the data for most sections will be written to a single
file as described above. If the data for a section is not present in a
given file, the SHF_SUNW_ABSENT section header flag is set, and the sh_size field is 0.
This organization makes it possible to acquire a full list of section headers,
a complete symbol table, and a complete list of the primary and
ancillary objects from either of the primary or ancillary objects.
The following example illustrates the underlying implementation of ancillary objects. An ancillary object
is created by adding the -z ancillary command line option to an otherwise normal
compilation. The file utility shows that the result is an executable named a.out,
and an associated ancillary object named a.out.anc.
$ cat hello.c
#include <stdio.h>
int
main(int argc, char **argv)
{
(void) printf("hello, world\n");
return (0);
}
$ cc -g -zancillary hello.c
$ file a.out a.out.anc
a.out: ELF 32-bit LSB executable 80386 Version 1 [FPU], dynamically
linked, not stripped, ancillary object a.out.anc
a.out.anc: ELF 32-bit LSB ancillary 80386 Version 1, primary object a.out
$ ./a.out
hello worldThe resulting primary object is an ordinary executable that can be executed in
the usual manner. It is no different at runtime than an executable
built without the use of ancillary objects, and then stripped of non-allocable content using
the strip or mcs commands.
As previously described, the primary object and ancillary objects contain the same section
headers. To see how this works, it is helpful to use
the elfdump utility to display these section headers and compare them. The following
table shows the section header information for a selection of headers from the
previous link-edit example.
Index
Section Name
Type
Primary Flags
Ancillary Flags
Primary Size
Ancillary Size
13
.text
PROGBITS
ALLOC EXECINSTR
ALLOC EXECINSTR SUNW_ABSENT
0x131
0
20
.data
PROGBITS
WRITE
ALLOC
WRITE ALLOC SUNW_ABSENT
0x4c
0
21
.symtab
SYMTAB
0
0
0x450
0x450
22
.strtab
STRTAB
STRINGS
STRINGS
0x1ad
0x1ad
24
.debug_info
PROGBITS
SUNW_ABSENT
0
0
0x1a7
28
.shstrtab
STRTAB
STRINGS
STRINGS
0x118
0x118
29
.SUNW_ancillary
SUNW_ancillary
0
0
0x30
0x30
The data for most sections is only present in one of the
two files, and absent from the other file. The SHF_SUNW_ABSENT section header flag is
set when the data is absent. The data for allocable sections needed at
runtime are found in the primary object. The data for non-allocable sections used
for debugging but not needed at runtime are placed in the ancillary file.
A small set of non-allocable sections are fully present in both files. These
are the .SUNW_ancillary section used to relate the primary and ancillary objects together,
the section name string table .shstrtab, as well as the symbol table.symtab,
and its associated string table .strtab.
It is possible to strip the symbol table from the primary object.
A debugger that encounters an object without a symbol table can use the
.SUNW_ancillary section to locate the ancillary object, and access the symbol contained within.
The primary object, and all associated ancillary objects, contain a .SUNW_ancillary section
that allows all the objects to be identified and related together.
$ elfdump -T SUNW_ancillary a.out a.out.anc
a.out:
Ancillary Section: .SUNW_ancillary
index tag value
[0] ANC_SUNW_CHECKSUM 0x8724
[1] ANC_SUNW_MEMBER 0x1 a.out
[2] ANC_SUNW_CHECKSUM 0x8724
[3] ANC_SUNW_MEMBER 0x1a3 a.out.anc
[4] ANC_SUNW_CHECKSUM 0xfbe2
[5] ANC_SUNW_NULL 0
a.out.anc:
Ancillary Section: .SUNW_ancillary
index tag value
[0] ANC_SUNW_CHECKSUM 0xfbe2
[1] ANC_SUNW_MEMBER 0x1 a.out
[2] ANC_SUNW_CHECKSUM 0x8724
[3] ANC_SUNW_MEMBER 0x1a3 a.out.anc
[4] ANC_SUNW_CHECKSUM 0xfbe2
[5] ANC_SUNW_NULL 0 The ancillary sections for both objects contain the same number of elements, and
are identical except for the first element. Each object, starting with the primary
object, is introduced with a MEMBER element that gives the file name, followed
by a CHECKSUM that identifies the object. In this example, the primary object
is a.out, and has a checksum of 0x8724. The ancillary object is a.out.anc,
and has a checksum of 0xfbe2. The first element in a .SUNW_ancillary section,
preceding the MEMBER element for the primary object, is always a CHECKSUM
element, containing the checksum for the file being examined.
The presence of a .SUNW_ancillary section in an object indicates that the object has associated ancillary objects.
The names of the primary and all associated ancillary objects can
be obtained from the ancillary section from any one of the files.
It is possible to determine which file is being examined from the
larger set of files by comparing the first checksum value to the
checksum of each member that follows.
Debugger Access and Use of Ancillary Objects
Debuggers and other observability tools must merge the information found in the primary
and ancillary object files in order to build a complete view of the
object. This is equivalent to processing the information from a single file. This
merging is simplified by the primary object and ancillary objects containing the same
section headers, and a single symbol table.
The following steps can be used by a debugger to assemble the
information contained in these files.
Starting with the primary object, or any of the ancillary objects, locate the .SUNW_ancillary
section. The presence of this section identifies the object as part of
an ancillary group, contains information that can be used to obtain a
complete list of the files and determine which of those files is the one
currently being examined.
Create a section header array in memory, using the section header array from the object being examined as an initial template.
Open and read each file identified by the .SUNW_ancillary
section in turn. For each file, fill in the in-memory section header
array with the information for each section that does not have the SHF_SUNW_ABSENT flag set.
The result will be a complete in-memory copy of the section headers with
pointers to the data for all sections. Once this information has been acquired,
the debugger can proceed as it would in the single file case,
to access and control the running program.
Note - The ELF definition of ancillary objects provides for a single primary object, and
an arbitrary number of ancillary objects. At this time, the Oracle Solaris link-editor
only produces a single ancillary object containing all non-allocable sections. This may change
in the future. Debuggers and other observability tools should be written to handle
the general case of multiple ancillary objects.
ELF Implementation Details (From the Solaris Linker and Libraries Guide)
To implement ancillary objects, it was necessary to extend the
ELF format to add a new object type (ET_SUNW_ANCILLARY), a new
section type (SHT_SUNW_ANCILLARY), and 2 new section header
flags (SHF_SUNW_ABSENT, SHF_SUNW_PRIMARY). In this section, I will
detail these changes, in the form of diffs to the Solaris
Linker and Libraries manual.
Part IV ELF Application Binary Interface
Chapter 13: Object File Format
Object File Format
Edit Note: This existing section
at the beginning of the chapter describes the ELF header. There's a
table of object file types, which now includes the new
ET_SUNW_ANCILLARY type.
e_type
Identifies the object file type, as listed in the following table.
NameValueMeaning
ET_NONE0No file type
ET_REL1Relocatable file
ET_EXEC2Executable file
ET_DYN3Shared object file
ET_CORE4Core file
ET_LOSUNW0xfefeStart operating system specific range
ET_SUNW_ANCILLARY0xfefeAncillary object file
ET_HISUNW0xfefdEnd operating system specific range
ET_LOPROC0xff00Start processor-specific range
ET_HIPROC0xffffEnd processor-specific range
Sections
Edit Note: This overview section defines the section header
structure, and provides a high level description of known sections. It
was updated to define the new SHF_SUNW_ABSENT and SHF_SUNW_PRIMARY flags
and the new SHT_SUNW_ANCILLARY section.
...
sh_type
Categorizes the section's contents and semantics. Section types and their descriptions are listed in Table 13-5.
sh_flags
Sections support 1-bit flags that describe miscellaneous attributes.
Flag definitions are listed in Table 13-8.
...
Table 13-5 ELF Section Types, sh_type
NameValue
.
.
.
SHT_LOSUNW0x6fffffee
SHT_SUNW_ancillary0x6fffffee
.
.
.
...
SHT_LOSUNW - SHT_HISUNW
Values in this inclusive range are reserved for Oracle Solaris OS semantics.
SHT_SUNW_ANCILLARY
Present when a given object is part of a group of ancillary objects.
Contains information required to identify all the files that make up
the group. See Ancillary Section.
...
Table 13-8 ELF Section Attribute Flags
NameValue
.
.
.
SHF_MASKOS0x0ff00000
SHF_SUNW_NODISCARD0x00100000
SHF_SUNW_ABSENT0x00200000
SHF_SUNW_PRIMARY0x00400000
SHF_MASKPROC0xf0000000
.
.
.
...
SHF_SUNW_ABSENT
Indicates that the data for this section is not present in this file.
When ancillary objects are created, the primary object
and any ancillary objects, will all have the same section header array,
to facilitate merging them to form a complete view of the object, and to
allow them to use the same symbol tables. Each file contains a subset
of the section data. The data for allocable sections is written to the
primary object while the data for non-allocable sections is written to
an ancillary file. The SHF_SUNW_ABSENT flag is used to indicate that the data
for the section is not present in the object being examined.
When the SHF_SUNW_ABSENT flag is set, the sh_size field of the section
header must be 0.
An application encountering an SHF_SUNW_ABSENT section can choose to ignore
the section, or to search for the section data within one
of the related ancillary files.
SHF_SUNW_PRIMARY
The default behavior when ancillary objects are created is to write
all allocable sections to the primary object and all non-allocable
sections to the ancillary objects. The SHF_SUNW_PRIMARY flag overrides
this behavior. Any output section containing one more input section
with the SHF_SUNW_PRIMARY flag set is written to the primary object without
regard for its allocable status.
...
Two members in the section header, sh_link, and sh_info, hold special
information, depending on section type.
Table 13-9 ELF sh_link and sh_info Interpretation
sh_typesh_linksh_info
.
.
.
SHT_SUNW_ANCILLARY
The section header index of the associated string table.
0
.
.
.
Special Sections
Edit Note: This section describes the
sections used in Solaris ELF objects, using the types defined in the
previous description of section types. It was updated
to define the new .SUNW_ancillary (SHT_SUNW_ANCILLARY) section.
Various sections hold program and control information. Sections in the following
table are used by the system and have the indicated types and attributes.
Table 13-10 ELF Special Sections
NameTypeAttribute
.
.
.
.SUNW_ancillarySHT_SUNW_ancillaryNone
.
.
.
...
.SUNW_ancillary
Present when a given object is part of a group of ancillary objects.
Contains information required to identify all the files that make up
the group. See Ancillary Section for details.
...
Ancillary Section
Edit Note: This new section provides the format reference describing the
layout of a .SUNW_ancillary section and the meaning of the various tags.
Note that these sections use the same tag/value concept used for
dynamic and capabilities sections, and will be familiar to anyone
used to working with ELF.
In addition to the primary output object, the Solaris link-editor
can produce one or more ancillary objects. Ancillary objects contain
non-allocable sections that would normally be written to the primary
object. When ancillary objects are produced, the primary object and all
of the associated ancillary objects contain a SHT_SUNW_ancillary section,
containing information that identifies these related objects. Given any
one object from such a group, the ancillary section provides the information
needed to identify and interpret the others.
This section contains an array of the following structures. See sys/elf.h.
typedef struct {
Elf32_Word a_tag;
union {
Elf32_Word a_val;
Elf32_Addr a_ptr;
} a_un;
} Elf32_Ancillary;
typedef struct {
Elf64_Xword a_tag;
union {
Elf64_Xword a_val;
Elf64_Addr a_ptr;
} a_un;
} Elf64_Ancillary;
For each object with this type, a_tag controls the interpretation of a_un.
a_val
These objects represent integer values with various
interpretations.
a_ptr
These objects represent file offsets or addresses.
The following ancillary tags exist.
Table 13-NEW1 ELF Ancillary Array Tags
NameValuea_un
ANC_SUNW_NULL0Ignored
ANC_SUNW_CHECKSUM1a_val
ANC_SUNW_MEMBER2a_ptr
ANC_SUNW_NULL
Marks the end of the ancillary section.
ANC_SUNW_CHECKSUM
Provides the checksum for a file in the c_val element. When
ANC_SUNW_CHECKSUM precedes the first instance of ANC_SUNW_MEMBER,
it provides the checksum for the object from which the ancillary
section is being read. When it follows an ANC_SUNW_MEMBER tag,
it provides the checksum for that member.
ANC_SUNW_MEMBER
Specifies an object name. The a_ptr element contains the string table
offset of a null-terminated string, that provides the file name.
An ancillary section must always contain an ANC_SUNW_CHECKSUM before
the first instance of ANC_SUNW_MEMBER, identifying the current object.
Following that, there should be an ANC_SUNW_MEMBER for each object
that makes up the complete set of objects. Each ANC_SUNW_MEMBER should
be followed by an ANC_SUNW_CHECKSUM for that object. A typical ancillary
section will therefore be structured as:
TagMeaning
ANC_SUNW_CHECKSUMChecksum of this object
ANC_SUNW_MEMBERName of object #1
ANC_SUNW_CHECKSUMChecksum for object #1
.
.
.
ANC_SUNW_MEMBERName of object N
ANC_SUNW_CHECKSUMChecksum for object N
ANC_SUNW_NULL
An object can therefore identify itself by comparing the initial
ANC_SUNW_CHECKSUM to each of the ones that follow, until it finds
a match.
Related Other Work
The GNU developers have also encountered the need/desire to support
separate debug information files, and use the solution
detailed at
http://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html.
At the current time, the separate debug file is constructed by building
the standard object first, and then copying the debug data out of it
in a separate post processing step, Hence, it is limited to a total of 4GB
of code and debug data, just as a single object file would
be. They are aware of this, and I have seen online comments indicating that
they may add direct support for generating these separate files to their
link-editor.
It is worth noting that the GNU objcopy utility is available on Solaris,
and that the Studio dbx debugger is able to use these GNU style separate
debug files even on Solaris. Although this is interesting in terms giving
Linux users a familiar environment on Solaris, the 4GB limit means it is
not an answer to the problem of very large 32-bit objects. We
have also encountered issues with objcopy not understanding Solaris-specific
ELF sections, when using this approach.
The GNU community also has a current effort to adapt
their DWARF debug sections in order to move them to separate files before
passing the relocatable objects to the linker. The details of
Project Fission can be found at
http://gcc.gnu.org/wiki/DebugFission.
The goal of this project appears to be to reduce the amount of data
seen by the link-editor. The primary effort revolves around moving
DWARF data to separate .dwo files so that the link-editor never encounters
them. The details of modifying the DWARF data to be usable in this form
are involved — please see the above URL for details.
