sb-heapdump
===========

sb-heapdump is a library for SBCL which writes graphs of Lisp objects to disk in the same format SBCL normally uses in memory.

sb-heapdump is currently supported on the x86, x86-64, and PPC platforms.

Features:

  * sb-heapdump supports all kinds of Lisp objects SBCL knows

  * reads data back very quickly

  * is highly SBCL specific.  No attempt at portability is made.

  * does expressly not define a forward- or backward-compatible format. Upgrades to SBCL will break support for previously dumped heapfiles.

  * as an extension, can dump entire packages with all their definitions

FIXME:

  * See below for various FIXME comments.

  * sb-heapdump keeps a global hash table of all code objects referencing foreign symbols.  Make sure to require sb-heapdump before loading any fasls referencing the linkage table.

  * Also note that an effort is made to fully support generic functions and CLOS classes and instances, but support for this is considered experimental until someone tells me that all the various caches CLOS keeps are faithfully preserved by dumping.

  * separating tagged and untagged objects would help gc performance

  * so would starting a new region every few pages (I think)

  * what about functions without a fixup vector?

Reading heapfiles
-----------------

 -- Function: LOAD-DUMPFILE (pathname &key customizer suppress-initializer)
     Load the dumpfile from PATHNAME, then run the initializer specified included in the dumpfile, if any.  Call the initializer with the object that has been loaded back and return the initializer's return value.  If no initializer is run, return the object directly.

     Multiple heap file segments can be concatenated into one file.  In this case, LOAD-DUMPFILE will load all segments found in orde.  The last segment's value will be returned.  Concatenation can be done using :if-exists :append while dumping, or simply using cat(1).

     Keyword arguments:

       * SUPRRESS-INITIALIZER (default nil) -- if true, suppress running the initializer and directly return the object in the dump file.

       * CUSTOMIZER -- override the LOAD-TIME-CUSTOMIZER specified when dumping. See below.

Dumping objects and packages
----------------------------

 -- Function: DUMP-OBJECT (object pathname &key if-exists initializer customizer load-time-customizer force print-statistics base-address)
     Write OBJECT to a heapfile at PATHNAME.

     Recursively walk all the graph of objects referenced from OBJECT and dump them too, except for objects assumed to be `unique'.  Unique objects are not dumped unless specified using FORCE; instead they will be assumed to exist in the target image already and references to them will be fixup up at load time.  It is an error if such an object cannot be found then.  See below for a details.

     Keyword arguments:

       * IF-EXISTS (one of :error (default), :rename-and-delete, or :append) -- passed to OPEN.  When using :append, a new segment fill be added to an existing heap file.  See LOAD-DUMPFILE for details.

       * INITIALIZER -- if specified, a function object of one argument to be run after the heap file has been loaded back into memory by LOAD-DUMPFILE. See there for details.

       * CUSTOMIZER -- An optional function of one argument called for every object dumped.  Possible return values:

            * (a) T Dumping of the object will then proceed normally.)

            * (b) As multiple values, (NIL; replacement object) The replacement value will be substituted for every reference to the original value while dumping.

            * (c) As multiple values, (:FIXUP; data1; data2) The  object will be replaced by a fixup to be resolved at load time. LOAD-DUMPFILE will call LOAD-TIME-CUSTOMIZER with data1 and data2 as its arguments and substitute references to the original object for its return value.

       * LOAD-TIME-CUSTOMIZER -- function to be dumped into the heapfile to resolve user fixups as specified in the description of CUSTOMIZER. Can be overriden at load time using the CUSTOMIZER argument to LOAD-HEAPFILE.

       * FORCE -- An optional list of objects specifying that these objects are to be dumped directly even if they would have been replaced with fixups otherwise.

       * PRINT-STATISTICS (boolean) -- print statistics about the number and kinds of objects dumped before returning

       * BASE-ADDRESS -- a memory address as an integer, aligned to a page boundary.  Write the heapfile so that it can be mapped without relocation if memory starting with BASE-ADDRESS is free (and lies within dynamic space).

 -- Function: DUMP-PACKAGES (packages pathname &key if-exists print-statistics customizer load-time-customizer initializer base-address systems system-packages)

     Dump the entire PACKAGES specified into a dumpfile.  This is roughly equivalent to (DUMP-OBJECT packages pathname :FORCE packages) except that it collects additional information about objects named by symbols in the packages specified (including function and class definitions) and makes sure to restore this data after loading.

     Keyword arguments:

       * INITIALIZER -- called with the list of packages after other initialization has been completed.

       * IF-EXISTS, PRINT-STATISTICS, CUSTOMIZER, LOAD-TIME-CUSTOMIZER, BASE-ADDRESS -- cf. DUMP-OBJECT

       * SYSTEMS -- list of ASDF system designators.  If specified, prepend a segment to the dumpfile containing the ASDF systems with an initializer that will restore them and require their dependencies before loading the main segment containing PACKAGES.

       * SYSTEM-PACKAGES -- list of packages that SYSTEMS were defined in.

Note that Lisp software can cause extensive changes to a Lisp image while it is loaded and run, many of which are not necessarily reflected in the actualy home package(s) of the software.  DUMP-PACKAGES cannot automatically determine which parts of the current Lisp image "belong" to the software that is to be dumped.  To make such software work with DUMP-PACKAGES, users will often have to customize the dumping procedure.  One way to do this is by specifying a custom INITIALIZER. For example, if the software stores data on the plist of symbols not contained in the packages to be dumped, write an initializer that restores these plists after loading.

Optimizing heapfiles
--------------------

Heap files that cannot be mapped to the base-address they were targetted for will be relocated automatically.  Multiple heap files expected to be loaded together (and heap files containing several segments) can be relocated in advance to avoid overlap and unnecessary relocation at load time.

(However, note that relocation is relatively fast and heap files generated by DUMP-PACKAGE usually spend more time in the fixup and initialization steps than in relocation.)

 -- Function: RELOCATE-DUMPFILES (pathnames &optional base-address)

     Rewrite the dumpfiles so that they will, by default, load into non-overlapping parts of memory, starting with BASE-ADDRESS.

 -- Function: RELOCATE-DUMPFILE (pathname &optional base-address)

     Rewrite the dumpfile at PATHNAME so that it will load to BASE-ADDRESS by default.

Integration into REQUIRE and ASDF
---------------------------------

sb-heapdump installs itself as a provider for REQUIRE.  Modules are searched in each directory specified by SB-HEAPDUMP:*CENTRAL-REGISTRY* with the downcased module name as file name and file type ".heap".

Heap files store in a registry directory should have been dumped using the :SYSTEMS argument to DUMP-PACKAGE.

Dependencies of the systems as declared using :DEPENDS-ON are loaded using REQUIRE.

Once a heap file has been found and loaded, it is automatically registered as an ASDF system and ignored by the sb-heapfile's module provider, so further invocations of REQUIRE and ASDF functions will compile and load its components as usual.

 -- Variable: *CENTRAL-REGISTRY*

     A list of directory designators evaluated and searched in order when looking for heapfile modules.  Defaults to the current directory and $SBCL_HOME in this order.

 -- Generic Function: DUMP-SYSTEM (system)

     Convenience function that ASDF systems can define a method that will dump the system into a file.  See demo.lisp in the sb-heapdump distribution for examples.

Executable files
----------------

 -- Function: MAKE-EXECUTABLE (heapfile &key output-pathname if-exists main-function)

     Create a file called OUTPUT-PATHNAME consisting of a trampoline binary and a copy of HEAPFILE.  (Optionally, an additional heapfile segment is appended that calls MAIN-FUNCTION with the binary's command line arguments in its initializer.)

     When executed, the generated file will run the `sbcl' binary as found in $PATH to load itself.

     OUTPUT-PATHNAME defaults to the name obtained by removing the type component from the pathname HEAPFILE.  For example, `foo.heap' is copied into `foo'.

DUMP-OBJECT behaviour for specific classes
------------------------------------------

The following types of objects can be dumped and are always dumped literally:

  * Immediate values (FIXNUM and CHARACTER)

  * BIGNUMs, SINGLE-FLOAT, DOUBLE-FLOAT, RATIO, COMPLEX

  * Lists

  * ARRAY (all types of arrays are supported, including single- and multi-dimensional arrays of all array element types known by SBCL, whether simple or not.  This includes strings.)

  * Instances (technically, SB-KERNEL:INSTANCE), including structure instances, CLOS instances, and conditions.  [Note: CLOS support is experimental.]

  * Code components (if specified literally; see below for the fixup behaviour of functions)

  * Closures

  * Uninterned symbols

  * Value cells (fixme: whatever that is anyway)

  * System area pointers (SAPs)

  * Weak pointers.  (The weak pointer value will be dumped and the weak reference to it preserved if the value is either (i) reachable through a non-weak reference from the object graph being dumped or (ii) treated as a fixup.  Else the weak pointer will load as a broken reference.)

The following types of objects are dumped only if specified by the FORCE argument, otherwise they are replaced by fixups.

  * Packages

  * Interned symbols (forcing a package also forces all symbols with that package as their home package)

  * Classes (technically, all of SB-KERNEL:LAYOUT, SB-KERNEL:CLASSOID and SB-KERNEL:CLASS).  Forcing a symbol also forces classes named by that symbol. [FIXME! KLUDGE! There is an unnamed class in SBCL.  It is currently dumped unconditionally, which cannot be right.]

  * NAMED-TYPE: Named types are replaced by a fixup if named by a symbol that is not being forced.  The fixup will automatically re-create the named type at load time, if necessary.

The following types of objects are dumped according to more complex heuristics.  (Notionally, these objects will be replaced by a fixup if they are identified by a symbol that is not forced.)

  * Except as noted below, ordinary functions (simple-funs) are replaced by fixups if all of the following conditions are true: (i) The function object itself is not being forced.  (ii) The function is named by a symbol or is named (SETF symbol).  (iii) The symbol is not being forced.  (iv) FDEFINITION for that function name actually returns the function object in question.  --- If a function is not replaced by a fixup, its code component is dumped, which implies dumping all its other entry points.

  * funcallable instances -- FIXME: except as noted below, funcallable instances are currently dumped unconditionally.  That can't be right, shouldn't the rules for simple-fun's apply here, too?

  * Generic functions: Slot accessors (SB-PCL::SLOT-ACCESSOR) are never dumped and instead recreated while loading the heap file, if necessary. Other generic function are treated like ordinary functions (see above).

  * An FDEFN object is replaced by a fixup unless any of the following conditions is true: (i) The function it points to is dumped literally.  (ii) Its name is a forced symbol.  (iii) Its name is a list containing a forced symbol.  (iv) It points to a CTOR or SLOT-ACCESSOR.

The following types of objects are never dumped literally:

  * Although technically simple functions, SB-PCL::FAST-METHODs are never dumped literally and instead recreated while loading the heap file, if necessary.  [FIXME!  there are fast methods that are closures, what happens then?]

  * Although technically funcallable instances, SB-PCL::CTORs are never dumped and instead recreated while loading the heap file, if necessary.

  * ARRAY-TYPEs are never dumped and instead recreated while loading the heap file, if necessary.

fixme: are there CTYPE structures other than named-type and array-type that can and need to be fixed up?  [union-type has a cache, but does the compiler depend on that?]

DUMP-PACKAGES details
---------------------

For every package and every symbol that has one of these packages as its home package, DUMP-PACKAGE installs an initializer that will restore

  * all class cells named by this symbol

  * for all SPECIALIZER-DIRECT-METHODs of those classes, the method-function-plist of their FAST-METHODs as well as most info-types in the compiler's INFO database for:

  * the symbol itself

  * the name (SETF symbol)

  * the slot reader, writer, and boundp accessors for all slot definitions of classes named by this symbol

  * the names of all the FAST-METHODs