+ xo-alloc + xo-object + xo-alloc docs + GC utests

docs/CMakeLists.txt

@@ -0,0 +1,9 @@
+# xo-alloc/docs/CMakeLists.txt
+
+xo_doxygen_collect_deps()
+xo_docdir_doxygen_config()
+xo_docdir_sphinx_config(
+    index.rst install.rst introduction.rst implementation.rst)
+
+# see xo-reader/doc or xo-unit/doc for working examples
+# example.rst install.rst implementation.rst

docs/README

@@ -0,0 +1,41 @@
+standalone build
+
+        +-----------------------------------------------+
+        | cmake                                         |
+        |   CMakeLists.txt                              |
+        |   $PREFIX/share/cmake/xo_macros/xo_cxx.cmake  |
+        +-----------------------------------------------+
+           |
+           |                                                  +----------------------+
+           +------------------------------------------------->| .build/docs/Doxyfile |
+           |                                                  +----------------------+
+           |                                                     ^
+           |                                 (cmake)             |
+           |                                        /------------/
+           |                                        |
+           |     +---------------------------------------+          +-----------------+
+           +---->| doxygen                               |--------->| .build/docs/dox |
+           |     |   $PREFIX/share/xo-macros/Doxyfile.in | (doxygen)|   +- html/      |
+           |     +---------------------------------------+          |   +- xml/       |
+           |                                                        +-----------------+
+           |                                                             |
+           |                                                             |(sphinx)
+           |                                                             |
+           |                                                             v
+           |     +---------------------------------------+        +--------------------+
+           \---->| sphinx                                |------->| .build/docs/sphinx |
+                 |   +- conf.py                          |        |   +- html/         |
+                 |   +- _static/                         |        +--------------------+
+                 |   +- *.rst                            |
+                 +---------------------------------------+
+
+umbrella build relies on top-level cmake macros
+
+files
+
+   README            this file
+   CMakeLists.txt    build entry point
+   conf.py           sphinx config
+   _static           static files for sphinx
+
+   index.rst         toplevel sphinx document; entry point

docs/_static/README

@@ -0,0 +1 @@
+add any static {.html, .js, ..} files for sphinx to pickup here

docs/conf.py

@@ -0,0 +1,39 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+project = 'xo alloc documentation'
+copyright = '2025, Roland Conybeare'
+author = 'Roland Conybeare'
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+#extensions = []
+extensions = [ "breathe",
+               "sphinx.ext.mathjax",     # inline math
+               "sphinx.ext.autodoc",     # generate info from docstrings
+               "sphinxcontrib.ditaa",    # diagrams-through-ascii-art
+               "sphinxcontrib.plantuml"  # text -> uml diagrams
+              ]
+
+# note: breathe requires doxygen xml output -> must have GENERATE_XML = YES in Doxyfile.in
+#       match project name in Doxyfile.in
+breathe_default_project = "xodoxxml"
+
+templates_path = ['_templates']
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+pygments_style = 'sphinx'
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+#html_theme = 'alabaster'
+html_theme = 'sphinx_rtd_theme'
+html_static_path = ['_static']
+html_favicon = '_static/img/favicon.ico'

docs/implementation.rst

@@ -0,0 +1,202 @@
+.. _implementation:
+
+.. toctree::
+   :maxdepth: 2
+
+Library
+=======
+
+Library dependency tower for *xo-alloc*:
+
+.. ditaa::
+
+    +------------------------------------------+
+    |                xo_alloc                  |
+    +------------------------------------------+
+    |              xo_indentlog                |
+    +------------------------------------------+
+
+Install instructions :doc:`here<install>`
+
+Components
+==========
+
+Abstraction tower for *xo-alloc* components:
+
+.. ditaa::
+    :--scale: 0.85
+
+    +----------------+-------------+
+    |     IAlloc     |   Object    |
+    +----------------+-------------+
+
+    +-------------+    +-------------+
+    |      GC     |    | Forwarding1 |
+    +-------------+    +-------------+
+    |  ListAlloc  |
+    +-------------+
+    |  ArenaAlloc |
+    +-------------+
+
+* *IAlloc*
+  Allocator interface.
+
+* *Object*
+    Root Object Interface for types participating in garbage collection
+
+* *GC*
+    Incremental compacting garbage collector.
+
+* *ListAlloc*
+    Auto-expanding allocator.  Contains a collection of ArenaAllocs
+
+* *ArenaAlloc*
+    Arena allocator (a.k.a bump allocator).
+
+* *Object*
+    Interface for types that participate in garbage collection
+
+* *Forwarding1*
+    Forwarding pointer. Supports the Object interface;
+    used internally by GC during evacuation.
+
+Key Points
+----------
+
+* Allocators can be reset, but do not support freeing of individual allocs.
+* GC works with types that implement auxiliary GC-support methods.
+  Such types must inherit Object.
+* A region may uses multiple arenas, but because of allocation activity
+  since the last GC.  If necessary, GC will allocate a new to-space with a
+  single arena that's large enough to accomodate all objects that might survive
+  from a from-space that has acquired multiple arenas.
+  Intent is to scale up to find application's working set size, then stabilize
+
+Components
+==========
+
+Allocators
+----------
+
+Inheritance
+^^^^^^^^^^^
+
+.. uml::
+    :caption: allocators
+    :scale: 99%
+    :align: center
+
+    class IAlloc {
+     + alloc()
+     + alloc_gc_copy()
+     + checkpoint()
+     + clear()
+    }
+
+    class ArenaAlloc {
+     + free_ptr()
+     - lo_ : byte*
+     - checkpoint_ : byte*
+     - limit_ : byte*
+    }
+
+    IAlloc <|-- ArenaAlloc
+
+    class ListAlloc {
+     + expand()
+     + free_ptr()
+     - start_z_
+     - hd_
+     - full_l_
+    }
+
+    IAlloc <|-- ListAlloc
+
+    class GC {
+     + add_gc_root()
+     + request_gc()
+     + gc_statistics()
+     - gc_root_v_[] : Object**
+     - nursery_[2] : ListAlloc*
+     - tenured_[2] : ListAlloc*
+    }
+
+    IAlloc <|-- GC
+
+
+Composition
+^^^^^^^^^^^
+
+.. uml::
+    :caption: allocator composition
+    :scale: 99%
+    :align: center
+
+    object gc<<GC>>
+    gc : nursery[from] = n0
+    gc : nursery[to] = n1
+    gc : tenured[from] = t0
+    gc : tenured[to] = t1
+
+    object n0<<ListAlloc>>
+
+    object n1<<ListAlloc>>
+
+    object t0<<ListAlloc>>
+
+    object t1<<ListAlloc>>
+
+    gc o-- n0
+    gc o-- n1
+    gc o-- t0
+    gc o-- t1
+
+
+Each ListAlloc composes like this:
+
+.. uml::
+    :caption: ListAlloc composition
+    :scale: 99%
+    :align: center
+
+    object x<<ListAlloc>>
+    x : hd_ = a0
+    x : full_l = {a1, a2}
+
+    object a0<<ArenaAlloc>>
+    a0 : lo_ = 0
+    a0 : free_ = 12345
+    a0 : hi_ = 1000000
+
+    object a1<<ArenaAlloc>>
+
+    object a2<<ArenaAlloc>>
+
+     x o-- a0
+     x o-- a1
+     x o-- a2
+
+Here *a1* and *a2* are full, while *a0* can still allocate memory.
+
+Objects
+
+.. uml::
+    :caption: objects
+    :scale: 99%
+    :align: center
+
+    class Object {
+     + _is_forwarded()
+     + _offset_destination()
+     + _forward_to()
+     + _destination()
+     + _shallow_size()
+     + _shallow_copy()
+     + _forward_children()
+    }
+
+    class Forwarding1 {
+     - dest_ : Object*
+    }
+
+    Object <|-- Forwarding1

docs/index.rst

@@ -0,0 +1,14 @@
+# xo-alloc documentation master file
+
+xo-alloc documentation
+======================
+
+xo-alloc provides arena allocators and a generation garbage collector
+
+.. toctree::
+   :maxdepth: 2
+   :caption: xo-alloc contents
+
+   install
+   introduction
+   implementation

docs/install.rst

@@ -0,0 +1,120 @@
+.. _install:
+
+.. toctree::
+   :maxdepth: 2
+
+Source
+======
+
+Source code lives on github `here`_
+
+.. _here: https://github.com/rconybea/xo-alloc
+
+To clone from git:
+
+.. code-block:: bash
+
+    git clone https://github.com/rconybea/xo-alloc
+
+Tested with gcc 13.3
+
+Install
+=======
+
+One-step Install
+----------------
+
+Install along with the reset of *XO* from `xo-umbrella2 source`_
+
+.. _xo-umbrella2 source: https://github.com/rconybea/xo-umbrella2
+
+Minimal Install
+---------------
+
+To build+install just required dependencies:
+``xo-alloc`` uses several supporting libraries from the *XO* project:
+
+- `xo-indentlog source`_ (structured logging)
+- `xo-cmake source`_ (shared cmake macros)
+
+.. _xo-indentlog source: https://github.com/rconybea/indentlog
+.. _xo-cmake source: https://github.com/rconybea/xo-cmake
+
+Building from source
+--------------------
+
+Install scripts for XO libraries depend on helper scripts installed from `xo-cmake`.
+
+Preamble:
+
+.. code-block:: bash
+
+    mkdir -p ~/proj/xo
+    cd ~/proj/xo
+
+    git clone https://github.com/rconybea/xo-cmake
+
+    PREFIX=/usr/local  # ..or desired installation prefix
+
+    # want PREFIX/bin in PATH to use xo-cmake helpers
+    PATH=$PREFIX/bin:$PATH
+
+Install `xo-cmake`:
+
+.. code-block:: bash
+
+    cmake -B xo-cmake/.build -S xo-cmake
+    cmake --install xo-cmake/.build
+
+Install remaining dependencie(s) in topological order:
+
+.. code-block:: bash
+
+    xo-build --clone --configure --build --install xo-indentlog
+    xo-build --clone --configure --build --install xo-alloc
+
+Directories under ``PREFIX`` will then contain:
+
+.. code-block::
+
+    PREFIX
+    +- bin
+    |  +- xo-build
+    |  +- xo-cmake-config
+    |  \- xo-cmake-lcov-harness
+    +- include
+    |  \- xo
+    |     +- alloc/
+    |     \- indentlog/
+    +- lib
+    |  +- cmake
+    |  |  +- xo_alloc/
+    |  |  \- indentlog/
+    |  +- lib*.so
+    +- share
+       +- cmake
+       |  \- xo_macros
+       |     +- code-coverage.cmake
+       |     +- xo-project-macros.cmake
+       |     \- xo_cxx.cmake
+       +- etc
+       |  \- xo
+       |     \- subsystem-list
+       \- xo-macros
+          +- Doxyfile.in
+          +- gen-ccov.in
+          \- xo-bootstrap-macros.cmake
+
+CMake Support
+-------------
+
+To use built-in cmake support, when using ``xo-alloc`` from another project:
+
+Make sure ``PREFIX/lib/cmake`` is searched by cmake (for example include it in ``CMAKE_PREFIX_PATH``)
+
+Add to your ``CMakeLists.txt``:
+
+.. code-block:: cmake
+
+    FindPackage(xo_alloc CONFIG REQUIRED)
+    target_link_libraries(mytarget INTERFACE xo_alloc)

docs/introduction.rst

@@ -0,0 +1,268 @@
+.. _introduction:
+
+.. toctree
+   :maxdepth: 2
+
+Introduction
+============
+
+The ``xo-alloc`` library provides a in incremental, generational collector for c++ code.
+
+Features:
+
+* *incremental* - can reasonably expect short pause times.
+* *generational* - focuses effort on collecting young objects,
+  on the basis that they're more likely to be garbage.
+* *compacting* - each garbage collection cycle evacuates survivors to contiguous memory,
+  so effect is to defragment.
+* *collects cycles* - collection algorithm naturally collects cyclic references
+
+Tradeoffs:
+
+* Application is responsible for spilling register values and protecting hardware stack,
+  since garbage collector cannot indepndently distinguish collectable object pointers from
+  non-pointer values.
+
+* GC will not spontaneously run without permission.  Instead will set a pending bit, with GC
+  occurring only when application releases it (e.g. when stack+registers are known to be empty of values
+  subject to GC).
+
+* GC implementation is single-threaded.  It cannot run in parallel with the mutator (i.e. application code)
+  In return this allows GC to be only lightly coupled with application.
+
+* GC divides each generation into separate from- and to- spaces.  A collection cycle copies surviving
+  objects out of from-space. Once complete, the entire from-space is treated as empty, and available to
+  become to-space on a future cycle.  This means that at any time only half of allocated memory is available
+  to the application; the rest is waiting to receive survivors from the next GC cycle.
+
+Design
+------
+
+Garbage Collector
+^^^^^^^^^^^^^^^^^
+
+The garbage collector supports two generations, labelled *nursery* and *tenured*.
+Nursery objects that survive two collection cycles are promoted to tenured space.
+Nursery and tenured objects are kept in separate memory areas, instead of being interspersed.
+
+Collection cycles come in two flavors:
+
+1. *incremental* collections - these collect only the nursery space.
+
+2. *full* collections - these collect both nursery and tenured spaces.
+   Full collection may incur noticeable GC pauses.
+
+Application Interaction
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Application code that interacts with GC has several responsibilities.
+
+1. application must explicitly invoke GC, when convenient. Since in general any GC-eligible object
+   may get moved by the collector: once a collection cycle completes,
+   it's up to the application to re-load pointers from memory addresses
+   (GC roots) that have been shared with the collector.
+
+2. application must identify a set of GC roots.  GC preserves everything reachable from any GC root
+
+3. The collector needs to know how to traverse GC-managed objects.
+   We teach it this by requiring that such objects inherit the ``xo::Object`` interface,
+   and implement auxiliary function detailed below.
+
+4. GC also needs to know when a mutation alters a pointer from one GC-managed object to another.
+   In particular, GC needs to track pointers from tenured space into nursery space,
+   and update them when an incremental collection moves nursery objects.
+   We do this by requiring application code use a GC-provided assignment primitive
+   on GC-eligible pointers.
+
+
+Example GC Use
+--------------
+
+.. code-block:: cpp
+   :linenos:
+
+    #include "xo/object/List.hpp"     // polymorphic List with GC support
+    #include "xo/object/String.hpp"   // string type with GC support
+    #include "xo/alloc/GC.hpp"
+
+    int main() {
+        using xo::gc::Config;
+        using xo::obj::String;
+        using xo::obj::List;
+        using xo::gp;
+
+        Config config = { .initial_nursery_z_ = 50*1000,
+                          .initial_tenured_z_ = 10*1000*1000,
+                          .debug_flag_ = false };
+
+        up<GC> gc = GC::make(config);
+
+        Object::mm = gc; // use GC for allocation of Object (+ derived classes)
+
+        gc->disable_gc(); // gc forbidden
+
+        // tiny example data structure
+        gp<String> s1 = String::copy("hello");
+        gp<String> s2 = String::copy(", ");
+        gp<String> s3 = String::copy("world!");
+        gp<List> list = List::cons(s1, List::cons(s2, List::cons(s3, List::nil)));
+
+        // tell GC what to preserve
+        gc->add_gc_root(reinterpret_cast<Object **>(list.ptr_address());
+
+        gc->enable_gc();  // triggers immediate gc
+
+        // s1, s2, s3 invalid.
+        // list at new address
+
+        std::cout << "list.size=" << list->size << std::endl;
+    }
+
+GC-Eligible Types
+-----------------
+
+Or, how to inherit ``xo::Object`` and provide GC support
+
+A type Foo that inherits ``xo::Object`` needs to provide overrides for Object methods ``_shallow_size()``,
+``_shallow_copy()`` and ``_forward_children()``:
+
+Typical Pattern
+^^^^^^^^^^^^^^^
+
+GC support methods look something like this:
+
+* class definition
+
+.. code-block:: cpp
+    :linenos:
+
+     #include "xo/alloc/Object.hpp"
+
+     namespace xo {
+         class Foo : public xo::Object {
+         public:
+             ...
+             virtual std::size_t _shallow_size() const override;
+             virtual Object * _shallow_copy() const override;
+             virtual std::size_t _forward_children() override;
+         };
+     }
+
+* use overloaded ``operator new``
+
+A GC-eligible class will allocate instances using the ``MMPtr`` overload.
+This allocates memory in GC-owned space
+
+.. code-block::cpp
+    :linenos:
+
+     gp<Foo> Foo::make(...) {
+         ...
+         return new MMPtr(mm) Foo(...);
+     }
+
+* ``_shallow_size()`` returns the amount of memory used by the subject:
+
+.. code-block:: cpp
+    :linenos:
+
+     std::size_t Foo::_shallow_size() const { return sizeof(Foo); }
+
+* ``_shallow_copy()`` is invoked during GC to create a copy of the subject
+
+  It should use the ``xo::Cpof`` argument to ``operator new``.
+
+.. code-block:: cpp
+    :linenos:
+
+     Object *
+     Foo::_shallow_copy() const;
+
+* ``_forward_children()`` is invoked during GC to vist child ``xo::Object`` pointers
+  to make sure they survive
+
+.. code-block:: cpp
+    :linenos:
+
+     std::size_t
+     Foo::_forward_children();
+
+Atomic Types Without Object Pointers
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Plain-old-data classes without embedded pointers
+
+.. code-block:: cpp
+    :linenos:
+
+     Object *
+     Foo::_shallow_copy() const {
+         return new (Cpof(this)) Foo(*this);
+     }
+
+.. code-block:: cpp
+    :linenos:
+
+     std::size_t
+     Foo::_forward_children() { return Foo::_shallow_size(); }
+
+For example see ``xo::obj::String`` in ``xo-object``
+
+Non-GC Objects
+^^^^^^^^^^^^^^
+
+A class *Foo* that inherits ``xo::Object`` can opt-out of garbage collection by
+omitting the ``MMptr(mm)`` overload.
+
+In that case `Foo::_shallow_size()`, `Foo::_shallow_copy()` and `Foo::_forward_children()`
+will not be called:
+
+.. code-block:: cpp
+    :linenos:
+
+     std::size_t Foo::_shallow_size() const { return sizeof(Foo); }
+     Object *    Foo::_shallow_copy() const { assert(false); return nullptr; }
+     std::size_t Foo::_forward_children()   { assert(false); return 0; }
+
+For example see ``xo::obj::Boolean`` in ``xo-object``
+
+Structs Containing Object Pointers
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A class with object pointers needs to tell GC how to traverse them
+
+.. code-block:: cpp
+    :linenos:
+
+     #include "xo/alloc/Object.hpp"
+
+     namespace xo {
+         class Foo : public xo::Object {
+         public:
+             ...
+             virtual std::size_t _shallow_size() const override;
+             virtual Object * _shallow_copy() const override;
+             virtual std::size_t _forward_children() override;
+
+         private:
+             gp<Object> bar_;
+             gp<Object> quux_;
+         };
+     }
+
+* ``_forward_children()`` is invoked during GC to fixup child pointers
+  that refer to forwarding objects:
+
+.. code-block:: cpp
+    :linenos:
+
+     std::size_t
+     Foo::_forward_children()
+     {
+         Object::_forward_inplace(bar_);
+         Object::_forward_inplace(quux_);
+
+         return Foo::_shallow_size();
+     }
+
+For example see ``xo::obj::List`` in ``xo-object``