roland/xo-umbrella2
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

+ docs in xo-alloc2/ + misc improvements

Browse source
This commit is contained in:
Roland Conybeare 2025-12-11 22:18:09 -05:00
commit f52184f84c
29 changed files with 868 additions and 178 deletions
Download patch file Download diff file Expand all files Collapse all files

2
CMakeLists.txt
View file

@ -128,6 +128,6 @@ add_subdirectory(xo-imgui)
# ----------------------------------------------------------------
# documentation. must follow add_subdirectory() for satellite projects
xo_umbrella_doxygen_deps(xo_facet xo_alloc indentlog xo_flatstring xo_ratio xo_unit xo_tokenizer xo_reader xo_interpreter xo_jit)
xo_umbrella_doxygen_deps(xo_facet xo_alloc xo_alloc2 indentlog xo_flatstring xo_ratio xo_unit xo_tokenizer xo_reader xo_interpreter xo_jit)
xo_umbrella_doxygen_config()
xo_umbrella_sphinx_config(index.rst docs/install.rst docs/glossary.rst)

29
LICENSE Normal file
View file

@ -0,0 +1,29 @@
Copyright (c) 2024 Roland Conybeare <git3ub@nym.hush.com>, All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Please also refer to the file .github/CONTRIBUTING.md, which clarifies licensing of
external contributions to this project including patches, pull requests, etc.

34
README.md
View file

@ -187,6 +187,40 @@ $ cmake -B .build -S . -DXO_ENABLE_VULKAN=1 -DXO_ENABLE_EXAMPLES=1 -DCMAKE_INSTA
Currently not supporting a nix sandbox build for xo-imgui
## Directory Layout
(not in alphabetical order)
```
xo-umbrella2/
|
+- CMakeLists.txt top-level cmake config
+- cmake
| \- xo-bootstrap-macros.cmake configure xo cmake support for build
+- compile_commands.json symlink to path/to/build/compile_commands.json for LSP
|
+- conf.py sphinx config for project documentation
+- index.rst root of xo-umbrella2 doc tree
+- Doxyfile.in doxygen config template; cmake will prepare Doxyfile in build dir
+- _static/ static inputs to sphinx
|
+- etc
| +- hostegl/ sample video driver symlinks for WSL2
| \- hostubuntu/ sample video driver symlinks for ubuntu
|
+- default.nix top-level nix build (works w/ stock nixpkgs 25.05)
+- pkgs/ per-satellite nix builds. see xo-umbrella2/default.nix
| +- xo-callback.nix nix build for xo-umbrella2/xo-callback
| ..etc..
| \- xo-webutil.nix
|
+- xo-alloc/ xo-alloc subproject. independent git repo, using git subtree
+- xo-alloc2/ xo-alloc2 subproject.
..etc..
\- xo-webutil/
```
## To view docs from WSL
1. find wsl IP address

4
conf.py
View file

@ -6,7 +6,7 @@
# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
project = 'xo documentation'
project = 'xo umbrella2 documentation'
copyright = '2025, Roland Conybeare'
author = 'Roland Conybeare'
@ -17,7 +17,7 @@ author = 'Roland Conybeare'
extensions = [ "breathe",
"sphinx.ext.mathjax", # inline math
"sphinx.ext.autodoc", # generate info from docstrings
# "sphinxcontrib.ditaa", # diagrams-through-ascii-art
"sphinxcontrib.ditaa", # diagrams-through-ascii-art
"sphinxcontrib.plantuml", # text -> uml diagrams
]

1
index.rst
View file

@ -13,6 +13,7 @@ Some features: kalman filters, stochastic processes, complex event processing, s
docs/install
xo-facet/docs/index
xo-alloc2/docs/index
xo-alloc/docs/index
xo-indentlog/docs/index
xo-flatstring/docs/index

2
xo-alloc2/CMakeLists.txt
View file

@ -27,6 +27,6 @@ add_subdirectory(utest)
# docs targets depend on other library/utest/exec targets above,
# --> must come after them.
#
#add_subdirectory(docs)
add_subdirectory(docs)
# end CMakeLists.txt

39
xo-alloc2/docs/AAllocator-reference.rst Normal file
View file

@ -0,0 +1,39 @@
.. _AAllocator-reference:
AAllocator Reference
====================
Abstract interface facet for arena allocator.
Provides simple arena allocation.
Context
-------
.. ditaa::
:--scale: 0.99
+--------------------------------+
| IAllocator_DArena |
+--------------------------------+
| IAllocator_Xfer |
+--------------------------------+
| IAllocator_ImplType |
+--------------+-----------------+
|cBLU | DArena |
| AAllocator +-----------------+
| | ArenaConfig |
+--------------+-----------------+
.. code-block:: cpp
#incldue <xo/alloc2/AAllocator.hpp>
Class
-----
.. doxygenclass:: xo::mm::AAllocator
Methods
-------
.. doxygengroup:: mm-allocator-methods

35
xo-alloc2/docs/ArenaConfig-reference.rst Normal file
View file

@ -0,0 +1,35 @@
.. _ArenaConfig-reference:
ArenaConfig Reference
=====================
Configuration for an arena allocator
Context
-------
.. ditaa::
:--scale: 0.99
+--------------------------------+
| IAllocator_DArena |
+--------------------------------+
| IAllocator_Xfer |
+--------------------------------+
| IAllocator_ImplType |
+--------------+-----------------+
| | DArena |
| AAllocator +-----------------+
| | ArenaConfig cBLU|
+--------------+-----------------+
.. code-block:: cpp
#include <xo/alloc2/DArena.hpp>
Class
-----
.. doxygenclass:: xo::mm::ArenaConfig
.. doxygenclass:: mm-arenaconfig-instance-vars

17
xo-alloc2/docs/CMakeLists.txt Normal file
View file

@ -0,0 +1,17 @@
# xo-alloc2/docs/CMakeLists.txt
xo_doxygen_collect_deps()
xo_docdir_doxygen_config()
xo_docdir_sphinx_config(
index.rst
glossary.rst
implementation.rst
ArenaConfig-reference.rst
DArena-reference.rst
#install.rst
#introduction.rst
#implementation.rst
)
# see xo-reader/doc or xo-unit/doc for working examples
# example.rst install.rst implementation.rst

64
xo-alloc2/docs/DArena-reference.rst Normal file
View file

@ -0,0 +1,64 @@
.. _DArena-reference:
DArena Reference
================
Native representation for arena allocator
Context
-------
.. ditaa::
:--scale: 0.99
+--------------------------------+
| IAllocator_DArena |
+--------------------------------+
| IAllocator_Xfer |
+--------------------------------+
| IAllocator_ImplType |
+--------------+-----------------+
| | DArena cBLU|
| AAllocator +-----------------+
| | ArenaConfig |
+--------------+-----------------+
.. code-block:: cpp
#include <xo/alloc2/DArena.hpp>
Arena memory layout::
<----------------------------size-------------------------->
<------------committed-----------><-------uncommitted------>
<--allocated-->
XXXXXXXXXXXXXXX___________________..........................
^ ^ ^ ^
lo free limit hi
[X] allocated: in use
[_] committed: physical memory obtained
[.] uncommitted: mapped in virtual memory, not backed by memory
Class
-----
.. doxygenclass:: xo::mm::DArena
Member Variables
----------------
.. doxygengroup:: mm-arena-instance-vars
Type Traits
-----------
.. doxygengroup:: mm-arena-traits
Constructors
------------
.. doxygengroup:: mm-arena-ctors

70
xo-alloc2/docs/README Normal file
View file

@ -0,0 +1,70 @@
build
+-----------------------------------------------+
| cmake |
| CMakeLists.txt |
| $PREFIX/share/cmake/xo_macros/xo_cxx.cmake |
+-----------------------------------------------+
|
| +----------------------+
+------------------------------------------------->| .build/docs/Doxyfile |
| +----------------------+
| |
| /------------/
| |
| v
| +---------------------------------------+ +-----------------+
+---->| doxygen |--->| .build/docs/dox |
| | $PREFIX/share/xo-macros/Doxyfile.in | | +- html/ |
| +---------------------------------------+ | +- xml/ |
| +-----------------+
| |
| /------------/
| |
| v
| +---------------------------------------+ +--------------------+
\---->| sphinx |--->| .build/docs/sphinx |
| +- conf.py | | +- html/ |
| +- _static/ | +--------------------+
| +- *.rst |
+---------------------------------------+
files
README this file
CMakeLists.txt build entry point
conf.py sphinx config
_static static files for sphinx
map
index.rst
+- install.rst
+- examples.rst
+- unit-quantities.rst
+- classes.rst
+- glossary.rst
...
examples
.. doxygenclass:: ${c++ class name}
:project:
:path:
:members:
:protected-members:
:private-members:
:undoc-members:
:member-groups:
:members-only:
:outline:
:no-link:
:allow-dot-graphs:
.. doxygendefine:: ${c preprocessor define}
.. doxygenconcept:: ${c++ concept definition}
.. doxygenenum:: ${c++ enum definition}
.. doxygenfunction:: ${c++ function name}

1
xo-alloc2/docs/_static/README vendored Normal file
View file

@ -0,0 +1 @@
add any static {.html, .js, ..} files for sphinx to pickup here

0
xo-facet/docs/_static/favicon.ico → xo-alloc2/docs/_static/img/favicon.ico vendored
View file

Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 303 KiB

After

Width:  |  Height:  |  Size: 303 KiB

Before After
Before After

39
xo-alloc2/docs/conf.py Normal file
View file

@ -0,0 +1,39 @@
# Configuration file for the Sphinx documentation builder.
#
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
project = 'xo alloc2 documentation'
copyright = '2025, Roland Conybeare'
author = 'Roland Conybeare'
# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
#extensions = []
extensions = [ "breathe",
"sphinx.ext.mathjax", # inline math
"sphinx.ext.autodoc", # generate info from docstrings
"sphinxcontrib.ditaa", # diagrams-through-ascii-art
"sphinxcontrib.plantuml" # text -> uml diagrams
]
# note: breathe requires doxygen xml output -> must have GENERATE_XML = YES in Doxyfile.in
# match project name in Doxyfile.in
breathe_default_project = "xodoxxml"
templates_path = ['_templates']
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
pygments_style = 'sphinx'
# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
#html_theme = 'alabaster'
html_theme = 'sphinx_rtd_theme'
html_static_path = ['_static']
html_favicon = '_static/img/favicon.ico'

24
xo-alloc2/docs/glossary.rst Normal file
View file

@ -0,0 +1,24 @@
.. _glossary:
Glossary
--------
.. glossary::
FOMO
| facet object model
page
| a (4k) page of virtual memory.
| O/S manages virtual memory in chunks of this size.
hugepage
| large (2MB) VM page; use to reduce page fault expense and TLB pressure.
THP
| transparent huge pages
TLB
| translation lookaside buffer
VM
| virtual memory

51
xo-alloc2/docs/implementation.rst Normal file
View file

@ -0,0 +1,51 @@
.. _implementation:
Components
==========
Library dependency tower for *xo-alloc2*
.. ditaa::
+-----------------+
| xo_alloc2 |
+-----------------+
| xo_facet |
+-----------------+
| xo_cmake |
+-----------------+
Abstraction tower for *xo-alloc2* components
.. ditaa::
:--scale: 0.99
+--------------------------------+
| IAllocator_DArena |
+--------------------------------+
| IAllocator_Xfer |
+--------------------------------+
| IAllocator_ImplType |
+--------------+-----------------+
| | DArena |
| AAllocator +-----------------+
| | ArenaConfig |
+--------------+-----------------+
.. list-table:: Descriptions
:header-rows: 1
:widths: 20 90
* - Component
- Description
* - ``AAllocator``
- allocator facet (abstract interface)
* - ``DArena``
- arena representation
* - ``IAllocator_ImplType<D>``
- lookup implementation for allocator A
with representation D.
* - ``IAllocator_Xfer<D>``
- transfer interface. downcast to native state.
* - ``IAllocator_DArena``
- allocator implementation for ``DArena``

28
xo-alloc2/docs/index.rst Normal file
View file

@ -0,0 +1,28 @@
# xo-alloc2 documentation master file
xo-alloc2 documentation
=======================
xo-alloc2 is intended to provide fast vm-aware arena allocation.
Next-generation version of xo-alloc.
At present (Dec 2025) xo-alloc is fully functional,
while xo-alloc2 is aspirational.
Features:
* allocates uncommitted virtual memory, and commits on demand.
* ses THP (Transparent Huge Pages) when available.
Implemented using FOMO (faceted rust-like object model) from xo-facet
.. toctree::
:maxdepth: 2
:caption: xo-alloc2 contents
implementation
AAllocator-reference
ArenaConfig-reference
DArena-reference
glossary
genindex
search

60
xo-alloc2/include/xo/alloc2/AAllocator.hpp
View file

@ -14,9 +14,12 @@ namespace xo {
using Copaque = const void *;
using Opaque = void *;
/** Abstract facet for allocation
/** @class AAllocator
* @brief Abstract facet for allocation
*
**/
struct AAllocator {
/*
* <----------------------------size-------------------------->
* <------------committed-----------><-------uncommitted------>
* <--allocated-->
@ -26,40 +29,71 @@ namespace xo {
* allocated: in use
* committed: physical memory obtained
* uncommitted: mapped in virtual memory, not backed by memory
**/
struct AAllocator {
/** RTTI: unique id# for actual runtime data repr **/
*/
/** @defgroup mm-allocator-methods Allocator methods **/
///@{
/** RTTI: unique id# for actual runtime data representation **/
virtual int32_t _typeseq() = 0;
/** optional name for this allocator.
/** optional name for allocator @p d
* Labeling, for diagnostics.
**/
virtual const std::string & name(Copaque d) = 0;
/** allocator size in bytes (up to reserved limit)
* includes allocated and uncomitted memory
* for allocator @p d.
* Includes allocated and uncomitted memory
**/
virtual std::size_t size(Copaque d) = 0;
/** committed size (physical addresses obtained)
* for allocator @p d.
**/
virtual std::size_t committed(Copaque d) = 0;
/** true iff pointer @p in range of this allocator
/** true iff allocator @p d is responsible for memory at address @p p.
**/
virtual bool contains(Copaque d, const void * p) = 0;
/** allocate @p z bytes of memory. **/
/** allocate @p z bytes of memory from allocator @p d. **/
virtual std::byte * alloc(Opaque d, std::size_t z) = 0;
/** reset allocator to empty state **/
/** reset allocator @p d to empty state **/
virtual void clear(Opaque d) = 0;
/** **/
/** destruct allocator @p d **/
virtual void destruct_data(Opaque d) = 0;
};
///@}
}; /*AAllocator*/
template <typename DRepr>
struct IAllocator_Impl;
template <typename DRepr>
using IAllocator_ImplType = IAllocator_Impl<DRepr>::ImplType;
struct IAllocator_Any : public AAllocator {
using Impl = Allocator_ImplType<xo::facet::DVariantPlaceholder>;
// from AAllocator
int32_t _typeseq() override { return s_typeseq; }
const std::string & name(Copaque d) override { assert(false); static std::string x; return x; }
std::size_t size(Copaque d) override { assert(false); return 0ul; }
std::size_t committed(Copaque d) override { assert(false); reutrn 0ul; }
bool contains(Copaque d, const void * p) override { assert(false); return false; }
std::byte * alloc(Opaque d, std::size_t z) override { assert(false); return nullptr; }
void clear(Opaque d) override { assert(false); }
void destruct_data(Opaque d) override { assert(false); }
static int32_t s_typeseq;
static bool _valid;
}
/** @class IAllocator_Xfer
**/
template <typename DRepr>
struct IAllocator_Xfer : public AAllocator {
// parallel interface to AAllocator, with specific data type
using Impl = IAllocator_Impl<DRepr>;
using Impl = IAllocator_ImplType<DRepr>;
static const DRepr & _dcast(Copaque d) { return *(const DRepr *)d; }

72
xo-alloc2/include/xo/alloc2/DArena.hpp
View file

@ -10,12 +10,17 @@
namespace xo {
namespace mm {
/** ArenaConfig
/** @class ArenaConfig
*
* @brief configuration for a @ref DArena instance
**/
struct ArenaConfig {
/** @defgroup mm-arenaconfig-instance-vars ArenaConfig members **/
///@{
/** optional name, for diagnostics **/
std::string name_;
/** arena size -- hard max = reserved virtual memory **/
/** desired arena size -- hard max = reserved virtual memory **/
std::size_t size_;
/** hugepage size -- using huge pages relieves some TLB pressure
* (provided you use their full extent :)
@ -23,31 +28,70 @@ namespace xo {
std::size_t hugepage_z_ = 2 * 1024 * 1024;
/** true to enable debug logging **/
bool debug_flag_ = false;
///@}
};
/** Arena allocator state
/** @class DArena
*
* @brief represent arena allocator state
*
* Provides minimal RAII functionality around memory mapping.
* For allocation see @ref IAllocator_DArena
**/
struct DArena {
/*
* <----------------------------size-------------------------->
* <------------committed-----------><-------uncommitted------>
* <--allocated-->
*
* XXXXXXXXXXXXXXX___________________..........................
*
* allocated: in use
* committed: physical memory obtained
* uncommitted: mapped in virtual memory, not backed by memory
**/
struct DArena {
/** [lo, hi) already-mapped address range **/
DArena(const ArenaConfig & cfg,
std::byte * lo,
std::byte * hi);
* [X] allocated: in use
* [_] committed: physical memory obtained
* [.] uncommitted: mapped in virtual memory, not backed by memory
*/
/** @defgroup mm-arena-traits arena type traits **/
///@{
/** @brief an amount of memory **/
using size_type = std::size_t;
/** @brief a contiguous memory range **/
using range_type = std::pair<std::byte*,std::byte*>;
///@}
/** @defgroup mm-arena-ctors arena constructors and destructors **/
///@{
/** create arena per configuration @p cfg. **/
static DArena map(const ArenaConfig & cfg);
/** ctor from already-mapped (but not committed) address range **/
DArena(const ArenaConfig & cfg, size_type page_z, std::byte * lo, std::byte * hi);
/** DArena is not copyable **/
DArena(const DArena & other) = delete;
/** move ctor **/
DArena(DArena && other);
/** dtor releases mapped memory **/
~DArena();
///@}
/** obtain uncommitted contiguous memory range comprising
* a whole multiple of @p hugepage_z bytes, of at least size @p req_z,
* aligned on a @p hugepage_z boundary
**/
static range_type map_aligned_range(size_type req_z, size_type hugepage_z);
/** @defgroup mm-arena-instance-vars **/
///@{
/** arena configuration **/
ArenaConfig config_;
/** size of a VM page (via getpagesize()). Likely 4k **/
/** size of a VM page (obtained automatically via getpagesize()). Likely 4k **/
std::size_t page_z_ = 0;
/** arena owns memory in range [@ref lo_, @ref hi_)
@ -74,6 +118,8 @@ namespace xo {
* Memory in range [@ref limit_, @ref hi_) is uncommitted
**/
std::byte * hi_ = nullptr;
///@}
};
} /*namespace mm*/

12
xo-alloc2/include/xo/alloc2/IAllocator_DArena.hpp
View file

@ -9,12 +9,8 @@
namespace xo {
namespace mm {
template <>
struct IAllocator_Impl<DArena> {
static const std::string & name(const DArena & s) {
return s.name_;
}
struct IAllocator_DArena {
static const std::string & name(const DArena & s);
static std::size_t size(const DArena & s);
static std::size_t committed(const DArena & s);
static bool contains(const DArena & s, const void * p);
@ -23,6 +19,10 @@ namespace xo {
static void destruct_data(DArena & s);
};
template <>
struct IAllocator_Impl<DArena> {
using ImplType = IAllocator_DArena;
};
} /*namespace mm*/
} /*namespace xo*/

38
xo-alloc2/include/xo/alloc2/RAllocator.hpp Normal file
View file

@ -0,0 +1,38 @@
/** @file RAllocator.hpp
*
* @author Roland Conybeare, Dec 2025
**/
#pragma once
#include "xo/facet/RRouter.hpp"
namespace xo {
namespace mm {
/** @class RAllocator **/
template <typename Object>
struct RAllocator : public Object {
using ObjectType = Object;
RAllocator() {}
RAllocator(Object::DataPtr data) : Object{std::move(data)} {}
int32_t _typeseq() const { return Object::iface()->_typeseq(); }
static bool _valid;
};
template <typename Object>
bool
RAllocator<Object>::_valid = facet::valid_object_router<Object>();
}
namespace facet {
template <typename Object>
struct RoutingFor<xo::mm::AAllocator, Object> {
using RoutingType = xo::mm::RAllocator<Object>;
};
}
}
/* end RAllocator.hpp */

19
xo-alloc2/src/alloc2/AAllocator.cpp Normal file
View file

@ -0,0 +1,19 @@
/** @file AAllocator.cpp **/
#include "xo/alloc2/AAllocator.hpp"
namespace xo {
using xo::facet::DVariantPlaceholder;
using xo::facet::typeseq;
using xo::facet::valid_facet_implementation;
namespace mm {
int32_t
IAllocator_Any::s_typeseq = typeseq::id<DVariantPlaceholder>;
bool
IAllocator_Any::_valid = valid_facet_implementation<AAllocator, IAllocator_Any>;
}
}
/* end AAlocator.cpp */

1
xo-alloc2/src/alloc2/CMakeLists.txt
View file

@ -2,6 +2,7 @@
set(SELF_LIB xo_alloc2)
set(SELF_SRCS
AAllocator.cpp
DArena.cpp
IAllocator_DArena.cpp
)

194
xo-alloc2/src/alloc2/DArena.cpp
View file

@ -9,57 +9,77 @@
#include <cassert>
#include <sys/mman.h> // for ::munmap()
#include <unistd.h> // for ::getpagesize()
#include <string.h> // for ::memset()
namespace xo {
using std::byte;
using std::size_t;
namespace mm {
DArena::DArena(const ArenaConfig & cfg,
std::byte * lo,
std::byte * hi
)
/** Map a contiguous uncommitted memory range comprising
* a whole multiple of @p hugepage_z, with at least
* @p req_z bytes.
*
* Memory will also be aligned on @p hugepage_z boundary
* (2MB in practice)
*
* - @p req_z is rounded up to a multiple of @p hugepage_z
* - Resulting uncommitted address range not backed by
* physical memory.
* - since hugpage-aligned, can find base of mapped range
* by masking off the bottom log2(align_z) bits.
* May rely on this for GC metadata
* - opt-in to transparent huge pages (THP).
* Reduces page-fault time by a lot, in return for
* lower VM granularity
* - rejecting inferior MAP_HUGETLB|MAP_HUGE_2MB flags on ::mmap here:
* - requires previously-reserved memory in /proc/sys/vm/nr_hugepages
* - reserved pages permenently resident in RAM, never swapped
* - memory cost incurred even if no application is using said pages
*
* TODO: for OSX -> need something else here.
* MAP_ALIGNED_SUPER with mmap() and/or
* use mach_vm_allocate()
*
* @return pair giving mapped address range [lo, hi)
**/
auto
DArena::map_aligned_range(size_t req_z, size_t hugepage_z) -> range_type
{
//scope log(XO_DEBUG(debug_flag), xtag("name", name));
// 1. round up to multiple of hugepage_z
size_t target_z = padding::with_padding(req_z, hugepage_z); // 4.
this->page_z_ = getpagesize();
// 1. need k pagetable entries where k is lub {k | k * .page_z >= z}
// 2. base will be aligned with .page_z but likely not with .hugepage_z
// 3. bad to have misalignment, because misaligned {prefix, suffix} of [base, base+z)
// will use 4k pages instead of 2mb pages
// 2. mmap() will give us page-aligned memory,
// but not hugepage-aligned.
//
// strategy:
// 4. round up z to multiple of hugepage_z_
// 5. over-request so reserved range contains an aligned subrange of size z
// 6. unmap misaligned prefix
// 7. unmap misaligned suffix.
// 8. enable huge pages for now-aligned remainder of reserved range
// Over-request by hugepage_z to ensure
// hugepage-aligned subrange of size target_z
//
// Z. note: rejecting inferior MAP_HUGETLB|MAP_HUGE_2MB flags on ::mmap here:
// Za. requires previously-reserved memory in /proc/sys/vm/nr_hugepages
// Zb. reserved pages permenently resident in RAM, never swapped
// Zc. memory cost incurred even if no application is using said pages
std::size_t z = cfg.size_;
z = padding::with_padding(z, config_.hugepage_z_); // 4.
// 5.
byte * base = reinterpret_cast<byte *>(
::mmap(nullptr,
z + config_.hugepage_z_,
byte * base = (byte *)(::mmap(nullptr,
target_z + hugepage_z,
PROT_NONE,
MAP_PRIVATE | MAP_ANONYMOUS,
-1, 0));
// on mmap success: upper limit of mapped address range
byte * hi = base + (target_z + hugepage_z);
// lowest hugepage-aligned address in [base, hi)
byte * aligned_base = (byte *)(padding::with_padding((size_t)base, hugepage_z));
// end of hugeppage-aligned range starting at aligned_base
byte * aligned_hi = aligned_base + target_z;
#ifdef NOT_YET
log && log("acquired memory [lo,hi) using mmap",
xtag("lo", base),
xtag("z", z),
xtag("hi", reinterpret_cast<byte *>(base) + z));
xtag("req_z", req_z),
xtag("target_z", target_z),
xtag("hi", (byte *)(base) + z));
#endif
// 3. assess mmap success
{
if (base == MAP_FAILED) {
assert(false);
#ifdef NOPE
@ -68,48 +88,56 @@ namespace xo {
#endif
}
byte * aligned_base = reinterpret_cast<byte *>
(padding::with_padding(reinterpret_cast<size_t>(base),
config_.hugepage_z_));
assert(reinterpret_cast<size_t>(aligned_base) % config_.hugepage_z_ == 0);
assert((size_t)aligned_base % hugepage_z == 0);
assert(aligned_base >= base);
assert(aligned_base < base + config_.hugepage_z_);
if (base < aligned_base) {
size_t prefix = aligned_base - base;
::munmap(base, prefix); // 6.
assert(aligned_base < base + hugepage_z);
}
byte * aligned_hi = aligned_base + z;
byte * hi = base + z + config_.hugepage_z_;
// 4. release unaligned prefix
if (base < aligned_base) {
size_t ua_prefix = aligned_base - base;
::munmap(base, ua_prefix);
}
// 5. release unaligned suffix
if (aligned_hi < hi) {
size_t suffix = hi - aligned_hi;
::munmap(aligned_hi, suffix); // 7.
::munmap(aligned_hi, suffix);
}
#ifdef __linux__
/** opt-in to huge pages, provided they're available.
* otherwise fallback gracefully
/** linux:
* opt-in to transparent huge pages (THP)
* provided OS configured to support them.
* otherwise fallback gracefully.
*
* Huge pages -> use fewer TLB entries + faster
* shorter path through page table.
*
* When we commit (i.e. obtain physical memory on page fault),
* typically expect to pay ~1us per superpage.
* Much better than ~500us to commit 512 4k VM pages.
*
* But wasted if we don't use the memory.
**/
::madvise(aligned_base, z, MADV_HUGEPAGE); // 8.
::madvise(aligned_base, target_z, MADV_HUGEPAGE); // 8.
#endif
// TODO: for OSX -> need something else here.
// MAP_ALIGNED_SUPER with mmap() and/or
// use mach_vm_allocate()
//
this->lo_ = aligned_base;
this->committed_z_ = 0;
//this->checkpoint_ = lo_;
this->free_ = lo_;
this->limit_ = lo_;
this->hi_ = lo_ + z;
return std::make_pair(aligned_base, aligned_hi);
}
DArena
DArena::map(const ArenaConfig & cfg)
{
//scope log(XO_DEBUG(debug_flag), xtag("name", name));
auto [lo, hi] = map_aligned_range(cfg.size_, cfg.hugepage_z_);
if (!lo) {
// control here implies mmap() failed silently
if (!lo_) {
assert(false);
#ifdef NOPE
throw std::runtime_error(tostr("ArenaAlloc: allocation failed",
@ -117,11 +145,47 @@ namespace xo {
#endif
}
size_t page_z = getpagesize();
#ifdef NOPE
log && log(xtag("lo", (void*)lo_),
xtag("page_z", page_z_),
xtag("hugepage_z", hugepage_z_));
#endif
return DArena(cfg, page_z, lo, hi);
} /*map*/
DArena::DArena(const ArenaConfig & cfg,
size_type page_z,
byte * lo,
byte * hi) : config_{cfg},
page_z_{page_z},
lo_{lo},
committed_z_{0},
free_{lo},
limit_{lo},
hi_{hi}
{
//retval.checkpoint_ = lo_;
}
DArena::DArena(DArena && other) {
config_ = other.config_;
page_z_ = other.page_z_;
lo_ = other.lo_;
committed_z_ = other.committed_z_;
free_ = other.free_;
limit_ = other.limit_;
hi_ = other.hi_;
other.config_ = ArenaConfig();
other.lo_ = nullptr;
other.committed_z_ = 0;
other.free_ = nullptr;
other.limit_ = nullptr;
other.hi_ = nullptr;
}
DArena::~DArena()
@ -136,12 +200,12 @@ namespace xo {
}
// hygiene
lo_ = nullptr;
committed_z_ = 0;
this->lo_ = nullptr;
this->committed_z_ = 0;
// checkpoint_ = nullptr;
free_ = nullptr;
limit_ = nullptr;
hi_ = nullptr;
this->free_ = nullptr;
this->limit_ = nullptr;
this->hi_ = nullptr;
}
}
} /*namespace xo*/

17
xo-alloc2/src/alloc2/IAllocator_DArena.cpp
View file

@ -9,25 +9,30 @@
namespace xo {
namespace mm {
const std::string &
IAllocator_DArena::name(const DArena & s) {
return s.config_.name_;
}
std::size_t
IAllocator_Impl<DArena>::size(const DArena & s) {
IAllocator_DArena::size(const DArena & s) {
return s.limit_ - s.lo_;
}
std::size_t
IAllocator_Impl<DArena>::committed(const DArena & s) {
IAllocator_DArena::committed(const DArena & s) {
return s.committed_z_;
}
bool
IAllocator_Impl<DArena>::contains(const DArena & s,
IAllocator_DArena::contains(const DArena & s,
const void * p)
{
return (s.lo_ <= p) && (p < s.hi_);
}
std::byte *
IAllocator_Impl<DArena>::alloc(const DArena & s,
IAllocator_DArena::alloc(const DArena & s,
std::size_t z)
{
(void)s;
@ -39,14 +44,14 @@ namespace xo {
}
void
IAllocator_Impl<DArena>::clear(DArena & s)
IAllocator_DArena::clear(DArena & s)
{
s.free_ = s.lo_;
//s.checkpoint_ = s.lo_;
}
void
IAllocator_Impl<DArena>::destruct_data(DArena & s)
IAllocator_DArena::destruct_data(DArena & s)
{
s.~DArena();
}

1
xo-alloc2/utest/CMakeLists.txt
View file

@ -9,6 +9,7 @@ set(UTEST_SRCS
if (ENABLE_TESTING)
xo_add_utest_executable(${UTEST_EXE} ${UTEST_SRCS})
xo_self_dependency(${UTEST_EXE} xo_alloc2)
xo_headeronly_dependency(${UTEST_EXE} xo_facet)
xo_external_target_dependency(${UTEST_EXE} Catch2 Catch2::Catch2)
endif()

66
xo-alloc2/utest/arena.test.cpp
View file

@ -6,12 +6,19 @@
#include "xo/alloc2/AAllocator.hpp"
#include "xo/alloc2/DArena.hpp"
#include "xo/alloc2/IAllocator_DArena.hpp"
#include "xo/alloc2/RAllocator.hpp"
#include "xo/alloc2/padding.hpp"
#include "xo/facet/obj.hpp"
#include <catch2/catch.hpp>
namespace xo {
using xo::mm::AAllocator;
using xo::mm::IAllocator_Xfer;
using xo::mm::DArena;
using xo::mm::ArenaConfig;
using xo::facet::obj;
using std::byte;
using std::size_t;
namespace ut {
TEST_CASE("IAllocator_Xfer_DArena", "[alloc2]")
@ -20,6 +27,65 @@ namespace xo {
REQUIRE(IAllocator_Xfer<DArena>::_valid);
}
TEST_CASE("DArena", "[alloc2][DArena]")
{
ArenaConfig cfg { .name_ = "testarena",
.size_ = 1 };
DArena arena = DArena::map(cfg);
REQUIRE(arena.config_.name_ == cfg.name_);
REQUIRE(arena.lo_ != nullptr);
REQUIRE(arena.free_ == arena.lo_);
REQUIRE(arena.limit_ == arena.lo_);
REQUIRE(arena.hi_ != nullptr);
REQUIRE(arena.hi_ > arena.lo_);
REQUIRE(((size_t)arena.hi_ - (size_t)arena.lo_) % cfg.hugepage_z_ == 0);
REQUIRE(arena.lo_ + cfg.size_ <= arena.hi_);
/* verify arena.lo_ is aligned on a hugepage boundary */
REQUIRE(((size_t)(arena.lo_) & (cfg.hugepage_z_ - 1)) == 0);
/* verify arena.hi_ is aligned on a hugepage boundary */
REQUIRE(((size_t)(arena.hi_) & (cfg.hugepage_z_ - 1)) == 0);
byte * lo = arena.lo_;
byte * free = arena.free_;
byte * limit = arena.limit_;
byte * hi = arena.hi_;
size_t committed_z = arena.committed_z_;
DArena arena2 = std::move(arena);
REQUIRE(arena.lo_ == nullptr);
REQUIRE(arena.free_ == nullptr);
REQUIRE(arena.limit_ == nullptr);
REQUIRE(arena.hi_ == nullptr);
REQUIRE(arena.committed_z_ == 0);
REQUIRE(arena.lo_ == nullptr);
REQUIRE(arena2.lo_ == lo);
REQUIRE(arena2.free_ == free);
REQUIRE(arena2.limit_ == limit);
REQUIRE(arena2.hi_ == hi);
REQUIRE(arena2.committed_z_ == committed_z);
}
TEST_CASE("allocator-any-1", "[alloc2][AAllocator]")
{
#ifdef NOPE
ArenaConfig cfg { .name_ = "testarena",
.size_ = 1 };
DArena arena = DArena::map(cfg);
#endif
/* empty allocator */
obj<AAllocator> alloc1;
REQUIRE(!alloc1);
REQUIRE(alloc1.iface() != nullptr);
REQUIRE(alloc1.data() == nullptr);
}
} /*namespace ut*/
} /*namespace xo*/

40
xo-facet/docs/implementation.rst
View file

@ -25,51 +25,33 @@ Abstraction tower for *xo-facet* components.
+--------------------------------+
| OObject(A,D) |
+--------------------------------+
| FacetImplmentationType(A,D) |
| FacetImplType(A,D) |
+----------------+---------------+
| facet [A] | data [D] |
+----------------+---------------+
Decorated with sample method calls, to reveal type recovery
.. ditaa::
:--scale: 0.85
+--------------------------------+
| obj(A,D) | x.foo()
+--------------------------------+
| RRouter(A,D) | x.foo()
+--------------------------------+
| OObject(A,D) | x.iface_.foo(x.data_)
+--------------------------------+
| FacetImplmentationType(A,D) | x.foo(void*data)
+----------------+---------------+
| facet A | data D | virtual x.foo(void* data)
+----------------+---------------+
.. list-table:: Descriptions
:header-rows: 1
:widths: 30 30 60
:widths: 18 30 50
* - Component
- Use
- Description
* - obj<A,D>
- x.foo()
* - ``obj<A,D>``
- ``x.foo()``
- convenience wrapper with interface A, with state D*
* - RRouter<A,D>
- x.foo()
* - ``RRouter<A,D>``
- ``x.foo()``
- auto injects data pointer
* - OObject<A,D>
- x.iface()->foo(x.data(), ..)
* - ``OObject<A,D>``
- ``x.iface()->foo(x.data(), ..)``
- fat object pointer. combine i/face + data pointer.
* - FacetImplementationType<A,D>
- x.foo(void* data, ..)
* - ``FacetImplType<A,D>``
- ``x.foo(void* data, ..)``
- implement facet for a particular state datatype;
explicit type-erased state
* - facet
- x.foo(void* data, ..)=0
- ``x.foo(void* data, ..)=0``
- fully abstract interface; explicit type-erased state
.. uml::

2
xo-ordinaltree/utest/CMakeLists.txt
View file

@ -4,6 +4,7 @@
set(SELF_EXE utest.tree)
set(SELF_SOURCE_FILES tree_utest_main.cpp redblacktree.cpp bplustree.cpp RedBlackTree-gc.test.cpp)
if (ENABLE_TESTING)
add_executable(${SELF_EXE} ${SELF_SOURCE_FILES})
xo_include_options2(${SELF_EXE})
@ -26,5 +27,6 @@ xo_dependency(${SELF_EXE} randomgen)
# 3rd part dependency: catch2:
xo_external_target_dependency(${SELF_EXE} Catch2 Catch2::Catch2)
endif()
# end ordinaltree/utest/CMakeLists.txt