cmake version 3.25.3
pybind11 version ???
nix build (see https://github.com:rconybea/xo-nix2) Consequences of nix build:
- Each package installed to a separate directory – no "common swimming pool" like /usr/lib
- Implies install directory always distinct from any directory containing build inputs
- Tends to reveal oversights in toolchain, as we'll see below
pybind library (xo-pyreflect) with dependency on a separate library (xo-reflect), that in turn has secondary dependencies (xo-refcnt, xo-indentlog). Note that xo-indentlog is header-only.

Expect this cmake script to work:

  find_package(pybind11)
  pybind11_add_module(pyreflect pyreflect.cpp)

  find_package(reflect CONFIG REQUIRED)
  target_link_libraries(pyreflect PUBLIC reflect)

Problem

Instead, link fails. Link line something like:
```
g++ -fPIC ... -o pyreflect.cpython-311-x86_64-linux-gnu.so /path/to/libreflect.so -lrefcnt -lindentlog
```
Two problems here:
1. directory containing librefcnt.so isn't on the link line (no -L/path/to/refcnt/dir for example).
2. libindentlog.so does not exist, since indentlog is header-only
Looked into intermediate outputs like lib/cmake/reflectTargets.cmake, excerpt:
```
    set_target_properties(reflect PROPERTIES
        INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include"
        INTERFACE_LINK_LIBRARIES "indentlog;refcnt"
    )
```
It's not obvious how xo_pyreflect can know that indentlog is header-only, while refcnt isn't (though could presumably extract the relevant libdir from find_package() with some work).

Workaround

Recognize that pyreflect link shouldn't need refcnt on the link line, since libreflect.so has a DT_NEEDED entry for it.

$ readelf -d /path/to/libreflect.so

Dynamic section at offset 0x17860 contains 34 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [librefcnt.so.1]
...

When building pyreflect, suppress transitive dependencies For example:

  # xo_cxx.cmake
  macro(xo_pybind11_dependency target dep)
      find_package(${dep} CONFIG REQUIRED)
      set_property(TARGET ${dep} PROPERTY INTERFACE_LINK_LIBRARIES "")
      target_link_libraries(${target} PUBLIC ${dep})
  endmacro()

Then in .cmake for pyreflect, something equivalent to:

  pybind11_add_module(pyreflect pyreflect.cpp)
  xo_pybind11_dependency(pyreflect reflect)

lsp (language server process)

mysterious lsp complaints about iostream headers (22feb2024)

Setup

working on bespoke streambuf implementation (cmake-examples/zstream/include/zstream/zstreambuf.hpp)

Problem

getting mysterious errors from flymake + emacs, e.g. message "<fstream> not found"

Solution

Had discarded cmake-examples/build directory, using cmake-examples/.build instead. This was to prevent build tree showing up when running tree in the cmake-examples directory
Left cmake-examples/compile_commands.json as a broken symlink referring to old build directory
This causes lsp understanding of compiler invocation to deteriorate as project acquires new files
Fix by swinging symlink to cmake-examples/.build/compile_commands.json, duh!

iostream

general api rant (25feb2024)

`istream.read()` doesn't report the number of bytes/chars read.

Instead of:

  istream & istream::read (char_type * s, std::streamsize count);

I'd prefer signature

  istream & istream::read (char_type * s, std::streamsize count, std::streamsize * p_gcount);

Developers are expected to use.

  std::streamsize istream::gcount () const;

I think this is inferior, since relies on state held by istream, that will be discarded on next read operation.

`istream.read(s, n)` expects always to read n chars.

It sets failbit if less than n chars read.

Apparent alternatives are unsatisfactory:

istream & readsome(s, n) isn't required to do any physical i/o; instead reports what's available already in memory
istream & get(s, n, delim) only reads up to first occurence of delim.
istream & get(s, n) is just a convenience for istream::get(s, n, '\n').
could try writing a loop using combination of istream::sync(), istream::readsome(), but that won't work if istream is actually unbuffered.
istream s; s.rdbuf()->sgetn(s, n) bypasses istream code for sentry object etc, and can't set istream's eofbit.

The following workaround is viable, except that it will read one-byte-at-a-time if input alternates between bytes values '\x0' and '\xff':

  template<typename istream>
  std::streamsize
  read_upto(istream & in, istream::char_type * s, std::streamsize n)
  {
      std::streamsize n_read = 0;

      constexpr char c_bits = '\x0'; /*any char value will do here*/

      char delim = c_bits;

      for (; in.good() && !in.eof() && (n_read < n); delim = delim ^ '\xff') {
          // each iteration alternates between {c_bits, ~c_bits} as delimiter,
          // so guarantees at least one byte progress every two iterations

          in.get(s, n, delim);

          std::streamsize nr = in.gcount();
          if (nr > 0) {
              n_read += nr;
              s += nr;
          }
      }

      return n_read;
  }

I'd prefer to support this behavior (without the performance-accident-waiting-to-happen) directly from istream.

Another strategy is to use istream::peek() to check for input and istream::readsome() to fetch it

  template<typename istream>
  std::streamsize
  read_upto(istream & in, istream::char_type * s, std::streamsize n)
  {
      std::streamsize n_read = 0;

      while (in.good() && !in.eof() && (n_read < n))) {
          in.peek();   /* ensure at least one byte available in streambuf */

          std::streamsize nr = in.readsome(s + n_read, n - n_read);

          n_read += nr;
      }
  }

This works if streambuf actually does buffering. It may be very slow if streambuf is unbuffered.

istream::sentry looks interesting, but doesn't do any reading (except to possibly skip whitespace).

gcc 12.2.0's implementation:

  template<typename _CharT, typename _Traits>
  basic_istream<_CharT, _Traits>::sentry::
  sentry(basic_istream<_CharT, _Traits>& __in, bool __noskip) : _M_ok(false)
  {
      ios_base::iostate __err = ios_base::goodbit;
      if (__in.good())
      {
          __try
          {
              if (__in.tie())
                  __in.tie()->flush();
              if (!__noskip && bool(__in.flags() & ios_base::skipws))
              {
                  const __int_type __eof = traits_type::eof();
                  __streambuf_type* __sb = __in.rdbuf();
                  __int_type __c = __sb->sgetc();

                  const __ctype_type& __ct = __check_facet(__in._M_ctype);
                  while (!traits_type::eq_int_type(__c, __eof)
                         && __ct.is(ctype_base::space,
                                    traits_type::to_char_type(__c)))
                      __c = __sb->snextc();

                  // _GLIBCXX_RESOLVE_LIB_DEFECTS
                  // 195. Should basic_istream::sentry's constructor ever
                  // set eofbit?
                  if (traits_type::eq_int_type(__c, __eof))
                      __err |= ios_base::eofbit;                // (A)
              }
          }
          __catch(__cxxabiv1::__forced_unwind&)
          {
              __in._M_setstate(ios_base::badbit);
              __throw_exception_again;
          }
          __catch(...)
          { __in._M_setstate(ios_base::badbit); }
      }

      if (__in.good() && __err == ios_base::goodbit)            // (B)
          _M_ok = true;
      else
      {
          __err |= ios_base::failbit;                           // (C)
          __in.setstate(__err);
      }
  }

with

  template<typename _Facet>
  inline const _Facet&
  __check_facet(const _Facet* __f)
  {
      if (!__f)
          __throw_bad_cast();
      return *__f;
  }

Note that if __noskipws is false and sentry encounters eof, then the line marked (A) executes –> test (B) fails –> (C) executes, flagging stream as in an 'unrecoverable error state'. The line (A) appears to be mandatory (in spite of the inline comment).

From https://cppreference.com:

explicit sentry( std::basic_istream<CharT, Traits>& is, bool noskipws = false );

Prepares the stream for formatted input.

If is.good() is false, calls is.setstate(std::ios_base::failbit) and returns. Otherwise, if is.tie() is not a null pointer, calls is.tie()->flush() to synchronize the output sequence with external streams. This call can be suppressed if the put area of is.tie() is empty. The implementation may defer the call to flush() until a call of is.rdbuf()->underflow() occurs. If no such call occurs before the sentry object is destroyed, it may be eliminated entirely.

If noskipws is zero and is.flags() & std::ios_base::skipws is nonzero, the function extracts and discards all whitespace characters until the next available character is not a whitespace character (as determined by the currently imbued locale in is). If is.rdbuf()->sbumpc() or is.rdbuf()->sgetc() returns traits::eof(), the function calls setstate(std::ios_base::failbit | std::ios_base::eofbit) (which may throw std::ios_base::failure).

Additional implementation-defined preparation may take place, which may call setstate(std::ios_base::failbit) (which may throw std::ios_base::failure).

If after preparation is completed, is.good() == true, then any subsequent calls to operator bool will return true.

However we can bypass this with __noskip_ set to true:

  template<typename istream>
  std::streamsize
  read_upto(istream & in, istream::char_type * s, std::streamsize n)
  {
      istream::sentry sentry(in, true /*noskipws*/);

      std::streamsize n_read = 0;

      if (sentry) {
          try {
              n_read = in.rdbuf()->sgetn(s, n);

              in.setstate(ios::eofbit);
          } catch(__cxxabiv1::__forced_unwind &)  {
              in.setstate(ios::failbit);
              throw;
          } catch(...) {
              in.setstate(ios::failbit);
          }
      }

      return n_read;
  }

Another alternative would be to post-process read(), and clear failbit if set along with eofbit:

  template<typename istream>
  std::streamsize
  read_upto(istream & in, istream::char_type * s, std::streamsize n)
      {
          in.read(s, n);

          std::streamsize n_read = in.gcount();

          if ((n_read < n) && in.eof() && in.fail()) {
              /* clear failbit */
              in.clear(in.rdstate() & ~std::ios::failbit);
          }

          return n_read;
      }

Iostream get isn't monotonic

iostream.get(s, n, delim) sets failbit if first character matches delim.

This interferes with using iostream.get as building block for a longer i/o sequence;

Tripped over this while writing zstream.read_until for my cmake-examples project:

Instead of:

  std::streamsize read_until(char_type * s,
                             std::streamsize n,
                             bool check_delim_flag,
                             char_type delim)
      {
          ...

          std::streamsize nr = 0;

          this->get(s, n, delim);
          nr = this->gcount();

          ...

          return nr;
      }

We need carve-out:

  std::streamsize read_until(char_type * s,
                             std::streamsize n,
                             bool check_delim_flag,
                             char_type delim)
      {
          ...

          std::streamsize nr = 0;

          int_type nextc = this->rdbuf_.sgetc();

          if (nextc == Traits::to_int_type(delim)) {
              nr = 0;
          } else {
              this->get(s, n, delim);

              nr = this->gcount();
          }

          ...

          return nr;
      }

Iostream position reporting isn't monotonic.

iostream.tellg() and iostream.putg() report current position w.r.t. beginning of stream for input (get) and output (put) respectively.

Unfortunately, they are not monotonic, and code like this is subtly broken:

  istream & input = ...; // some binary stream
  struct foo part1;
  struct foo part2;

  istream::pos_type p0 = input.tellg();

  input >> part1 >> part2;

  istream::pos_type p1 = input.tellg();

  istream::pos_type n_read = p1 - p0;

If stream reaches end-of-file at the end of part2, then in fact reading was successful, but p1 will be -1, and n_read will be nonsense.

Presumably this is why iostream.gcount() exists: otherwise there'd be no way to determine how many bytes/chars a preceding read obtained.

A correct (but awkward and error-prone) implementation:

  istream & input = ...;
  struct foo part1;
  struct foo part2;

  std::streamsize n_read = 0;

  input >> part1;
  n_read += input.gcount();

  input >> part2;
  n_read += input.gcount();

Streambuf not responsible for eofbit.

istream.eofbit probably belongs in streambuf. streambuf has to recognize end-of-file anyway, since it's responsible for physical I/O. It might as well record and report it.

Stream position reporting from streambuf

It would be simpler for streambuf to support istream::tellg() and istream::tellp() directly instead of relying on streambuf::seekoff(). Argument here is that even for a non-seekable stream buffer, it still makes sense to support tellg() and tellp(). This requires streambuf author to implement at least a restricted version of streambuf::seekoff(), which muddies the waters.

(ed: switching to article-based format)

zlib

(Jan? 2024) zlib z_stream cannot be moved

gcc

(June 2024) get builtin preprocessor defines from gcc

llvm

(June 2024) injecting symbols into llvm jit
(June 2024) alloca and function pointers

17 KiB Raw Permalink Blame History Unescape Escape

Today I Learned - lessons from building a C++ development stack

wsl (windows services for linux)

nix

git

webhosting + forgejo + CI

cmake

pybind11 link difficulties with transitive library dependencies (7oct2023)

Setup

Problem

Workaround

lsp (language server process)

mysterious lsp complaints about iostream headers (22feb2024)

Setup

Problem

Solution

iostream

general api rant (25feb2024)

istream.read() doesn't report the number of bytes/chars read.

istream.read(s, n) expects always to read n chars.

Iostream get isn't monotonic

Iostream position reporting isn't monotonic.

Streambuf not responsible for eofbit.

Stream position reporting from streambuf

zlib

gcc

llvm

17 KiB

Raw Permalink Blame History

`istream.read()` doesn't report the number of bytes/chars read.

`istream.read(s, n)` expects always to read n chars.