wallet/docs/libraries/cpp_multithreading.md

title
C++ Multithreading

Computers have to handle many different things at once, for example screen, keyboard, drives, database, internet.

These are best represented as communicating concurrent processes, with channels, as in Go routines. Even algorithms that are not really handling many things at once, but are doing a single thing, such as everyone’s sample program, the sieve of Eratosthenes, are cleanly represented as communicating concurrent processes with channels.

On the other hand, also, not quite so cleanly, represented by asynch await which makes for much lighter weight code, more cleanly interfaceable with C++.

Concurrency is not the same thing as parallelism.

A node.js program is typically thousands of communicating concurrent processes, with absolutely no parallelism, in the sense that node.js is single threaded, but a node.js program typically has an enormous number of code continuations, each of which is in effect the state of a concurrent communicating process. Lightweight threads as in Go are threads that on hitting a pause get their stack state stashed into an event handler and executed by event oriented code, so one can always accomplish the same effect more efficiently by writing directly in event oriented code.

And it is frequently the case that when you cleverly implement many concurrent processes with more than one thread of execution, so that some of your many concurrent processes are executed in parallel, your program runs slower, rather than faster.

C++ multithreading is written around a way of coding that in practice does not seem all that useful – parallel bitbashing. The idea is that you are doing one thing, but dividing that one thing up between several threads to get more bits bashed per second, the archetypical example being a for loop performed in parallel, and then all the threads join after the loop is complete.

The normal case however is that you want to manage a thousand things at once, for example a thousand connections to the server. You are not worried about how many millions of floating point operations per second, but you are worried about processes sitting around doing nothing while waiting for network or disk operations to complete.

For this, you need concurrent communicating processes, as in Go or event orientation as in node.js or nginx, node.js, not necessarily parallelism, which C++ threads are designed around.

The need to deal with many peers and a potentially enormous number of clients suggests multiprocessing in the style of Go and node.js, rather than what C++ multiprocessing is designed around, suggests a very large number of processes that are concurrent, but not all that parallel, rather than a small number of processes that are concurrent and also substantially parallel. Representing a process by a thread runs into troubles at around sixty four threads.

It is probably efficient to represent interactions between peers as threads, but client/peer are going to need either events or Go lightweight threads, and client/client interactions are going to need events.

Existing operating systems run far more than sixty four threads, but this only works because grouped into processes, and most of those processes inactive. If you have more than sixty four concurrently active threads in an active process, with the intent that half a dozen or so of those active concurrent threads will be actually executing in parallel, as for example a browser with a thread for each tab, and sixty four tabs, that active process is likely to be not very active.

Thus scaling Apache, whether as threads on windows or processes under Linux, is apt to die.

Need the solutions implemented by Tokio, Actix, Node.js and Go

Not the solutions supplied by the C++ libraries, because we are worrying about servers, not massive bit bashing.

Go routines and channels can cleanly express both the kind of problems that node.js addresses, and also address the kind of problem that C++ threads address, typically that you divide a task into a dozen subtasks, and then wait for them all to complete before you take the next step, which are hard to express as node.js continuations. Goroutines are a more flexible and general solution, that make it easier to express a wider range of algorithms concisely and transparently, but I am not seeing any mass rush from node.js to Go. Most of the time, it is easy enough to write in code continuations inside an event handler.

The general concurrent task that Google’s massively distributed database is intended to express is that you have a thousand tasks each of which generate a thousand outputs, which get sorted, and each of the enormous number of items that sort into the same equivalence group gets aggregated in a commutative operation, which can therefore be handled by any number of processes in any order, and possibly the entire sort sequence gets aggregated in an associative operation, which can therefore be handled by any number of processes in any order.

The magic in the Google massively parallel database is that one can define a a massively parallel operation on a large number of items in a database simultaneously, much as one defines a join in SQL, and one can define another massively parallel operation as commutative and or associative operations on the sorted output of such a massively parallel operation. But we are not much interested in this capability. Though something resembling that is going to be needed when we have to shard.

doing node.js in C++

Dumb idea. We already have the node.js solution in a Rust library.

Actix and Tokio are the (somewhat Cish) solutions.

Use Go

Throw up hands in despair, and provide an interface linking Go to secure Zooko ids, similar to the existing interface linking it to Quic and SSL.

This solution has the substantial advantage that it would then be relatively easy to drop in the existing social networking software written in Go, such as Gitea.

We probably don’t want Go to start managing C++ spawned threads, but the Go documentation seems to claim that when a Go heavyweight thread gets stuck at a C mutex while executing C code, Go just spawns another to deal with the lightweight threads when the lightweight threads start piling up.

When a C++ thread wants to despatch an event to Go, it calls a Go routine with a select and a default, so that the Go routine will never attempt to pause the C++ spawned thread on the assumption that it is a Go spawned thread. But it would likely be safer to call Goroutines on a thread that was originally spawned by Go.

doing it in C the C way

Processes represented as threads. Channels have a mutex. A thread grabs total exclusive ownership of a channel whenever it takes something out or puts something in. If a channel is empty or full, it then waits on a condition on the mutex, and when the other thread grabs the mutex and makes the channel ready, it notices that the other process or processes are waiting on condition, the condition is now fulfilled, and sends a notify_one.

Or, when the channel is neither empty nor full, we have an atomic spin lock, and when sleeping might become necessary, then we go to full mutex resolution.

Which implies a whole pile of data global to all threads, which will have to be atomically changed.

This can be done by giving each thread two buffers for this global data subject to atomic operations, and single pointer or index that points to the currently ruling global data set. (The mutex is also of course global, but the flag saying whether to use atomics or mutex is located in a data structure managed by atomics.)

When a thread wants to atomically update a large object (which should be sixty four byte aligned) it constructs a copy of the current object, and atomically updates the pointer to the copy, if the pointer was not changed while it was constructing. The object is immutable while being pointed at.

Or we could have two such objects, with the thread spinning if one is in use and the other already grabbed, or momentarily sleeping if an atomic count indicates other threads are spinning on a switch awaiting completion.

The read thread, having read, stores its read pointer atomically with memory_order_release, ored with the flag saying if it is going to full mutex resolution. It then reads the write pointer with memory_order_acquire, that the write thread atomically wrote with memory_order_release, and if all is well, keeps on reading, and if it is blocked, or the write thread has gone to mutex resolution, sets its mutex resolution flag and proceeds to mutex resolution. When it is coming out of mutex resolution, about to release the mutex, it clears its mutex resolution flag. The mutex is near the flags by memory location, all part of one object that contains a mutex and atomic variables.

So the mutex flag is atomically set when the mutex has not yet been acquired, but the thread is unconditionally going to acquire it, but non atomically cleared when the mutex still belongs to the thread, but is unconditionally going to release it.

If many read threads reading from one channel, then each thread has to memory_order_acquire the read pointer, and then, instead of memory_order_releaseing it, has to do an atomic_compare_exchange_weak_explicit, and if it changed while it was reading abort its reads and start over.

Similarly if many write threads writing to one channel, each write thread will have first spin lock acquire the privilege of being the sole write thread writing, or spin lock acquire a range to write to. Thus in the most general case, we have a spin locked atomic write state that specifies an area that has been written to, an area that is being written to, and an area that is available to be acquired for writing, a spin locked atomic read state, and mutex that holds both the write state and the read state. In the case of a vector buffer with multiple writers, the atomic states are three wrapping atomic pointers that go through the buffer in the same direction,

We would like to use direct memory addresses, rather than vector or deque addresses, which might require us to write our own vector or deque. See the thread safe deque, which however relies entirely on locks and mutexes, and whose extension to atomic locks is not obvious.

Suppose you are doing atomic operations, but some operations might be expensive and lengthy. You really only want to spin lock on amending data that is small and all in close together in memory, so on your second spin, the lock has likely been released.

Well, if you might need to sleep a thread, you need a regular mutex, but how are you going to interface spin locks and regular mutexes?

You could cleverly do it with notifies, but I suspect it is costly compared to just using a plain old vanilla mutex. Instead you have some data protected by atomic locks, and some data protected by regular old mutexes, and any time the data protected by the regular old mutex might change, you atomically flag a change coming up, and every thread then grabs the mutex in order to look amend or even look at the data, until on coming out of the mutex with the data, they see the flag saying the mutex protected data might change is now clear.

After one has flagged the change coming up, and grabbed the mutex, wha happens if another thread is cheerfully amending the data in a fast operation, having started before you grabbed the mutex? The other thread has to be able to back out of that, and then try again, this try likely to be with mutex resolution. But what if the other thread wants to write into a great big vector, and reallocations of the vector are mutex protected. And we want atomic operations so that not everyone has to grab the mutex every time.

Well, any time you want to do something to the vector, it fits or it does not. And if it does not fit, then mutex time. You want all threads to switch to mutex resolution, before any thread actually goes to work reallocating the vector. So you are going to have to use the costly notify pattern. “I am out of space, so going to sleep until I can use the mutex to amend the vector. Wake me up when last thread using atomics has stopped using atomics that directly reference memory, and has switched to reading the mutex protected data, so that I can change the mutex protected data.”

The std::vector documentation says that vector access is just as efficient as array access, but I am a little puzzled by this claim, as a vector can be moved, and specifically requests that you have a no throw move operation for optimization, and having a no copy is standard where it contains things that might have ownership. (Which leads to complications when one has containers of containers, since C++ is apt to helpfully generate a broken copy implementation.)

Which would suggest that vector access is through indirection, and indirects with threading create problems.

lightweight threads in C

A lightweight thread is just a thread where, whenever a lightweight thread needs to be paused by its heavyweight thread, the heavyweight thread stores the current stack state in the heap, and move on to deal with other lightweight threads that need to be taken care of. Which collection of preserved lightweight thread stack states amount to a pile of event handlers that are awaiting events, and having received events, are then waiting for a heavyweight thread to process that event handler.

Thus one winds up with what suspect it the Tokio solution, a stack that is a tree, rather than a stack.

Hence the equivalence between node.js and nginx event oriented programming, and Go concurrent programming.

costs

Windows 10 is limited to sixty four threads total. If you attempt to create more threads than that, it still works, but performance is apt to bite, with arbitrary and artificial thread blocking. Hence goroutines, that implement unofficial threads inside the official threads.

Thread creation and destruction is fast, five to twenty microseconds, so thread pools do not buy you much, except that your memory is already going to be cached. Another source says 40 microseconds on windows, and fifty kilobytes per thread. So, a gigabyte of ram could have twenty thousand threads hanging around. Except that the windows thread scheduler dies on its ass.

There is a reasonable discussion of thread costs here

General message is that lots of languages have done it better, often immensely better, Go among them.

Checking the C++ threading libraries, they all single mindedly focus on the particular goal of parallelizing computationally intensive work. Which is not in fact terribly useful for anything you are interested in doing.

Atomics

typedef enum memory_order {
    memory_order_relaxed,   // relaxed
    memory_order_consume,   // consume
    /* No one, least of all compiler writers, understands what
    "consume" does.
    It has consequences which are difficult to understand or predict, 
    and which are apt to be inconsistent between architectures,
    libraries,  and compilers. */
    memory_order_acquire,   // acquire
    memory_order_release,   // release
    memory_order_acq_rel,   // acquire/release
    memory_order_seq_cst    // sequentially consistent
    /*  "sequentially consistent" interacts with the more commonly\
    used acquire and release in ways difficult to  understand or
    predict, and in ways  that  compiler and library writers
    disagree on. */
} memory_order;

I don’t think I understand how to use atomics correctly.

Atomic_compare_exchange_weak_explicit inside a while loop is a spin lock, and spin locks are complicated, apt to be inefficient, potentially catastrophic, and avoiding catastrophe is subtle and complex.

To cleanly express a concurrent algorithm you need a thousand communicating processes, as goroutines or node.js continuations, nearly all of which are sitting around waiting for the another thing to send them a message or be ready to receive their message, while atomics give you a fixed small number of threads all barreling full speed ahead. Whereupon you find yourself using spin locks.

Rather than moving data between threads, you need to move threads between data, between one continuation and the next.

Well, if you have a process that interacts with Sqlite, each thread has to have its own database connection, in which case it needs to be a pool of threads maybe you have a pool of database threads that do work received from a bunch of asynch tasks through a single fixed sized fifo queue, and send the results back through another fifo queue, with threads waking up when the queue gets more stuff in it, and going to sleep when the queue empties, with the last thread signalling “wake me up when there is something to do”, and pushback happening when buffer is full.

Go demonstrates that you can cleanly express algorithms as concurrent communicating processes using fixed size channels. An unbuffered channel is just a coprocess, with a single thread of execution switching between the two coprocesses, without any need for locks or atomics, but with a need for stack fixups. But Node.js seems to get by fine with code continuations instead of Go’s stack fixups.

A buffered channel is just a fixed size block of memory with alignment, size, and atomic wrapping read and write pointers.

Why do they need to be atomic?

So that the read thread can acquire the write pointer to see how much data is available, and release the read pointer so that the write thread can acquire the read pointer to see how much space is available, and conversely the write thread acquires the read pointer and releases the write pointer.And when write thread updates the write pointer it updates it after writing the data and does a release on the write pointer atomic, so that when the read thread does an acquire on the write pointer, all the data that the write pointer says was written will actually be there in the memory that read thread is looking at.

Multiple routines can send data into a single channel, and, with select, a single channel can receive data from any channels.

But, with go style programming, you are apt to have far more routines than actual hardware threads servicing them, so you are still going to need to sleep your threads, making atomic channels an optimization of limited value.

Your input buffer is empty. If you have one thread handling the one process for that input stream, going to have to sleep it. But this is costly. Better to have continuations that get executed when data is available in the channel, which means your channels are all piping to one thread, that then calls the appropriate code continuation. So how is one thread going to do a select on a thousand channels?

Well, we have a channel full of channels that need to be serviced. And when that channel empties, mutex.

Trouble is, I have not figured out how to have a thread wait on multiple channels. The C++ wait function does not implement a select. Well, it does, but you need a condition statement that looks over all the possible wake conditions. And it looks like all those wake conditions have to be on a single mutex, on which there is likely to be a lot of contention.

It seems that every thread grabs the lock, modifies the data protected by the lock, performs waits on potentially many condition variables all using the same lock and protected by the same lock, condition variables that look at conditions protected by the lock, then releases the lock immediately after firing the notify.

But it could happen that if we try to avoid unnecessarily grabbing the mutex, one thread sees the other thread awake, just when it is going to sleep, so I fear I have missed a spin lock somewhere in this story.

If we want to avoid unnecessary resort to mutex, we have to spin lock on a state machine that governs entry into mutex resolution. Each thread makes its decision based on the current state of channel and state machine, an does a Atomic_compare_exchange_weak_explicit to amend the state of the state machine. If the state machine has not changed, the decision goes through. If the state machine was changed, presumably by the other thread, it re-evaluates its decision and tries again.

Condition variables are designed to support the case where you have one thread or a potentially vast pool of threads waiting for work, but are not really designed to address the case where one thread is waiting for work from a potentially vast pool of threads, and I rather think I will have to handcraft a handler for this case from atomics and, ugh, dangerous spin loops implemented in atomics.

A zero capacity Go channel sort of corresponds to a C++ binary semaphore. A finite and small Go channel sort of corresponds to C++ finite and small semaphore. Maybe the solution is semaphores, rather than atomic variables. But I am just not seeing a match.

I notice that notifications seems to be built out of a critical section, with lots of grabbing a mutex and releasing a mutex, with far too much grabbing a mutex and releasing a mutex. Under the hood, likely a too-clever and complicated use of threads piling up on the same critical section. So maybe we need some spin state atomic state machine system that drops spinning threads to wait on a semaphore. Each thread on a channel drops the most recent state channel after reading, and most recent state after writing, onto an atomic variable.

But the most general case is many to many, with many processes doing a select on many channels. We want a thread to sleep if all the channels on which it is doing a select are blocked on the operation it wants to do, and we want processes waiting on a channel to keep being woken up, one at a time, as long a channel has stuff that processes are waiting on.

C++ Multithreading

std:aysnc is designed to support the case where threads spawn more threads if there is more work to do, and the pool of threads is not too large, and threads terminate when they are out of work, or do the work sequentially if doing it in parallel seems unlikely do yield benefits. C++ by default manages the decision for you.

Maybe the solution is to use threads where we need stack state, and continuations serviced by a single thread where we expect to handle one and only one reply. Node.js gets by fine on one thread and one database connection.

 #include &t;thread>
static_assert(__STDCPP_THREADS__==1, "Needs threads");
//  As thread resources have to be managed, need to be wrapped in
//  RAII
class ThreadRAII {
    std::thread & m_thread;
public:
//  As a thread object is moveable but not copyable, the thread obj
//  needs to be constructed inside the invocation of the ThreadRAII
//  constructor. */
    ThreadRAII(std::thread  & threadObj) : m_thread(threadObj){}
    ~ThreadRAII(){
        // Check if thread is joinable then detach the thread
        if(m_thread.joinable()){
            m_thread.detach();
            }
        }
    };

Examples of thread construction

    void foo(char *){
        …
    }

    class foo_functor
    {
    public:
        void operator()(char *){
            …
        }
    };
     

    int main(){
        ThreadRAII thread_one(std::thread (foo, "one"));
        ThreadRAII thread_two(
            std::thread (
                (foo_functor()),
                "two"
                )
            );
        const char three[]{"three"};
        ThreadRAII thread_lambda(
            std::thread(
                [three](){
                    …
                }
            )
        );
    }

C++ has a bunch of threading facilities that are designed for the case that a normal procedural program forks a bunch of tasks to do stuff in parallel, and then when they are all done, merges the results with join or promise and future, and then the main program does its thing.

This is not so useful when the main program is a event oriented, rather than procedural.

If the main program is event oriented, then each thread has to stick around for the duration, and has to have its own event queue, which C++ does not directly provide.

In this case threads communicate by posting events, and primitives that do thread synchronization (promise, future, join) are not terribly useful.

A thread grabs its event queue, using the mutex, pops out the next event, releases the mutex, and does its thing.

If the event queue is empty, then, without releasing it, the thread processing events waits on a condition variable. (which wait releases the mutex). When another thread grabs the event queue mutex and stuffs something into into the event queue, it fires the condition variable, which wakes up and restores the mutex of the thread that will process the event queue.

Mutexes need to construct RAII objects, one of which we will use in constructing the condition object.