wallet/docs/design/peer_socket.md

# katex title: >- Peer Socket sidebar: false notmine: false ... ::: myabstract [abstract:]{.bigbold} Most things follow the client server model, so it makes sense to have a distinction between server sockets and client sockets. But ultimately what we are doing is passing messages between entities and the revolutionary and subversive technologies, bittorrent, bitcoin, and bitmessage are peer to peer, so it makes sense that all sockets, however created wind up with same properties. ::: # factoring In order to pass messages, the socket has to know a whole lot of state. And in order handle messages, the entity handling the messages has to know a whole lot of state. So a socket api is an answer to the question how we factor this big pile of state into two smaller piles of state. Each big bundle of state represents a concurrent communicating process. Some of the state of this concurrent communicating process is on one side of our socket division, and is transparent to one side of our division. The application knows the internals of the some of the state, but the internals of socket state are opaque, while the socket knows the internals of the socket state, but the internals of the application state are opaque to it. The socket state machines think that they are passing messages of one class or a very small number of classes, to one big state machine, which messages contain an opaque block of bytes that application class serializes and deserializes. ## layer responsibilities The sockets layer just sends and receives arbitrary size blocks of opaque bytes over the wire between two machines. They can be sent with or without flow control and with or without reliability, but if the block is too big to fit in this connection's maximum packet size, the without flow control and without reliability option is ignored. Flow control and reliability is always applied to messages too big to fit in a packet. The despatch layer parses out the in-reply-to and the in-regards-to values from the opaque block of bytes and despatches them to the appropriate application layer state machine, which parses out the message type field, deserializes the message, and despatches it to the appropriate fully typed event handler of that state machine. # It is remarkable how much stuff can be done without concurrent communicating processes. Nostr is entirely implemented over request reply, except that a whole lot of requests and replies have an integer representing state, where the state likely winds up being a database rowid. The following discussion also applies if the reply-to field or in-regards-to field is associated with a database index rather than an instance of a class living in memory, and might well be handled by an instance of a class containing only a database index. # Representing concurrent communicating processes node.js represents them as continuations. Rust tokio represents them as something like continuations. Go represents them lightweight threads, which is a far more natural and easier to use representation, but under the hood they are something like continuations, and the abstraction leaks a little. The abstraction leaks a little in the case you have one concurrent process on one machine communicating with another concurrent process on another machine. Well, in C++, going to make instances of a class, that register call backs, and the callback is the event. Which had an instance of a class registered with the callback. Which in C++ is a pointer to a method of an object, which has no end of syntax that no one ever manages to get their head around. So if dog is method pointer with the argument bark, just say std::invoke(dog, bark) and let the compiler figure out how to do it. bark is, of course, the data supplied by the message and dog is the concurrent communicating process plus its registered callback. And since the process is sequential, it knows the data for the message that this is a reply to. A message may contain a reply-to field and or an in-regards-to field. In general, the in-regards-to field identifies the state machine on the server and the client, and remains unchanged for the life of the state machines. Therefore its handler function remains unchanged, though it may do different things depending on the state of the state machine and depending on the type of the message. If the message only has an in-regards-to field, then the callback function for it will normally be reginstered for the life of the councurrent process (instance) If it is an in-reply-to, the dispatch mechanism will unregister the handler when it dispatches the message. If you are going to receive multiple messages in response to a single message, then you create a new instance. In C, one represents actions of concurrent processes by a C function that takes a callback function, so in C++, a member function that takes a member function callback (warning, scary and counter intuitive syntax). Member to function pointers are a huge mess containing one hundred workarounds, and the best workaround is to not use them. People have a whole lot of ingenious ways to not use them, for example a base class that passes its primary function call to one of many derived classes. Which solution does not seem applicable to our problem. std:invoke is syntax sugar for calling weird and wonderful callable things - it figures out the syntax for you at compile time according to the type, and is strongly recommended, because with the wide variety of C++ callable things, no one can stretch their brain around the differing syntaxes. The many, many, clever ways of not using member pointers just do not cut it, for the return address on a message ultimately maps to a function pointer, or something that is exactly equivalent to a function pointer. Of course, we very frequently do not have any state, and you just cannot have a member function to a static function. One way around this problem is just to have one concurrent process whose state just does not change, one concurrent process that cheerfully handles messages from an unlimited number of correspondents, all using the same in-regards-to, which may well be a well known named number, the functional equivalent of a static web page. It is a concurrent process, like all the others, and has its own data like all the others, but its data does not change when it responds to a message, so never expects an in-reply-to response, or if does, creates a dynamic instance of another type to handle that. Because it does not remember what messages it sent out, the in-reply-to field is no use to it. Or, possibly our concurrent process, which is static and stateless in memory, nonetheless keeps state in the database, in which case it looks up the in-reply-to field in the database to find the context. But a database lookup can hang a thread, which we do not want to stall network facing threads. So we have a single database handling thread that sequentially handles a queue of messages from network facing threads driving network facing concurrent processes, drives database facing concurrent processes, which dispatch the result into a queue that is handled by network facing threads that drive network facing concurrent processes. So, a single thread that handles the network card, despatching message out from a queue in memory, and in from queue in memory, and does not usually or routinely do memory allocation or release, or handles them itself if they are standard, common, and known to be capable of being quickly handled, a single thread that handles concurrent systems that are purely memory to memory, but could involve dynamic allocation of memory, and a single thread that handles concurrent state machines that do database lookups and writes and possibly dynamic memory allocation, but do not directly interact with the network, handing that task over to concurrent state machines in the networking thread. So a message comes in through the wire, where it is handled by a concurrent process, probably a state machine with per connection state, though it might have substates, child concurrent processes, for reassembling one multipart message without hanging the next, It then passes that message to a state machine in the application layer, which is queued up in the queue for the thread or threads appropriate to its destination concurrent process, and receives messages from those threads, which it then despatches to the wire. A concurrent process is of course created by another concurrent process, so when it completes, does a callback on the concurrent process that created it, and any concurrent processes it has created are abruptly discarded. So our external messages and events involve a whole lot of purely internal messages and events. And the event handler has to know what internal object this message came from, which for external messages is the in-regards-to field, or is implicit in the in-reply-to field. If you could be receiving events from different kinds of objects about different matters, well, you have to have different kinds of handlers. And usually you are only receiving messages from only one such object, but in irritatingly many special cases, several such objects. But it does not make sense to write for the fully general case when the fully general case is so uncommon, so we handle this case ad-hoc by a special field, which is defined only for this message type, not defined as a general quality of all messages. It typically makes sense to assume we are handling only one kind of message, possibly of variant type, from one object, and in the other, special, cases, we address that case ad hoc by additional message fields. But if we support std:variant, there is a whole lot of overlap between handling things by a new variant, and handling things by a new callback member. The recipient must have associated a handler, consisting of a call back and an opaque pointer to the state of the concurrent process on the recipient with the messages referenced by at least one of these fields. In the event of conflicting values, the reply-to takes precedence, but the callback of the reply-to has access to both its data structure, and the in-regards-to dat structure, a pointer to which is normally in its state. The in-regards-to being the state machine, and the in-reply-to the event that modifies the state of the state machine. When we initialize a connection, we establish a state machine at both ends, both the application factor of the state machine, and the socket factor of the state machine. When I say we are using state machines, this is just the message handling event oriented architecture familiar in programming graphical user interfaces. Such a program consists of a pile of derived classes whose base classes have all the machinery for handling messages. Each instance of one of these classes is a state machine, which contains member functions that are event handlers. So when I say "state machine", I mean a class for handling events like the many window classes in wxWidgets. One big difference will be that we will be posting a lot of events that we expect to trigger events back to us. And we will want to know which posting the returning event came from. So we will want to create some data that is associated with the fired event, and when a resulting event is fired back to us, we can get the correct associated data, because we might fire a lot of events, and they might come back out of order. Gui code has this capability, but it is rarely used. ## Implementing concurrent state machines in C++ Most of this is me re-inventing Asio, which is part of the immense collection of packages of Msys2, Obviously I would be better off integrating Asio than rebuilding it from the ground up But I need to figure out what needs to be done, so that I can find the equivalent Asio functionality. Or maybe Asio is bad idea. Boost Asio was horribly broken. I am seeing lots of cool hot projects using Tokio, not seeing any cool hot projects use Asio. If Bittorrent DHT library did their own implementation of concurrent communicating processes, maybe Asio is broken at the core And for flow control, I am going to have to integrate Quic, though I will have to fork it to change its security model from certificate authorities to Zooko names. You can in theory easily plug any kind of socket into Asio, but I see a suspicious lack of people plugging Quic into it, because Quic contains a huge amount of functionality that Asio knows nothing of. But if I am forking it, can probably ignore or discard most of that functionality. Gui code is normally single threaded, because it is too much of a bitch to lock an instance of a message handling class when one of its member functions is handling a message (the re-entrancy problem). However the message plumbing has to know which class is going to handle the message (unless the message is being handled by a stateless state machine, which it often is) so there is no reason the message handling machinery could not atomically lock the class before calling its member function, and free it on return from its member function. State machines (message handling classes, as for example in a gui) are allocated in the heap and owned by the message handling machinery. The base constructor of the object plugs it into the message handling machinery. (Well, wxWidgets uses the base constructor with virtual classes to plug it in, but likely the curiously recurring template pattern would be saner as in ATL and WTL.) This means they have to be destroyed by sending a message to the message handling machinery, which eventually results in the destructor being called. The destructor does not have to worry about cleanup in all the base classes, because the message handling machinery is responsible for all that stuff. Our event despatcher does not call a function pointer, because our event handlers are member functions. We call an object of type std::function. We could also use pointer to member, which is more efficient. All this complicated machinery is needed because we assume our interaction is stateful. But suppose it is not. The request‑reply pattern, where the request contains all information that determines the reply is very common, probably the great majority of cases. This corresponds to an incoming message where the in‑regards‑to field and in‑reply‑to field is empty, because the incoming message initiates the conversation, and its type and content suffices to determine the reply. Or the incoming message causes the recipient to reply and also set up a state machine, or a great big pile of state machines (instances of a message handling class), which will handle the lengthy subsequent conversation, which when it eventually ends results in those objects being destroyed, while the connection continues to exist. In the case of an incoming message of that type, it is despatched to a fully re-entrant static function on the basis of its type. The message handling machinery calls a function pointer, not a class member. We don't use, should not use, and cannot use, all the message handling infrastructure that keeps track of state. ## receive a message with no in‑regards‑to field, no in‑reply‑to field This is directed to a re-entrant function, not a functor, because re‑entrant and stateless. It is directed according to message type. ### A message initiating a conversation It creates a state machine (instance of a message handling class) sends the start event to the state machine, and the state machine does whatever it does. The state machine records what message caused it to be created, and for its first message, uses it in the in‑reply‑to field, and for subsequent messages, for its in‑regards‑to field, ### A request-reply message. Which sends a message with the in-reply-to field set. The recipient is expected to have a hash-map associating this field with information as to what to do with the message. #### A request-reply message where counterparty matters. Suppose we want to read information about this entity from the database, and then write that information. Counterparty information is likely to be needed to be durable. Then we do the read-modify write as a single sql statement, and let the database serialize it. ## receive a message with no in‑regards‑to field, but with an in‑reply‑to field The dispatch layer looks up a hash-map table of functors, by the id of the field and id of the sender, and despatches the message to that functor to do whatever it does. When this is the last message expected in‑reply‑to the functor frees itself, removes itself from the hash-map. If a message arrives with no entry in the table, it is silently dropped. ## receive a message with an in‑regards‑to field, with or without an in‑reply‑to to field. Just as before, the dispatch table looks up the hash-map of state machines (instances of message handling classes) and dispatches the message to the stateful message handler, which figures out what to do with it according to its internal state. What to do with an in‑reply‑to field, if there is one, is something the stateful message handler will have to decide. It might have its own hashmap for the in‑reply‑to field, but this would result in state management and state transition of huge complexity. The expected usage is it has a small number of static fields in its state that reference a limited number of recently sent messages, and if the incoming message is not one of them, it treats it as an error. Typically the state machine is only capable of handling the response to its most recent message, and merely wants to be sure that this is a response to its most recent message. But it could have shot off half a dozen messages with the in‑regards‑to field set, and want to handle the response to each one differently. Though likely such a scenario would be easier to handle by creating half a dozen state machines, each handling its own conversation separately. On the other hand, if it is only going to be a fixed and finite set of conversations, it can put all ongoing state in a fixed and finite set of fields, each of which tracks the most recently sent message for which a reply is expected. ## A complex conversation. We want to avoid complex conversations, and stick to the request‑reply pattern as much as possible. But this is apt to result in the server putting a pile of opaque data (a cookie) its reply, which it expects to have sent back with every request. And these cookies can get rather large. Bob decides to initiate a complex conversation with Carol. He creates an instance of a state machine (instance of a message handling class) and sends a message with no in‑regards‑to field and no in‑reply‑to field but when he sends that initial message, his state machine gets put in, and owned by, the dispatch table according to the message id. Carol, on receiving the message, also creates a state machine, associated with that same message id, albeit the counterparty is Bob, rather than Carol, which state machine then sends a reply to that message with the in‑reply‑to field set, and which therefore Bob's dispatch layer dispatches to the appropriate state machine (message handler) And then it is back and forth between the two stateful message handlers both associated with the same message id until they shut themselves down. ## factoring layers. A layer is code containing state machines that receive messages on one machine, and then send messages on to other code on the same machine. The sockets layer is the code that receives messages from the application layer, and then sends them on the wire, and the code that receives messages from the wire, and sends messages to the application layer. But a single state machine at the application level could be handling several connections, and a single connection could have several state machines running independently, and the socket code should not need to care. We have a socket layer that receives messages containing an opaque block of bytes, and then sends a message to the application layer message despatch machinery, for whom the block is not opaque, but rather identifies a message type meaningful for the despatch layer, but meaningless for the socket layer. The state machine terminates when its job is done, freeing up any allocated memory, but the connection endures for the life of the program, and most of the data about a connection endures in an sql database between reboots. The connection is a long lived state machine running in the sockets layer, which sends and receives what are to it opaque blocks of bytes to and from the dispatch layer, and the dispatch layer interprets these blocks of bytes as having information (message type, in‑regards‑to field and in‑reply‑to field) that enables it to despatch the message to a particular method of a particular instance of a message handling class in C++, corresponding to a particular channel in Go. And these message handling classes are apt to be short lived, being destroyed when their task is complete. Because we can have many state machines on a connection, most of our state machines can have very little state, typically an infinite receive loop, an infinite send receive loop, or an infinite receive send loop, which have no state at all, are stateless. We factorize the state machine into many state machines to keep each one manageable. ## Comparison with concurrent interacting processes in Go These concurrent communicating processes are going to be sending messages to each other on the same machine. We need to model Go's goroutines. A goroutine is a function, and functions always terminate -- and in Go are unceremoniously and abruptly ended when their parent function ends, because they are variables inside its dataspace, as are their channels. And, in Go, a channel is typically passed by the parent to its children, though they can also be passed in a channel. Obviously this structure is impossible and inapplicable when processes may live, and usually do live, in different machines. The equivalent of Go channel is not a connection. Rather, one sends a message to the other to request it create a state machine, which will correspond to the in-regards-to message, and the equivalent of a Go channel is a message type, the in-regards-to message id, and the connection id. Which we pack into a single class so that we can use it the way Go uses channels. The sockets layer (or another state machine on the application layer) calls the callback routine with the message and the state. The sockets layer treats the application layer as one big state machine, and the information it sends up to the application enables the application layer to despatch the event to the correct factor of that state machine, which we have factored into as many very small, and preferably stateless, state machines as possible. We factor the potentially ginormous state machine into many small state machines (message handling classes), in the same style as Go factors a potentially ginormous Goroutine into many small goroutines. The socket code being a state machine composed of many small state machines, which communicates with the application layer over a very small number of channels, these channels containing blocks of bytes that are opaque to the socket code, but are serialized and deserialized by the application layer code. From the point of view of the application layer code, it is many state machines, and the socket layer is one big state machine. From the point of view of the socket code, it is many state machines, and the application layer is one big state machine. The application code, parsing the the in-reply-to message id, and the in-regard-to message id, figures out where to send the opaque block of bytes, and the recipient deserializes, and sends it to a routine that acts on an object of that deserialized class. Since the sockets layer does not know the internals of the message struct, the message has to be serialized and deserialized into the corresponding class by the dispatch layer, and thence to the application layer. Go code tends to consist of many concurrent processes continually being spawned by a master concurrent process, and themselves spawning more concurrent processes. # flow control and reliability If we want to transmit a big pile of data, a big message, well, this is the hard problem, for the sender has to throttle according to the recipient's readiness to handle it and the physical connections capability to transmit it. Quic is a UDP protocol that provides flow control, and the obvious thing to handle bulk data transfer is to fork it to use Zooko based keys. [Tailscale]:https://tailscale.com/blog/how-nat-traversal-works "How to communicate peer-to-peer through NAT firewalls"{target="_blank"} [Tailscale] has solved a problem very similar to the one I am trying to solve, albeit their solutions rely on a central human authority, which authority they ask for money and they recommend: > If you’re reaching for TCP because you want a > stream‑oriented connection when the NAT traversal is done, > consider using QUIC instead. It builds on top of UDP, > so we can focus on UDP for NAT traversal and still have a > nice stream protocol at the end. But to interface QUIC to a system capable of handling a massive number of state machines, going to need something like Tokio, because we want the thread to service other state machines while QUIC is stalling the output or waiting for input. Indeed, no matter what, if we stall in the socket layer rather than the application layer, which makes life a whole lot easier for the application programmer, going to need something like Tokio. Or we could open up Quic, which we have to do anyway to get it to use our keys rather than enemy controlled keys, and plug it into our C++ message passing layer. On the application side, we have to lock each state machine when it is active. It can only handle one message at at time. So the despatch layer has to queue up messages and stash them somewhere, and if it has too many messages stashed, it wants to push back on the state machine at the application layer at the other end of the wire. So the despatch layer at the receiving end has to from time to time tell the despatch layer at the sending end "I have n bytes in regard to message 'Y', and can receive m more. And when the despatch layer at the other end, which unlike the socket layer knows which state machine is communicating with which, has more than that amount of data to send, it then blocks and locks the state machine at its end in its send operation. The socket layer does not know about that and does not worry about that. What it worries about packets getting lost on the wire, and caches piling up in the middle of the wire. It adds to each message a send time and a receive time and if the despatch layer wants to send data faster than it thinks is suitable, it has to push back on the despatch layer. Which it does in the same style. It tells it the connection can handle up to m further bytes. Or we might have two despatch layers, one for sending and one for receiving, with the send state machine sending events to the receive state machine, but not vice versa, in which case the socket layer can block the send layer. # Tokio Most of this machinery seems like a re-implementation of Tokio-rust, which is a huge project. I don't wanna learn Tokio-rust, but equally I don't want to re-invent the wheel. Or perhaps we could just integrate QUICs internal message passing infrastructure to our message passing infrastructure. It probably already supports a message passing interface. Instead of synchronously writing data, you send a message to it to write some data, and hen it is done, it calls a callback. # Minimal system Prototype. Limit global bandwidth at the application state machine level -- they adjust their policy according to how much data is moving, and they spread the non response outgoing messages out to a constant rate (constant per counterparty, and uniformly interleaved.) Single threaded, hence no state machine locking. Tweet style limit on the size of messages, hence no fragmentation and re-assembly issue. Our socket layer becomes trivial - it just send blobs like a zeromq socket. If you are trying to download a sackload of data, you request a counterparty to send a certain amount to you at a given rate, he immediately responds (without regard to global bandwidth limits) with the first instalment, and a promise of further instalments at a certain time) Each instalment records how much has been sent, and when, when the next instalment is coming, and the schedule for further instalments. If you miss an instalment, you nack it after a delay. If he receives a nack, he replaces the promised instalments with the missing ones. The first thing we implement is everyone sharing a list of who they have successfully connected to, in recency order, and everyone keeps everyone else's list, which catastrophically fails to scale, and also how up to date their counter parties are with their own list, so that they do not have endlessly resend data (unless the counterparty has a catastrophic loss of data, and requests everything from the beginning.) We assume everyone has an open port, which is sucks intolerably, but once that is working we can handle ports behind firewalls, because we are doing UDP. Knowing who the other guy is connected to, and you are not, you can ask him to initiate a peer connection for the two of you, until you have enough connections that the keep alive works. And once everyone can connect to everyone else by their public username, then we can implement bitmessage.