wallet/docs/design/peer_socket.md

# katex title: >- Peer Socket sidebar: false notmine: false ... ::: myabstract [abstract:]{.bigbold} Most things follow the client server model, so it makes sense to have a distinction between server sockets and client sockets. But ultimately what we are doing is passing messages between entities and the revolutionary and subversive technologies, bittorrent, bitcoin, and bitmessage are peer to peer, so it makes sense that all sockets, however created wind up with same properties. ::: # factoring In order to pass messages, the socket has to know a whole lot of state. And in order handle messages, the entity handling the messages has to know a whole lot of state. So a socket api is an answer to the question how we factor this big pile of state into two smaller piles of state. Each big bundle of state represents a concurrent communicating process. Some of the state of this concurrent communicating process is on one side of our socket division, and is transparent to one side of our division. The application knows the internals of the some of the state, but the internals of socket state are opaque, while the socket knows the internals of the socket state, but the internals of the application state are opaque to it. The socket state machines think that they are passing messages of one class or a very small number of classes, to one big state machine, which messages contain an opaque block of bytes that application class serializes and deserializes. ## layer responsibilities The sockets layer just sends and receives arbitrary size blocks of opaque bytes over the wire between two machines. They can be sent with or without flow control and with or without reliability, but if the block is too big to fit in this connection's maximum packet size, the without flow control and without reliability option is ignored. Flow control and reliability is always applied to messages too big to fit in a packet. The despatch layer parses out the in-reply-to and the in-regards-to values from the opaque block of bytes and despatches them to the appropriate application layer state machine, which parses out the message type field, deserializes the message, and despatches it to the appropriate fully typed event handler of that state machine. # Representing concurrent communicating processes A message may contain a reply-to field and or an in-regards-to field. The recipient must have associated a handler, consisting of a call back and an opaque pointer to the state of the concurrent process on the recipient with the messages referenced by at least one of these fields. In the event of conflicting values, the reply-to takes precedence, but the callback of the reply-to has access to both its data structure, and the in-regards-to dat structure, a pointer to which is normally in its state. The in-regards-to being the state machine, and the in-reply-to the event that modifies the state of the state machine. When we initialize a connection, we establish a state machine at both ends, both the application factor of the state machine, and the socket factor of the state machine. But a single state machine at the application level could be handling several connections, and a single connection could have several state machines running independently, and the socket code should not need to care. Further, these concurrent communicating processes are going to be sending messages to each other on the same machine. We need to model Go's goroutines. A goroutine is a function, and functions always terminate -- and in Go are unceremoniously and abruptly ended when their parent function ends, because they are variables inside its dataspace, as are their channels. And, in Go, a channel is typically passed by the parent to its children, though they can also be passed in a channel. Obviously this structure is impossible and inapplicable when processes may live, and usually do live, in different machines. The equivalent of Go channel is not a connection. Rather, one sends a message to the other to request it create a state machine, which will be the in-regards-to message, and the equivalent of a Go channel is a message type, the in-regards-to message id, and the connection id. Which we pack into a single class so that we can use it the way Go uses channels. The sockets layer (or another state machine on the application layer) calls the callback routine with the message and the state. The sockets layer treats the application layer as one big state machine, and the information it sends up to the application enables the application layer to despatch the event to the correct factor of that state machine, which we have factored into as many very small, and preferably stateless, state machines as possible. We factor the potentially ginormous state machine into many small state machines, in the same style as Go factors a potentially ginormous Goroutine into many small goroutines. The socket code being a state machine composed of many small state machines, which communicates with the application layer over a very small number of channels, these channels containing blocks of bytes that are opaque to the socket code, but are serialized and deserialized by the application layer code. From the point of view of the application layer code, it is many state machines, and the socket layer is one big state machine. From the point of view of the socket code, it is many state machines, and the application layer is one big state machine. The application code, parsing the the in-reply-to message id, and the in-regard-to message id, figures out where to send the opaque block of bytes, and the recipient deserializes, and sends it to a routine that acts on an object of that deserialized class. Since the sockets layer does not know the internals of the message struct, the message has to be serialized and deserialized into the corresponding class by the application layer. Or perhaps the callback routine deserializes the object into a particular class, and then calls a routine for that class, but another state machine on the application layer would call the class specific routine directly. The equivalent of Go channel between one state machine on the application layer and another in the same application layer is directly calling what the class specific routine that the callback routine would call. The state machine terminates when its job is done, freeing up any allocated memory, but the connection endures for the life of the program, and most of the data about a connection endures in an sql database between reboots. Because we can have many state machines on a connection, most of our state machines can have very little state, typically an infinite receive loop, an infinite send receive loop, or an infinite receive send loop, which have no state at all, are stateless. We factorize the state machine into many state machines to keep each one manageable. Go code tends to consist of many concurrent processes continually being spawned by a master concurrent process, and themselves spawning more concurrent processes. For most state machines, we do not need recursion, so it is reasonable for their state to be a fixed allocation inside the state of their master concurrent process. In the unlikely event we do need recursion we usually only have one instance running at one time, so we can allocate an std::stack in the master concurrent process. And what if we do want to spawn many in parallel? Well, they will usually be stateless. What if they are not not stateless? Well that would require an std::vector of states. And if we need many running in parallel with recursion, an std::vector with each element containing an std::stack. And to avoid costly reallocations, we create the std::vector and the std::vectors underlying the std::stacks with realistic initial allocations that are only infrequently exceeded. # flow control and reliability If we want to transmit a big pile of data, a big message, well, this is the hard problem, for the sender has to throttle according to the recipient's readiness to handle it and the physical connections capability to transmit it. Quic is a UDP protocol that provides flow control, and the obvious thing to handle bulk data transfer is to fork it to use Zooko based keys. [Tailscale]:https://tailscale.com/blog/how-nat-traversal-works "How to communicate peer-to-peer through NAT firewalls"{target="_blank"} [Tailscale] has solved a problem very similar to the one I am trying to solve, albeit their solutions rely on a central human authority, and they recommend: > If you’re reaching for TCP because you want a > stream‑oriented connection when the NAT traversal is done, > consider using QUIC instead. It builds on top of UDP, > so we can focus on UDP for NAT traversal and still have a > nice stream protocol at the end. But to interface QUIC to a system capable of handling a massive number of state machines, going to need something like Tokio, because we want the thread to service other state machines while QUIC is stalling the output or waiting for input. Indeed, no matter what, if we stall in the socket layer rather than the application layer, which makes life a whole lot easier for the application programmer, going to need something like Tokio. On the application side, we have to lock each state machine when it is active. It can only handle one message at at time. So the despatch layer has to queue up messages and stash them somewhere, and if it has too many messages stashed, it wants to push back on the state machine at the application layer at the other end of the wire. So the despatch layer at the receiving end has to from time to time tell the despatch layer at the sending end "I have n bytes in regard to message 'Y', and can receive m more. And when the despatch layer at the other end, which unlike the socket layer knows which state machine is communicating with which, has more than that amount of data to send, it then blocks and locks the state machine at its end in its send operation. The socket layer does not know about that and does not worry about that. What it worries about packets getting lost on the wire, and caches piling up in the middle of the wire. It adds to each message a send time and a receive time and if the despatch layer wants to send data faster than it thinks is suitable, it has to push back on the despatch layer. Which it does in the same style. It tells it the connection can handle up to m further bytes. Or we might have two despatch layers, one for sending and one for receiving, with the send state machine sending events to the receive state machine, but not vice versa, in which case the socket layer can block the send layer. # Tokio Most of this machinery seems like a re-implementation of Tokio-rust, which is a huge project. I don't wanna learn Tokio-rust, but equally I don't want to re-invent the wheel. # Minimal system Prototype. Limit global bandwidth at the application state machine level -- they adjust their policy according to how much data is moving, and they spread the non response outgoing messages out to a constant rate (constant per counterparty, and uniformly interleaved.) Single threaded, hence no state machine locking. Tweet style limit on the size of messages, hence no fragmentation and re-assembly issue. Our socket layer becomes trivial - it just send blobs like a zeromq socket. If you are trying to download a sackload of data, you request a counterparty to send a certain amount to you at a given rate, he immediately responds (without regard to global bandwidth limits) with the first instalment, and a promise of further instalments at a certain time) Each instalment records how much has been sent, and when, when the next instalment is coming, and the schedule for further instalments. If you miss an instalment, you nack it after a delay. If he receives a nack, he replaces the promised instalments with the missing ones. The first thing we implement is everyone sharing a list of who they have successfully connected to, in recency order, and everyone keeps everyone else's list, which catastrophically fails to scale, and also how up to date their counter parties are with their own list, so that they do not have endlessly resend data (unless the counterparty has a catastrophic loss of data, and requests everything from the beginning.) We assume everyone has an open port, which is sucks intolerably, but once that is working we can handle ports behind firewalls, because we are doing UDP. Knowing who the other guy is connected to, and you are not, you can ask him to initiate a peer connection for the two of you, until you have enough connections that the keep alive works. And once everyone can connect to everyone else by their public username, then we can implement bitmessage.