wallet/docs/design/peer_socket.md

---
# katex
title: >-
	Peer Socket
sidebar: false
notmine: false
...

::: myabstract
[abstract:]{.bigbold}
Most things follow the client server model,
so it makes sense to have a distinction between server sockets
and client sockets.  But ultimately what we are doing is
passing messages between entities and the revolutionary
and subversive technologies, bittorrent, bitcoin, and
bitmessage are peer to peer, so it makes sense that all sockets,
however created wind up with same properties.
:::

# factoring

In order to pass messages, the socket has to know a whole lot of state.  And
in order handle messages, the entity handling the messages has to know a
whole lot of state.  So a socket api is an answer to the question how we
factor this big pile of state into two smaller piles of state.

Each big bundle of state represents a concurrent communicating process.
Some of the state of this concurrent communicating process is on one side
of our socket division, and is transparent to one side of our division. The
application knows the internals of the some of the state, but the internals
of socket state are opaque, while the socket knows the internals of the
socket state, but the internals of the application state are opaque to it.
The socket state machines think that they are passing messages of one class
or a very small number of classes, to one big state machine, which messages
contain an opaque block of bytes that application class serializes and
deserializes.

## layer responsibilities

The sockets layer just sends and receives arbitrary size blocks
of opaque bytes over the wire between two machines. 
They can be sent with or without flow control
and with or without reliability,
but if the block is too big to fit in this connection's maximum
packet size, the without flow control and without
reliability option is ignored.  Flow control and reliability is
always applied to messages too big to fit in a packet.

The despatch layer parses out the in-reply-to and the
in-regards-to values from the opaque block of bytes and despatches them
to the appropriate application layer state machine, which parses out
the message type field, deserializes the message,
and despatches it to the appropriate fully typed event handler
of that  state machine.


# Representing concurrent communicating processes

A message may contain a reply-to field and or an in-regards-to field.

The recipient must have associated a handler, consisting of a
call back and an opaque pointer to the state of the concurrent process
on the recipient with the messages referenced by at least one of
these fields. In the event of conflicting values, the reply-to takes
precedence, but the callback of the reply-to has access to both its
data structure, and the in-regards-to dat structure, a pointer to which
is normally in its state.  The in-regards-to being the state machine,
and the in-reply-to the event that modifies the
state of the state machine.

When we initialize a connection, we establish a state machine
at both ends, both the application factor of the state machine,
and the socket factor of the state machine.

But a single state machine at the application level could be
handling several connections, and a single connection could have
several state machines running independently, and the
socket code should not need to care.

Further, these concurrent communicating processes are going to
be sending messages to each other on the same machine.
We need to model Go's goroutines.

A goroutine is a function, and functions always terminate --
and in Go are unceremoniously and abruptly ended when their parent
function ends, because they are variables inside its dataspace,
as are their channels.
And, in Go, a channel is typically passed by the parent to its children,
though they can also be passed in a channel.
Obviously this structure is impossible and inapplicable
when processes may live, and usually do live,
in different machines.

The equivalent of Go channel is not a connection.  Rather,
one sends a message to the other to request it create a state machine,
which will be the in-regards-to message, and the equivalent of a
Go channel is a message type, the in-regards-to message id,
and the connection id.  Which we pack into a single class so that we
can use it the way Go uses channels.

The sockets layer (or another state machine on the application layer)
calls the callback routine with the message and the state.

The sockets layer treats the application layer as one big state
machine, and the information it sends up to the application
enables the application layer to despatch the event to the
correct factor of that state machine, which we have factored into
as many very small, and preferably stateless, state machines as possible.

We factor the potentially ginormous state machine into
many small state machines, in the same style as Go factors a potentially
ginormous Goroutine into many small goroutines.

The socket code being a state machine composed of many
small state machines, which communicates with the application layer
over a very small number of channels,
these channels containing blocks of bytes that are
opaque to the socket code,
but are serialized and deserialized by the application layer code.
From the point of view of the application layer code,
it is many state machines,
and the socket layer is one big state machine.
From the point of view of the socket code, it is many state machines,
and the application layer is one big state machine.
The application code, parsing the the in-reply-to message id,
and the in-regard-to message id, figures out where to send
the opaque block of bytes, and the recipient deserializes,
and sends it to a routine that acts on an object of that
deserialized class.

Since the sockets layer does not know the internals of the message struct, the message has 
to be serialized and deserialized into the corresponding class by the application layer.

Or perhaps the callback routine deserializes the object into a particular class, and then calls
a routine for that class, but another state machine on the application layer would call the 
class specific routine directly. The equivalent of Go channel between one state machine on the
application layer and another in the same application layer is directly calling what
the class specific routine that the callback routine would call.

The state machine terminates when its job is done,
freeing up any allocated memory,
but the connection endures for the life of the program,
and most of the data about a connection endures in
an sql database between reboots.

Because we can have many state machines on a connection,
most of our state machines can have very little state,
typically an infinite receive loop, an infinite send receive loop,
or an infinite receive send loop, which have no state at all,
are stateless.  We factorize the state machine into many state machines
to keep each one manageable.

Go code tends to consist of many concurrent processes
continually being spawned by a master concurrent process,
and themselves spawning more concurrent processes.
For most state machines, we do not need recursion,
so it is reasonable for their state to be a fixed allocation
inside the state of their master concurrent process.
In the unlikely event we do need recursion
 we usually only have one instance running at one time,
so we can allocate an `std::stack` in the master concurrent process.

And what if we do want to spawn many in parallel?
Well, they will usually be stateless.
What if they are not not stateless?
Well that would require an `std::vector` of states.
And if we need many running in parallel with recursion,
an `std::vector` with each element containing an `std::stack`.
And to avoid costly reallocations, we create the `std::vector`
and the `std::vector`s underlying the `std::stack`s with
realistic initial allocations that are only infrequently exceeded.

# flow control and reliability

If we want to transmit a big pile of data, a big message, well,
this is the hard problem, for the sender has to throttle according
to the recipient's readiness to handle it and the physical connections capability to transmit it.

Quic is a UDP protocol that provides flow control, and the obvious thing
to handle bulk data transfer is to fork it to use Zooko based keys.

[Tailscale]:https://tailscale.com/blog/how-nat-traversal-works
"How to communicate peer-to-peer through NAT firewalls"{target="_blank"}

[Tailscale] has solved a problem very similar to the one I am trying to solve, albeit their solutions rely on a central human authority, and they recommend:

> If you’re reaching for TCP because you want a
> stream‑oriented connection when the NAT traversal is done,
> consider using QUIC instead. It builds on top of UDP,
> so we can focus on UDP for NAT traversal and still have a
> nice stream protocol at the end.

But to interface QUIC to a system capable of handling a massive
number of state machines, going to need something like Tokio,
because we want the thread to service other state machines while
QUIC is stalling the output or waiting for input.  Indeed, no
matter what, if we stall in the socket layer rather than the
application layer, which makes life a whole lot easier for the
application programmer, going to need something like Tokio.

On the application side, we have to lock each state machine
when it is active.  It can only handle one message at at time.
So the despatch layer has to queue up messages and stash them somewhere,
and if it has too many messages stashed,
it wants to push back on the state machine at the application layer
at the other end of the wire.  So the despatch layer at the receiving end
has to from time to time tell the despatch layer at the sending end
"I have `n` bytes in regard to message 'Y', and can receive `m` more.
And when the despatch layer at the other end, which unlike the socket
layer knows which state machine is communicating with which,
has more than that amount of data to send, it then blocks
and locks the state machine at its end in its send operation.

The socket layer does not know about that and does not worry about that.
What it worries about packets getting lost on the wire, and caches
piling up in the middle of the wire.
It adds to each message a send time and a receive time
and if the despatch layer wants to send data faster
than it thinks is suitable, it has to push back on the despatch layer.
Which it does in the same style.
It tells it the connection can handle up to `m` further bytes.
Or we might have two despatch layers, one for sending and one for
receiving, with the send state machine sending events to the receive state
machine, but not vice versa, in which case the socket layer
*can* block the send layer.

# Tokio

Most of this machinery seems like a re-implementation of Tokio-rust,
which is a huge project.  I don't wanna learn Tokio-rust, but equally
I don't want to re-invent the wheel.

# Minimal system

Prototype.  Limit global bandwidth at the application
state machine level -- they adjust their policy according to how much
data is moving, and they spread the non response outgoing
messages out to a constant rate (constant per counterparty,
and uniformly interleaved.)

Single threaded, hence no state machine locking.

Tweet style limit on the size of messages, hence no fragmentation
and re-assembly issue.  Our socket layer becomes trivial - it just
send blobs like a zeromq socket.

If you are trying to download a sackload of data, you request a counterparty to send a certain amount to you at a given rate, he immediately responds (without regard to global bandwidth limits) with the first instalment, and a promise of further instalments at a certain time)

Each instalment records how much has been sent, and when, when the next instalment is coming, and the schedule for further instalments.

If you miss an instalment, you nack it after a delay.  If he receives
a nack, he replaces the promised instalments with the missing ones.

The first thing we implement is everyone sharing a list of who they have successfully connected to, in recency order, and everyone keeps everyone else's list, which catastrophically fails to scale, and also how up to date their counter parties are with their own list, so that they do not have
endlessly resend data (unless the counterparty has a catastrophic loss of data, and requests everything from the beginning.)

We assume everyone has an open port, which is sucks intolerably, but once that is working we can handle ports behind firewalls, because we are doing UDP.  Knowing who the other guy is connected to, and you are not, you can ask him to initiate a peer connection for the two of you, until you have
enough connections that the keep alive works.

And once everyone can connect to everyone else by their public username, then we can implement bitmessage.