Still thinking about consensus

docs/blockdag_consensus.md

@@ -1,7 +1,93 @@
 ---
+# katex
 title: Blockdag Consensus
 ---
 
+# The solution outlined
+
+In a decentralized peer to peer network, it is impossible to avoid forks, and
+even with Practical Byzantine Fault Tolerant Consensus and Raft which
+have a vast and ingenious mathematical structure to prove that forks are
+impossible, and with a known and fixed very small number of peers all
+inside a single data centre with very good network connections, they
+wound up furtively allowing forks through the back door, to be resolved
+later, possibly much later, because getting a fork free answer could often
+take a very long time, and the network, though theoretically always live, in
+that it would theoretically deliver a definitive result eventually as long as
+$f+1$ non faulty peers were still at it, “eventually” was sometimes longer
+than people were willing to wait for.
+
+Practical Byzantine Fault Tolerant Consensus is horrendously complicated
+and hard to understand and becomes a lot more complicated when you
+allow forks through the backdoor. Byzantine fault tolerant Raft consensus
+is simpler, but in practice they wind up allowing backdoor forks anyway,
+despite incredibly clever academic ingenuity to produce a system that they
+could prove cannot fork, and when you have forks in the backdoor, Raft
+becomes at least as complicated and difficult to understand as Practical
+Byzantine Fault Tolerant Consensus, perhaps worse.
+
+So, your consensus mechanism must reduce, but cannot eliminate, forks
+and therefore should not try. (There are a bunch of impossibility proofs,
+which any proposed consensus mechanism must thread a very narrow path
+between) and when a fork happens, as it likely very often will, has to
+resolve that fork by everyone moving to the branch of the fork that has the
+most support.
+
+In proof of work consensus, you slow down the production of forks by
+making everyone wade through molasses, and you can tell how much
+support a fork has by how fast it advances through the molasses, so
+everyone moves to the branch of the fork that has made it furthest through
+the molasses.
+
+But this turns out to limit the consensus bandwidth to about ten
+transactions per second, though if the lightning network gets running as it
+should, this may well suffice for quite a bit longer. (Lightning needs fixes
+at the time of writing.)
+
+Making everyone wade through molasses is just a bad idea. And it has
+produced a network that is alarmingly centralized and vulnerable to state
+power. There are very few big miners.
+
+But you don’t want all the peers to report in on which fork they are on,
+because too much data, which is the problem with all the dag consensus
+systems. Proof of work produces a very efficient and compact form of
+evidence of support for a fork.
+
+## The solution, in overly simple outline form
+
+For each peer that could be on the network, including those that have been
+sleeping in a cold wallet for years, each peer keeps a running cumulative
+total of that peers stake. With every new block, the peers stake is added to
+its total for that block.
+
+On each block of the chain, a peer’s rank is the bit position of the highest
+bit of the running total that rolled over, up to a maximum.
+
+So if Bob has a third of the stake of Carol, and $N$ is a rank that corresponds to bit position higher than the stake of either of them, then
+Bob gets to be rank $N$ or higher one third as often as Carol. But even if
+his stake is very low, he gets to be high rank every now and then.
+
+Each peer gets to be rank $N+1$ half as often as he gets to be rank $N$, and he gets to be a rank higher than $N$ as often as he gets to be rank $N$.
+
+Any group of two or more peers can propose a next block, and if lots of
+groups do so, we have a fork. If a group of $m$ peers propose a block, and the
+lowest rank of any member of the group is rank $N$, then the weight of the block is $2^N\ln(8m+9)$ Every correctly behaving peer will copy, circulate, and attempt to build on the highest weighted chain of blocks of which he is aware and will ignore all others. Correctly behaving peers will wait longer the lower their rank before attempting to participate in block formation, will wait longer before participating in attempting to form a low weighted block, and will not attempt to form a new block if a block of which they already have a copy would be higher weight. (Because they would abandon it immediately in favour of a block already formed.)
+
+This produces the same effect as wading through molasses, without the heavy wading, because the chain with the highest ranking peers signing its blocks obviously has more support.
+
+Fork production is limited, because there are not that many high ranking peers, because low ranking peers hold back for higher ranking peers to take care of block formation, and because forks are resolved in favour of the block of highest weight.
+
+In the course of attempting to form a group, peers will find out what other high ranking peers are online, so, if we make the Hedera assumption that everyone always gets through eventually and that everyone knows who is online, there will only be one block formed, and it will be formed by the set of peers that can form the highest ranking block. Of course, that assumption I seriously doubt.
+
+Suppose that two blocks of equal weight are produced. Well, then, we obviously have enough high ranking peers online and active to produce a higher weighted block, and some of them should do so, and if they don’t, chances are that the next block built on one block will have higher weight than the next block built on the other block.
+
+When a peer signs the proposed block, he will attach a sequence number
+to his signature. If a peer encounters two blocks at the same chain
+position, a fork, that have the same peer signing it, (a peer can propose as
+many blocks as it likes) he should discount the signature with the lower
+sequence number, lowering the weight of the block with the lower
+sequence number. If the two signatures have the same sequence number,
+he discounts both signatures, lowering the weight of both blocks.
 # Hedera, Bitcoin Proof of Work, and Paxos
 
 ## Paxos
@@ -25,6 +111,10 @@ out of scope of the algorithm as given in the interests of maximum
 generality, but the algorithm as given is not in fact very general and makes
 no sense and is no use without them.
 
+The algorithm, and algorithms like it and closely related to it do not scale
+to large numbers of peers, but understanding how and why it works is
+useful to understanding how to create an algorithm that will work.
+
 Despite the studious effort to be as generic as possible by omitting all of
 the details required to make it actually do anything useful, the algorithm as
 given is the simplest and most minimal example of the concept,