still much to be done on consensus and patricia trees

docs/blockdag_consensus.md

@@ -3,19 +3,29 @@
 title: Blockdag Consensus
 ---
 
-# The solution outlined
+# The problem
+
+For the reasons discussed, proof of work is incapable of handling a very large number of transactions per second.  To replace fiat money, we need a consensus algorithm capable of a thousand times greater consensus bandwidth. There are plenty of consensus algorithms that can handle much higher consensus bandwidth, but they do not scale to large numbers of peers.  They are usually implemented with a fixed number of peers, usually three peers, perhaps five, all of which have high reliability connections to each other in a single data centre.
+
+In a decentralized open entry peer to peer network, you are apt to get a very large number of peers, which keep unpredictably appearing and disappearing and frequently have unreliable and slow connections.
+
+Existing proof of stake crypto currencies handle this by "staking" which is in practice under the rug centralization.  They are not really a decentralized peer to peer network with open entry.
+
+## The solution outlined
+
+### A manageable number of peers forming consensus
 
 In a decentralized peer to peer network, it is impossible to avoid forks, and
 even with Practical Byzantine Fault Tolerant Consensus and Raft which
-have a vast and ingenious mathematical structure to prove that forks are
-impossible, and with a known and fixed very small number of peers all
-inside a single data centre with very good network connections, they
-wound up furtively allowing forks through the back door, to be resolved
-later, possibly much later, because getting a fork free answer could often
-take a very long time, and the network, though theoretically always live, in
-that it would theoretically deliver a definitive result eventually as long as
-$f+1$ non faulty peers were still at it, “eventually” was sometimes longer
-than people were willing to wait for.
+have a vast, complex, and ingenious mathematical structure to prove that
+forks are impossible, and with a known and fixed very small number of
+peers all inside a single data centre with very good network connections,
+they wound up "optimizing" the algorithm to furtively allow forks through
+the back door, to be resolved later, possibly much later, because getting a
+fork free answer could sometimes take a very long time, and though the
+network was theoretically always live, in that it would theoretically deliver
+a definitive result eventually as long as $f+1$ non faulty peers were still
+at it, “eventually” was sometimes longer than people were willing  to wait.
 
 Practical Byzantine Fault Tolerant Consensus is horrendously complicated
 and hard to understand and becomes a lot more complicated when you
@@ -30,8 +40,9 @@ So, your consensus mechanism must reduce, but cannot eliminate, forks
 and therefore should not try. (There are a bunch of impossibility proofs,
 which any proposed consensus mechanism must thread a very narrow path
 between) and when a fork happens, as it likely very often will, has to
-resolve that fork by everyone moving to the branch of the fork that has the
-most support.
+resolve that fork by everyone eventually moving to the prong of the fork
+that has the most support, as happens in Bitcoin proof of work.  Which is
+why Bitcoin proof of work scales to large numbers of peers.
 
 In proof of work consensus, you slow down the production of forks by
 making everyone wade through molasses, and you can tell how much
@@ -50,10 +61,18 @@ power. There are very few big miners.
 
 But you don’t want all the peers to report in on which fork they are on,
 because too much data, which is the problem with all the dag consensus
-systems. Proof of work produces a very efficient and compact form of
-evidence of support for a fork.
+systems.  They all, like Practical Byzantine Fault Tolerant Consensus and
+Byzantine Fault Tolerant Raft, of which they are variants, rely on on all
+the peers telling all the other peers what branch they are on.  Proof of work
+produces a very efficient and compact form of evidence how much support
+there is for a prong of a fork.
 
-## The solution, in overly simple outline form
+### Sampling the peers
+
+So we have to sample the peers, or rather have each peer form the same
+representative sample.  And then we implement something similar to Paxos
+and Raft within that small sample.  And sometimes peers will disagree about
+which sample, resulting in a fork, which has to be resolved.
 
 For each peer that could be on the network, including those that have been
 sleeping in a cold wallet for years, each peer keeps a running cumulative
@@ -62,38 +81,73 @@ its total.
 
 On each block of the chain, a peer’s rank is the bit position of the highest
 bit of the running total that rolled over when its stake was added for that
-block, up to a maximum. (The cumulative total does not have unlimited
-bits.)
+block.  
 
 So if Bob has a third of the stake of Carol, and $N$ is a rank that
 corresponds to bit position higher than the stake of either of them, then
-Bob gets to be rank $N$ or higher one third as often as Carol. But even if
+Bob gets to be rank $R$ or higher one third as often as Carol. But even if
 his stake is very low, he gets to be high rank every now and then.
 
-The highest ranking peers get to decide, and the likelihood of being a high
-ranking peer depends on stake.
+A small group of the highest ranking peers get to decide on the next block,
+and the likelihood of being a high ranking peer depends on stake.
 
-Each peer gets to be rank $N+1$ half as often as he gets to be rank $N$, and he gets to be a rank higher than $N$ as often as he gets to be rank $N$.
+Each peer is going to use a consensus created by those few peers that are
+high ranking at this block height of the blockchain, and since there are few
+such peers, there will usually only be one such consensus.
 
-Any group of two or more peers can propose a next block, and if lots of
-groups do so, we have a fork. If a group of $m$ peers propose a block, and
-the lowest rank of any member of the group is rank $N$, then the rank of
-the block is $N$. A higher rank block beats a lower rank block, and if two
-blocks are the same rank, the larger group beats the smaller group.
+Each peer gets to be rank $R+1$ half as often as he gets to be rank $N$, and he gets to be a rank higher than $R$ as often as he gets to be rank $N$.
 
-If two blocks have the same rank, their weight is proportional to $m$.
+The algorithm for producing the next block ensures that if the Hedera assumptions are true, (that everyone of high rank knows what high ranking
+peers are online, and everyone always gets through) a single next block
+will be produced by the unanimous agreement of the highest ranked peers,
+but we don't rely on this always being true. Sometimes, probably often,
+there will be multiple next blocks, and entire sequences of blocks. In
+which case each peer  has to rank them and weight them and choose the
+highest ranked and weighted block, or the highest weighted sequence of blocks.
 
-If a peer believes that there are enough peers of its own rank or higher
-online to form a block with higher rank or the same rank and greater
-weight than any existing block, it  attempts to contact them and form such
-a block.
+[running average]:./running_average.html
+"how to calculate the running average\
 
-The intention is that in the common case, the highest weighted block possible
-will be one that can and must be formed by very few peers.
+dogs"
+{target="_blank"}
 
-This does not describe how that form consensus.  It describes how we get the
-problem of forming consensus down to sufficiently few peers that it becomes
-manageable, and how we manage to keep on going should they fail.
+
+The rank of a block is rank of the lowest rank peer of the group forming the block, or one plus the [running average] of previous blocks rounded up to the nearest integer, whichever is less.
+
+The weight of a block is $m*2^R$, where $m$ is the size of the group
+forming the block and $R$ is the rank of the block.
+
+If the number and weight of online peers is stable, there will be two or
+more peers higher than the running average rank as often as there are less
+than two peers of that rank or higher.  So the next block will be typically
+be formed by the consensus of two peers of the running average rank,
+or close to that.
+
+If a peer is of rank to form a block of higher or equal rank to any existing
+block, and believes that there are enough peers online to form a block with
+higher rank or the same rank and greater weight than any existing block, it
+attempts to contact them and form such a block.
+
+If the group forming a block contains two or more members capable of
+forming a block with higher rank (because it also contains members of
+lower rank than those members and lower than the running average plus
+one), the group is invalid, and if it purportedly forms a block, the block is
+invalid.
+
+This rule is to prevent peers from conspiring to  manufacture a fork with a falsely high weight under the rule for evaluating forks.
+
+The intention is that in the common case, the highest ranked and weighted
+block possible will be one that can and must be formed by very few peers,
+so that most of the time, they likely succeed in forming the highest
+possible weighted and ranked block.  To the extent that the Hedera
+assumptions hold true, they will succeed.
+
+This does not describe how they form consensus.  It describes how we get
+the problem of forming consensus down to sufficiently few peers that it
+becomes manageable, and how we deal with a group failing to form
+consensus in a reasonable time.  (Another group forms, possibly excluding
+the worst behaved or least well connected members of the unsuccessful
+group)
 
 Unlike Paxos and and Raft, we don't need a consensus process for creating
 the next block that is guaranteed to succeed eventually, which is important
@@ -101,11 +155,19 @@ for if one of the peers has connection problems, or is deliberately trying
 to foul things up, "eventually" can take a very long time.  Rather, should
 one group fail, another group will succeed.
 
-Correctly behaving peers will wait longer the lower their rank before attempting to participate in block formation, will wait longer before participating in attempting to form a low weighted block, and will not attempt to form a new block if a block of which they already have a copy would be higher weight. (Because they would abandon it immediately in favour of a block already formed.)
+Correctly behaving peers will wait longer the lower their rank before attempting to participate in block formation, will wait longer before participating in an attempt to form a low weighted block, and will not attempt to form a new block if a block of which they already have a copy would be higher rank or weight. (Because they would abandon it immediately in favour of a block already formed.)
 
-In the course of attempting to form a group, peers will find out what other high ranking peers are online, so, if we make the Hedera assumption that everyone always gets through eventually and that everyone knows who is online, there will only be one block formed, and it will be formed by the set of peers that can form the highest ranking block. Of course, that assumption I seriously doubt.
+In the course of attempting to form a group, peers will find out what other high ranking peers are online, so, if the Hedera assumption (that everyone
+always gets through eventually and that everyone knows who is online) is
+true, or true enough, there will only be one block formed, and it will be
+formed by the set of peers that can form the highest ranking block. Of course, the Hedera assumption is not always going to be true..
 
-Suppose that two blocks of equal weight are produced. Well, then, we obviously have enough high ranking peers online and active to produce a higher weighted block, and some of them should do so, and if they don’t, chances are that the next block built on one block will have higher weight than the next block built on the other block.
+Suppose that two successor blocks of equal weight are produced: Well,
+then we have enough high ranking peers online and active to produce a
+higher weighted block, and some of them should do so.  And if they fail to
+do so, then we take the hash of the public keys of the peers forming the
+block, their ranks, and the block height of the preceding block, and the
+largest hash wins.
 
 When a peer signs the proposed block, he will attach a sequence number
 to his signature. If a peer encounters two inconsistent blocks that have the
@@ -117,32 +179,35 @@ have the same sequence number, he discounts both signatures, lowering the weight
 That is how we resolve two proposed successor blocks of the of the same
 blockchain.
 
-Fork production is limited, because there are not that many high ranking peers, because low ranking peers hold back for higher ranking peers to take care of block formation,
+Fork production is limited, because there are not that many high ranking peers, and because low ranking peers hold back for higher ranking peers to take care of block formation,
 
-But we are going to get forks anyway.
+But we are going to get forks anyway.  Not often, but sometimes.  I lack
+confidence in the Hedera assumptions.
 
 What do we do if we have a fork, and several blocks have been created on
-each prong of the fork?
+one or both prongs of the fork?
 
- If for each block on one prong of the fork the group forming the block
-beats or equals the groups of all of the blocks on the other prong of the
-fork, and at least one group wins at least once, that prong wins.
+Then we calculate the weight of the prong:
 
-Which is to say, if every group on one prong is of higher or equal rank than any of the groups on the other prong, and at least as numerous as any of the groups of equal rank on the other prong, and at least one group on that prong is higher rank than one group on the other prong, or more numerous than one group of equal rank on the other prong, then that prong wins.
+$$\displaystyle\sum\limits_{i}2^{R_i}$$
 
-Otherwise, if some win  and some lose, then to compare the prongs of a
-fork, the weight of a prong of a fork with $k$ blocks is
-$$\displaystyle\sum\limits_{i}^k \frac{m_i*2^{N_i}}{k}$$
-where $m_i$ is the size of the group that formed that block, and $N_i$ is the
-rank of the lowest ranked member of that group.
+Where $i$ is the peer, and R_i$ is the rank of the last block on that prong of
+the fork that he has participated in forming.
 
-This value, the weight a prong of the fork, will over time for large deep
-forks approximate the stake of the peers online on that prong, without the
+If two prongs have the same weight, hash all the public keys of the signatories, their ranks, and the block height of the root of the fork and take the prong with the largest hash.
+
+This value, the weight of a prong of the fork, will over time for large deep
+forks, approximate the stake of the peers online on that prong, without the
 painful cost taking a poll of all peers online, and without the considerable
 risk that that poll will be jammed by hostile parties.
 
 Each correctly behaving peer will circulate the highest weighted prong of the fork of which it is aware, ignore all lower weighted prongs, and only attempt to create successor blocks for the highest weighted prong.
 
+To add a block to the chain requires a certain minimum time, and the
+lower the rank of the group forming that block, the longer that time.  If a
+peer sees new block that appears to it to be unreasonably and improperly
+early, or a fork prong with more blocks than it could have, it will ignore it.
+
 Every correctly behaving peer will copy, circulate, and attempt to build on the highest weighted chain of blocks of which he is aware and will ignore all others.
 
 This produces the same effect as wading through molasses, without the