cheng/wallet
0
0
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

still much to be done on consensus and patricia trees

Browse Source
This commit is contained in:
reaction.la 2022-02-26 22:48:09 +10:00
parent 4e671723fe
commit 93df1f31f2
No known key found for this signature in database
GPG Key ID: 99914792148C8388
2 changed files with 118 additions and 53 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

171
docs/blockdag_consensus.md
View File

@ -3,19 +3,29 @@
title: Blockdag Consensus title: Blockdag Consensus
--- ---
# The solution outlined # The problem
For the reasons discussed, proof of work is incapable of handling a very large number of transactions per second. To replace fiat money, we need a consensus algorithm capable of a thousand times greater consensus bandwidth. There are plenty of consensus algorithms that can handle much higher consensus bandwidth, but they do not scale to large numbers of peers. They are usually implemented with a fixed number of peers, usually three peers, perhaps five, all of which have high reliability connections to each other in a single data centre.
In a decentralized open entry peer to peer network, you are apt to get a very large number of peers, which keep unpredictably appearing and disappearing and frequently have unreliable and slow connections.
Existing proof of stake crypto currencies handle this by "staking" which is in practice under the rug centralization. They are not really a decentralized peer to peer network with open entry.
## The solution outlined
### A manageable number of peers forming consensus
In a decentralized peer to peer network, it is impossible to avoid forks, and In a decentralized peer to peer network, it is impossible to avoid forks, and
even with Practical Byzantine Fault Tolerant Consensus and Raft which even with Practical Byzantine Fault Tolerant Consensus and Raft which
have a vast and ingenious mathematical structure to prove that forks are have a vast, complex, and ingenious mathematical structure to prove that
impossible, and with a known and fixed very small number of peers all forks are impossible, and with a known and fixed very small number of
inside a single data centre with very good network connections, they peers all inside a single data centre with very good network connections,
wound up furtively allowing forks through the back door, to be resolved they wound up "optimizing" the algorithm to furtively allow forks through
later, possibly much later, because getting a fork free answer could often the back door, to be resolved later, possibly much later, because getting a
take a very long time, and the network, though theoretically always live, in fork free answer could sometimes take a very long time, and though the
that it would theoretically deliver a definitive result eventually as long as network was theoretically always live, in that it would theoretically deliver
$f+1$ non faulty peers were still at it, “eventually” was sometimes longer a definitive result eventually as long as $f+1$ non faulty peers were still
than people were willing to wait for. at it, “eventually” was sometimes longer than people were willing to wait.
Practical Byzantine Fault Tolerant Consensus is horrendously complicated Practical Byzantine Fault Tolerant Consensus is horrendously complicated
and hard to understand and becomes a lot more complicated when you and hard to understand and becomes a lot more complicated when you
@ -30,8 +40,9 @@ So, your consensus mechanism must reduce, but cannot eliminate, forks
and therefore should not try. (There are a bunch of impossibility proofs, and therefore should not try. (There are a bunch of impossibility proofs,
which any proposed consensus mechanism must thread a very narrow path which any proposed consensus mechanism must thread a very narrow path
between) and when a fork happens, as it likely very often will, has to between) and when a fork happens, as it likely very often will, has to
resolve that fork by everyone moving to the branch of the fork that has the resolve that fork by everyone eventually moving to the prong of the fork
most support. that has the most support, as happens in Bitcoin proof of work. Which is
why Bitcoin proof of work scales to large numbers of peers.
In proof of work consensus, you slow down the production of forks by In proof of work consensus, you slow down the production of forks by
making everyone wade through molasses, and you can tell how much making everyone wade through molasses, and you can tell how much
@ -50,10 +61,18 @@ power. There are very few big miners.
But you don’t want all the peers to report in on which fork they are on, But you don’t want all the peers to report in on which fork they are on,
because too much data, which is the problem with all the dag consensus because too much data, which is the problem with all the dag consensus
systems. Proof of work produces a very efficient and compact form of systems. They all, like Practical Byzantine Fault Tolerant Consensus and
evidence of support for a fork. Byzantine Fault Tolerant Raft, of which they are variants, rely on on all
the peers telling all the other peers what branch they are on. Proof of work
produces a very efficient and compact form of evidence how much support
there is for a prong of a fork.
## The solution, in overly simple outline form ### Sampling the peers
So we have to sample the peers, or rather have each peer form the same
representative sample. And then we implement something similar to Paxos
and Raft within that small sample. And sometimes peers will disagree about
which sample, resulting in a fork, which has to be resolved.
For each peer that could be on the network, including those that have been For each peer that could be on the network, including those that have been
sleeping in a cold wallet for years, each peer keeps a running cumulative sleeping in a cold wallet for years, each peer keeps a running cumulative
@ -62,38 +81,73 @@ its total.
On each block of the chain, a peer’s rank is the bit position of the highest On each block of the chain, a peer’s rank is the bit position of the highest
bit of the running total that rolled over when its stake was added for that bit of the running total that rolled over when its stake was added for that
block, up to a maximum. (The cumulative total does not have unlimited block.
bits.)
So if Bob has a third of the stake of Carol, and $N$ is a rank that So if Bob has a third of the stake of Carol, and $N$ is a rank that
corresponds to bit position higher than the stake of either of them, then corresponds to bit position higher than the stake of either of them, then
Bob gets to be rank $N$ or higher one third as often as Carol. But even if Bob gets to be rank $R$ or higher one third as often as Carol. But even if
his stake is very low, he gets to be high rank every now and then. his stake is very low, he gets to be high rank every now and then.
The highest ranking peers get to decide, and the likelihood of being a high A small group of the highest ranking peers get to decide on the next block,
ranking peer depends on stake. and the likelihood of being a high ranking peer depends on stake.
Each peer gets to be rank $N+1$ half as often as he gets to be rank $N$, and he gets to be a rank higher than $N$ as often as he gets to be rank $N$. Each peer is going to use a consensus created by those few peers that are
high ranking at this block height of the blockchain, and since there are few
such peers, there will usually only be one such consensus.
Any group of two or more peers can propose a next block, and if lots of Each peer gets to be rank $R+1$ half as often as he gets to be rank $N$, and he gets to be a rank higher than $R$ as often as he gets to be rank $N$.
groups do so, we have a fork. If a group of $m$ peers propose a block, and
the lowest rank of any member of the group is rank $N$, then the rank of
the block is $N$. A higher rank block beats a lower rank block, and if two
blocks are the same rank, the larger group beats the smaller group.
If two blocks have the same rank, their weight is proportional to $m$. The algorithm for producing the next block ensures that if the Hedera assumptions are true, (that everyone of high rank knows what high ranking
peers are online, and everyone always gets through) a single next block
will be produced by the unanimous agreement of the highest ranked peers,
but we don't rely on this always being true. Sometimes, probably often,
there will be multiple next blocks, and entire sequences of blocks. In
which case each peer has to rank them and weight them and choose the
highest ranked and weighted block, or the highest weighted sequence of blocks.
If a peer believes that there are enough peers of its own rank or higher [running average]:./running_average.html
online to form a block with higher rank or the same rank and greater "how to calculate the running average\
weight than any existing block, it attempts to contact them and form such
a block.
The intention is that in the common case, the highest weighted block possible dogs"
will be one that can and must be formed by very few peers. {target="_blank"}
This does not describe how that form consensus. It describes how we get the
problem of forming consensus down to sufficiently few peers that it becomes The rank of a block is rank of the lowest rank peer of the group forming the block, or one plus the [running average] of previous blocks rounded up to the nearest integer, whichever is less.
manageable, and how we manage to keep on going should they fail.
The weight of a block is $m*2^R$, where $m$ is the size of the group
forming the block and $R$ is the rank of the block.
If the number and weight of online peers is stable, there will be two or
more peers higher than the running average rank as often as there are less
than two peers of that rank or higher. So the next block will be typically
be formed by the consensus of two peers of the running average rank,
or close to that.
If a peer is of rank to form a block of higher or equal rank to any existing
block, and believes that there are enough peers online to form a block with
higher rank or the same rank and greater weight than any existing block, it
attempts to contact them and form such a block.
If the group forming a block contains two or more members capable of
forming a block with higher rank (because it also contains members of
lower rank than those members and lower than the running average plus
one), the group is invalid, and if it purportedly forms a block, the block is
invalid.
This rule is to prevent peers from conspiring to manufacture a fork with a falsely high weight under the rule for evaluating forks.
The intention is that in the common case, the highest ranked and weighted
block possible will be one that can and must be formed by very few peers,
so that most of the time, they likely succeed in forming the highest
possible weighted and ranked block. To the extent that the Hedera
assumptions hold true, they will succeed.
This does not describe how they form consensus. It describes how we get
the problem of forming consensus down to sufficiently few peers that it
becomes manageable, and how we deal with a group failing to form
consensus in a reasonable time. (Another group forms, possibly excluding
the worst behaved or least well connected members of the unsuccessful
group)
Unlike Paxos and and Raft, we don't need a consensus process for creating Unlike Paxos and and Raft, we don't need a consensus process for creating
the next block that is guaranteed to succeed eventually, which is important the next block that is guaranteed to succeed eventually, which is important
@ -101,11 +155,19 @@ for if one of the peers has connection problems, or is deliberately trying
to foul things up, "eventually" can take a very long time. Rather, should to foul things up, "eventually" can take a very long time. Rather, should
one group fail, another group will succeed. one group fail, another group will succeed.
Correctly behaving peers will wait longer the lower their rank before attempting to participate in block formation, will wait longer before participating in attempting to form a low weighted block, and will not attempt to form a new block if a block of which they already have a copy would be higher weight. (Because they would abandon it immediately in favour of a block already formed.) Correctly behaving peers will wait longer the lower their rank before attempting to participate in block formation, will wait longer before participating in an attempt to form a low weighted block, and will not attempt to form a new block if a block of which they already have a copy would be higher rank or weight. (Because they would abandon it immediately in favour of a block already formed.)
In the course of attempting to form a group, peers will find out what other high ranking peers are online, so, if we make the Hedera assumption that everyone always gets through eventually and that everyone knows who is online, there will only be one block formed, and it will be formed by the set of peers that can form the highest ranking block. Of course, that assumption I seriously doubt. In the course of attempting to form a group, peers will find out what other high ranking peers are online, so, if the Hedera assumption (that everyone
always gets through eventually and that everyone knows who is online) is
true, or true enough, there will only be one block formed, and it will be
formed by the set of peers that can form the highest ranking block. Of course, the Hedera assumption is not always going to be true..
Suppose that two blocks of equal weight are produced. Well, then, we obviously have enough high ranking peers online and active to produce a higher weighted block, and some of them should do so, and if they don’t, chances are that the next block built on one block will have higher weight than the next block built on the other block. Suppose that two successor blocks of equal weight are produced: Well,
then we have enough high ranking peers online and active to produce a
higher weighted block, and some of them should do so. And if they fail to
do so, then we take the hash of the public keys of the peers forming the
block, their ranks, and the block height of the preceding block, and the
largest hash wins.
When a peer signs the proposed block, he will attach a sequence number When a peer signs the proposed block, he will attach a sequence number
to his signature. If a peer encounters two inconsistent blocks that have the to his signature. If a peer encounters two inconsistent blocks that have the
@ -117,32 +179,35 @@ have the same sequence number, he discounts both signatures, lowering the weight
That is how we resolve two proposed successor blocks of the of the same That is how we resolve two proposed successor blocks of the of the same
blockchain. blockchain.
Fork production is limited, because there are not that many high ranking peers, because low ranking peers hold back for higher ranking peers to take care of block formation, Fork production is limited, because there are not that many high ranking peers, and because low ranking peers hold back for higher ranking peers to take care of block formation,
But we are going to get forks anyway. But we are going to get forks anyway. Not often, but sometimes. I lack
confidence in the Hedera assumptions.
What do we do if we have a fork, and several blocks have been created on What do we do if we have a fork, and several blocks have been created on
each prong of the fork? one or both prongs of the fork?
If for each block on one prong of the fork the group forming the block Then we calculate the weight of the prong:
beats or equals the groups of all of the blocks on the other prong of the
fork, and at least one group wins at least once, that prong wins.
Which is to say, if every group on one prong is of higher or equal rank than any of the groups on the other prong, and at least as numerous as any of the groups of equal rank on the other prong, and at least one group on that prong is higher rank than one group on the other prong, or more numerous than one group of equal rank on the other prong, then that prong wins. $$\displaystyle\sum\limits_{i}2^{R_i}$$
Otherwise, if some win and some lose, then to compare the prongs of a Where $i$ is the peer, and R_i$ is the rank of the last block on that prong of
fork, the weight of a prong of a fork with $k$ blocks is the fork that he has participated in forming.
$$\displaystyle\sum\limits_{i}^k \frac{m_i*2^{N_i}}{k}$$
where $m_i$ is the size of the group that formed that block, and $N_i$ is the
rank of the lowest ranked member of that group.
This value, the weight a prong of the fork, will over time for large deep If two prongs have the same weight, hash all the public keys of the signatories, their ranks, and the block height of the root of the fork and take the prong with the largest hash.
forks approximate the stake of the peers online on that prong, without the
This value, the weight of a prong of the fork, will over time for large deep
forks, approximate the stake of the peers online on that prong, without the
painful cost taking a poll of all peers online, and without the considerable painful cost taking a poll of all peers online, and without the considerable
risk that that poll will be jammed by hostile parties. risk that that poll will be jammed by hostile parties.
Each correctly behaving peer will circulate the highest weighted prong of the fork of which it is aware, ignore all lower weighted prongs, and only attempt to create successor blocks for the highest weighted prong. Each correctly behaving peer will circulate the highest weighted prong of the fork of which it is aware, ignore all lower weighted prongs, and only attempt to create successor blocks for the highest weighted prong.
To add a block to the chain requires a certain minimum time, and the
lower the rank of the group forming that block, the longer that time. If a
peer sees new block that appears to it to be unreasonably and improperly
early, or a fork prong with more blocks than it could have, it will ignore it.
Every correctly behaving peer will copy, circulate, and attempt to build on the highest weighted chain of blocks of which he is aware and will ignore all others. Every correctly behaving peer will copy, circulate, and attempt to build on the highest weighted chain of blocks of which he is aware and will ignore all others.
This produces the same effect as wading through molasses, without the This produces the same effect as wading through molasses, without the

0
docs/running_average.md Normal file
View File