From 4e671723fef93f7c1cb5463b995cc2f2d1ecfe2b Mon Sep 17 00:00:00 2001 From: "reaction.la" Date: Tue, 22 Feb 2022 16:56:03 +1000 Subject: [PATCH] Finally completed consensus algorithm design Also extensively modified the process of generating html files from markdown files --- LICENSE.html | 39 +---- NOTICE.html | 39 +---- README.html | 22 +-- RELEASE_NOTES.html | 39 +---- docs/blockdag_consensus.md | 154 +++++++++++------- docs/contributor_code_of_conduct.html | 21 +-- docs/cyperhpunk_manifesto.md | 2 +- .../cpp_automatic_memory_management.md | 6 +- docs/libraries/pandoc_templates/after.pandoc | 1 - docs/libraries/pandoc_templates/before.pandoc | 1 - docs/libraries/pandoc_templates/header.pandoc | 20 --- docs/libraries/pandoc_templates/style.css | 32 +--- docs/mkdocs.sh | 12 +- docs/pandoc_templates/beforedotdot.pandoc | 1 + docs/pandoc_templates/header.pandoc | 20 --- docs/pandoc_templates/icon.pandoc | 1 + docs/pandoc_templates/icondotdot.pandoc | 1 + docs/pandoc_templates/style.css | 17 +- pandoc_templates/style.css | 30 +--- 19 files changed, 154 insertions(+), 304 deletions(-) delete mode 100644 docs/libraries/pandoc_templates/after.pandoc delete mode 100644 docs/libraries/pandoc_templates/before.pandoc delete mode 100644 docs/libraries/pandoc_templates/header.pandoc create mode 100644 docs/pandoc_templates/beforedotdot.pandoc delete mode 100644 docs/pandoc_templates/header.pandoc create mode 100644 docs/pandoc_templates/icon.pandoc create mode 100644 docs/pandoc_templates/icondotdot.pandoc diff --git a/LICENSE.html b/LICENSE.html index 33f7174..3f48f5e 100644 --- a/LICENSE.html +++ b/LICENSE.html @@ -6,39 +6,18 @@ LICENSE - + - - -
diff --git a/NOTICE.html b/NOTICE.html index 4af90c4..9b5f972 100644 --- a/NOTICE.html +++ b/NOTICE.html @@ -6,39 +6,18 @@ NOTICE - + - - -
diff --git a/README.html b/README.html index 2bbaceb..8f8076a 100644 --- a/README.html +++ b/README.html @@ -77,30 +77,10 @@ code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */ .display.math{display: block; text-align: center; margin: 0.5rem auto;} - + - -
diff --git a/RELEASE_NOTES.html b/RELEASE_NOTES.html index 239beb0..b2d31eb 100644 --- a/RELEASE_NOTES.html +++ b/RELEASE_NOTES.html @@ -6,39 +6,18 @@ Release Notes - + - - -
diff --git a/docs/blockdag_consensus.md b/docs/blockdag_consensus.md index 34ae9ba..bc4ca20 100644 --- a/docs/blockdag_consensus.md +++ b/docs/blockdag_consensus.md @@ -58,36 +58,97 @@ evidence of support for a fork. For each peer that could be on the network, including those that have been sleeping in a cold wallet for years, each peer keeps a running cumulative total of that peers stake. With every new block, the peers stake is added to -its total for that block. +its total. On each block of the chain, a peer’s rank is the bit position of the highest -bit of the running total that rolled over, up to a maximum. +bit of the running total that rolled over when its stake was added for that +block, up to a maximum. (The cumulative total does not have unlimited +bits.) -So if Bob has a third of the stake of Carol, and $N$ is a rank that corresponds to bit position higher than the stake of either of them, then +So if Bob has a third of the stake of Carol, and $N$ is a rank that +corresponds to bit position higher than the stake of either of them, then Bob gets to be rank $N$ or higher one third as often as Carol. But even if his stake is very low, he gets to be high rank every now and then. +The highest ranking peers get to decide, and the likelihood of being a high +ranking peer depends on stake. + Each peer gets to be rank $N+1$ half as often as he gets to be rank $N$, and he gets to be a rank higher than $N$ as often as he gets to be rank $N$. Any group of two or more peers can propose a next block, and if lots of -groups do so, we have a fork. If a group of $m$ peers propose a block, and the -lowest rank of any member of the group is rank $N$, then the weight of the block is $2^N\ln(8m+9)$ Every correctly behaving peer will copy, circulate, and attempt to build on the highest weighted chain of blocks of which he is aware and will ignore all others. Correctly behaving peers will wait longer the lower their rank before attempting to participate in block formation, will wait longer before participating in attempting to form a low weighted block, and will not attempt to form a new block if a block of which they already have a copy would be higher weight. (Because they would abandon it immediately in favour of a block already formed.) +groups do so, we have a fork. If a group of $m$ peers propose a block, and +the lowest rank of any member of the group is rank $N$, then the rank of +the block is $N$. A higher rank block beats a lower rank block, and if two +blocks are the same rank, the larger group beats the smaller group. -This produces the same effect as wading through molasses, without the heavy wading, because the chain with the highest ranking peers signing its blocks obviously has more support. +If two blocks have the same rank, their weight is proportional to $m$. -Fork production is limited, because there are not that many high ranking peers, because low ranking peers hold back for higher ranking peers to take care of block formation, and because forks are resolved in favour of the block of highest weight. +If a peer believes that there are enough peers of its own rank or higher +online to form a block with higher rank or the same rank and greater +weight than any existing block, it attempts to contact them and form such +a block. + +The intention is that in the common case, the highest weighted block possible +will be one that can and must be formed by very few peers. + +This does not describe how that form consensus. It describes how we get the +problem of forming consensus down to sufficiently few peers that it becomes +manageable, and how we manage to keep on going should they fail. + +Unlike Paxos and and Raft, we don't need a consensus process for creating +the next block that is guaranteed to succeed eventually, which is important +for if one of the peers has connection problems, or is deliberately trying +to foul things up, "eventually" can take a very long time. Rather, should +one group fail, another group will succeed. + +Correctly behaving peers will wait longer the lower their rank before attempting to participate in block formation, will wait longer before participating in attempting to form a low weighted block, and will not attempt to form a new block if a block of which they already have a copy would be higher weight. (Because they would abandon it immediately in favour of a block already formed.) In the course of attempting to form a group, peers will find out what other high ranking peers are online, so, if we make the Hedera assumption that everyone always gets through eventually and that everyone knows who is online, there will only be one block formed, and it will be formed by the set of peers that can form the highest ranking block. Of course, that assumption I seriously doubt. Suppose that two blocks of equal weight are produced. Well, then, we obviously have enough high ranking peers online and active to produce a higher weighted block, and some of them should do so, and if they don’t, chances are that the next block built on one block will have higher weight than the next block built on the other block. When a peer signs the proposed block, he will attach a sequence number -to his signature. If a peer encounters two blocks at the same chain -position, a fork, that have the same peer signing it, (a peer can propose as -many blocks as it likes) he should discount the signature with the lower -sequence number, lowering the weight of the block with the lower -sequence number. If the two signatures have the same sequence number, -he discounts both signatures, lowering the weight of both blocks. +to his signature. If a peer encounters two inconsistent blocks that have the +same peer signing it, (a peer can propose as many blocks as it likes) he +should discount the signature with the lower sequence number, lowering the +weight of the block with the lower sequence number. If the two signatures +have the same sequence number, he discounts both signatures, lowering the weight of both blocks. + +That is how we resolve two proposed successor blocks of the of the same +blockchain. + +Fork production is limited, because there are not that many high ranking peers, because low ranking peers hold back for higher ranking peers to take care of block formation, + +But we are going to get forks anyway. + +What do we do if we have a fork, and several blocks have been created on +each prong of the fork? + + If for each block on one prong of the fork the group forming the block +beats or equals the groups of all of the blocks on the other prong of the +fork, and at least one group wins at least once, that prong wins. + +Which is to say, if every group on one prong is of higher or equal rank than any of the groups on the other prong, and at least as numerous as any of the groups of equal rank on the other prong, and at least one group on that prong is higher rank than one group on the other prong, or more numerous than one group of equal rank on the other prong, then that prong wins. + +Otherwise, if some win and some lose, then to compare the prongs of a +fork, the weight of a prong of a fork with $k$ blocks is +$$\displaystyle\sum\limits_{i}^k \frac{m_i*2^{N_i}}{k}$$ +where $m_i$ is the size of the group that formed that block, and $N_i$ is the +rank of the lowest ranked member of that group. + +This value, the weight a prong of the fork, will over time for large deep +forks approximate the stake of the peers online on that prong, without the +painful cost taking a poll of all peers online, and without the considerable +risk that that poll will be jammed by hostile parties. + +Each correctly behaving peer will circulate the highest weighted prong of the fork of which it is aware, ignore all lower weighted prongs, and only attempt to create successor blocks for the highest weighted prong. + +Every correctly behaving peer will copy, circulate, and attempt to build on the highest weighted chain of blocks of which he is aware and will ignore all others. + +This produces the same effect as wading through molasses, without the +heavy wading, because the chain with the most numerous and highest +ranking peers signing its blocks obviously has more support, just as in +Bitcoin, more wading through molasses indicates more support. # Hedera, Bitcoin Proof of Work, and Paxos ## Paxos @@ -218,6 +279,8 @@ long time before it can report a definite and final consensus, while Bitcoin takes a fairly definite time to report it is nearly certain about the consensus value and that value is unlikely is unlikely to change. +Further, in actual practice, particularly during a view change, Paxos and Raft are frequently in a state where a peer knows that one view is overwhelmingly likely to become final, and another view highly unlikely, but the state takes a while to finalize. And the client is waiting. So it winds up reporting a provisional result to the client. + # Bitcoin does not scale to competing with fiat currency Bitcoin is limited to ten transactions per second. Credit card networks @@ -484,27 +547,26 @@ limit, see: themselves can commit transactions through the peers, if the clients themselves hold the secret keys and do not need to trust the peers. -# Proposed blockdag implementation +# Calculating the stake of a peer -The specific details of many of these proposed systems are rather silly and -often vague, typical academic exercises unconcerned with real world -issues, but the general idea that the academics intend to illustrate is sound -and should work, certainly can be made to work. They need to be -understood as academic illustrations of the idea of the general algorithm -for fast and massive blockdag consensus, and not necessarily intended as -ready to roll implementations of that idea. - -Here is an even more vague outline of my variant of this idea, I name -Yabca “Yet another blockdag consensus algorithm”, +We intend that peers will hold no valuable or lasting secrets, that all the +value and the power will be in client wallets, and the client wallets with +most of the value, who should have most of the power, will seldom be online. I propose proof of stake. The stake of a peer is not the stake it owns, but the stake that it has injected into the blockchain on behalf of its clients -and that its clients have not spent yet. Each peer pays on behalf of its -clients for the amount of space it takes up on the blockchain, though it does -not pay in each block. It makes an advance payment that will cover many -transactions in many blocks. The money disappears, built in deflation, -instead of built in inflation. Each block is a record of what a peer has -injected +and that its clients have not spent yet, or stake that some client wallet +somewhere has chosen to be represented by that peer. Likely only the +whales will make a deliberate and conscious decision to have their stake +represented by a peer, and it will be a peer that they likely control, or that +someone they have some relationship with controls, but not necessarily a +peer that they use for transactions. + +Each peer pays on behalf of its clients for the amount of space it takes up +on the blockchain, though it does not pay in each block. It makes an +advance payment that will cover many transactions in many blocks. The +money disappears, built in deflation, instead of built in inflation. Each +block is a record of what a peer has injected The system does not pay the peers for generating a total order of transactions. Clients pay peers for injecting transactions. We want the @@ -512,24 +574,6 @@ power to be in the hands of people who own the money, thus governance will have a built in bias towards appreciation and deflation, rather than inflation. -The special sauce that makes each proposed blockdag different from each -of the others is how each peer decides what consensus is forming about -the leftmost edge of the dag, the graph analysis that each peer performs. -And this, my special sauce, I will explain when I have something running. - -Each peer adopts as its leftmost child for its latest block, a previous block -that looks like a good candidate for consensus, which looks like a good -candidate for consensus because the left child has a left child that looks -like consensus actually is forming around that grandchild , in part because -the left child has a … left child has a … left child that looks like it might -have consensus, until eventually, as new blocks pile on top of old blocks, we -actually do get consensus about the left most child sufficiently deep in -the dag from the latest blocks. - -The blockdag can run fast because all the forks that are continually -forming eventually get stuffed into the consensus total order somewhere. -So we don’t have to impose a speed limit to prevent excessive forking. - # Cost of storage on the blockchain. Tardigrade charges $120 per year for per terabyte of storage, $45 per terabyte of download @@ -540,13 +584,13 @@ Storage on the blockdag is going to cost more, because massively replicated, so say three hundred times as much, and is going to be optimized for tiny fragments of data while Tardigrade is optimized for enormous blocks of data, so say three times as much on top of that, a -thousand times as expensive to store should be in the right ballpark. +thousand times as expensive to store should be in the right ballpark. Maybe ten thousand. When you download, you are downloading from only a single peer on the blockdag, but you are downloading tiny fragments dispersed over a large pile of data, so again, a thousand times as expensive to download sounds like it might be in the right ballpark. - Then storing a chain of keys and the accompanying roots of total state, - with one new key per day for ten years will cost about two dollars over ten - years. +Then storing a chain of keys and the accompanying roots of total state, +with one new key per day for ten years will cost about two dollars over ten +years. Ten megabytes is a pretty big pile of human readable documentation. Let us suppose you want to store ten megabytes of human readable data and @@ -559,8 +603,4 @@ and, when executed, cause a change in mutable total state, typically that a new unspent coin record is added, and an old unspent coin record is deleted. -In another use, a valid update to a chain of signatures should cause a -change in the signature associated with a name, the association being -mutable state controlled by immutable data. Thus we can implement -corporations on the blockdag by a chain of signatures, each of which -represents [an n of m multisig](./PracticalLargeScaleDistributedKeyGeneration.pdf “Practical Large Scale Distributed Key Generation”). +A thousand times as expensive turns out to be quite economical. \ No newline at end of file diff --git a/docs/contributor_code_of_conduct.html b/docs/contributor_code_of_conduct.html index 4891844..8e20400 100644 --- a/docs/contributor_code_of_conduct.html +++ b/docs/contributor_code_of_conduct.html @@ -14,29 +14,10 @@ ul.task-list{list-style: none;} .display.math{display: block; text-align: center; margin: 0.5rem auto;} - + - diff --git a/docs/cyperhpunk_manifesto.md b/docs/cyperhpunk_manifesto.md index 26dee4f..9ef58c9 100644 --- a/docs/cyperhpunk_manifesto.md +++ b/docs/cyperhpunk_manifesto.md @@ -1,10 +1,10 @@ --- +# notmine description: >- “A Cypherpunk’s Manifesto” was written by Eric Hughes and published on March 9, 1993. robots: 'index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1' title: >- Eric Hughes: A Cypherpunk’s Manifesto -viewport: 'width=device-width, initial-scale=1.0' --- **Privacy is necessary for an open society in the electronic age. Privacy is not secrecy. A private matter is something one doesn’t want the whole world to know, but a secret matter is something one doesn’t want anybody to know. Privacy is the power to selectively reveal oneself to the world.** diff --git a/docs/libraries/cpp_automatic_memory_management.md b/docs/libraries/cpp_automatic_memory_management.md index 9c1f80e..134cd8f 100644 --- a/docs/libraries/cpp_automatic_memory_management.md +++ b/docs/libraries/cpp_automatic_memory_management.md @@ -5,7 +5,6 @@ title: # Memory Safety Modern, mostly memory safe C++, is enforced by:\ -- gsl - Microsoft safety checker - Guidelines - language checker @@ -16,9 +15,8 @@ the project gets bigger, some will slip through. static_assert(__cplusplus >= 201703, "C version of out of date"); -The gsl adds span for pointer arithmetic, where the -size of the array pointed to is kept with the pointer for safe iteration and -bounds checking during pointer maths. This should be available in the standard template library with C20. +Adds the std::span type, which makes pointer handling a whole lot simpler and safer. +The size of the array pointed to is kept with the pointer for safe iteration and bounds checking during pointer maths. Also, translates std::array and old type C arrays to the same type, which makes life much simpler and safer. You get all the new good range stuff from both of them. Modern C++ as handles arrays as arrays where possible, but they quickly decay to pointers – which you avoid using spans. std::array is a C array diff --git a/docs/libraries/pandoc_templates/after.pandoc b/docs/libraries/pandoc_templates/after.pandoc deleted file mode 100644 index c902203..0000000 --- a/docs/libraries/pandoc_templates/after.pandoc +++ /dev/null @@ -1 +0,0 @@ -

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

diff --git a/docs/libraries/pandoc_templates/before.pandoc b/docs/libraries/pandoc_templates/before.pandoc deleted file mode 100644 index ccec396..0000000 --- a/docs/libraries/pandoc_templates/before.pandoc +++ /dev/null @@ -1 +0,0 @@ -

To Home page

diff --git a/docs/libraries/pandoc_templates/header.pandoc b/docs/libraries/pandoc_templates/header.pandoc deleted file mode 100644 index 30bad69..0000000 --- a/docs/libraries/pandoc_templates/header.pandoc +++ /dev/null @@ -1,20 +0,0 @@ - - diff --git a/docs/libraries/pandoc_templates/style.css b/docs/libraries/pandoc_templates/style.css index d66feba..759f9ef 100644 --- a/docs/libraries/pandoc_templates/style.css +++ b/docs/libraries/pandoc_templates/style.css @@ -1,31 +1 @@ -body { - max-width: 30em; - margin-left: 1em; - } -p.center {text-align:center; - } - -table { - border-collapse: collapse; - } -td, th { - border: 1px solid #999; - padding: 0.5rem; - text-align: left; - } -code{white-space: pre-wrap; - } -span.smallcaps{font-variant: small-caps; - } -span.underline{text-decoration: underline; - } -div.column{display: inline-block; vertical-align: top; width: 50%; - } -div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em; - } -ul.task-list{list-style: none; - } -.display.math{display: block; text-align: center; margin: 0.5rem auto; - } -h1.title{text-align: center; font-size: xxx-large; - } \ No newline at end of file +../pandoc_templates/style.css \ No newline at end of file diff --git a/docs/mkdocs.sh b/docs/mkdocs.sh index 7327332..fb98630 100644 --- a/docs/mkdocs.sh +++ b/docs/mkdocs.sh @@ -11,8 +11,8 @@ elif [[ "$OSTYPE" == "cygwin" ]]; then elif [[ "$OSTYPE" == "msys" ]]; then osoptions="--fail-if-warnings --eol=lf " fi -templates="./pandoc_templates/" -options=$osoptions"--toc -N --toc-depth=5 --wrap=preserve --metadata=lang:en --include-in-header=$templates/header.pandoc --include-before-body=$templates/before.pandoc --css=$templates/style.css -o" +templates="./pandoc_templates" +options=$osoptions"--toc -N --toc-depth=5 --wrap=preserve --metadata=lang:en --include-in-header=$templates/icon.pandoc --include-before-body=$templates/before.pandoc --css=$templates/style.css -o" for f in *.md do len=${#f} @@ -41,6 +41,8 @@ if [[ $line =~ notmine$ ]]; fi done cd libraries +templates="../pandoc_templates" +options=$osoptions"--toc -N --toc-depth=5 --wrap=preserve --metadata=lang:en --include-in-header=$templates/icondotdot.pandoc --include-before-body=$templates/beforedotdot.pandoc --css=$templates/style.css --include-after-body=$templates/after.pandoc -o" for f in *.md do len=${#f} @@ -56,21 +58,21 @@ do katex=" --katex=./" fi done <$f - pandoc $katex $mine $options $base.html $base.md + pandoc $katex $options $base.html $base.md echo "$base.html from $f" #else # echo " $base.html up to date" fi done cd ../.. -templates=docs/pandoc_templates/ +templates=docs/pandoc_templates for f in *.md do len=${#f} base=${f:0:($len-3)} if [ $f -nt $base.html ]; then - pandoc $osoptions --wrap=preserve --from markdown --to html --metadata=lang:en --include-in-header=$templates/header.pandoc --css=$templates/style.css -o $base.html $base.md + pandoc $osoptions --wrap=preserve --from markdown --to html --metadata=lang:en --standalone --css=$templates/style.css -o $base.html $base.md echo "$base.html from $f" #else # echo " $base.html up to date" diff --git a/docs/pandoc_templates/beforedotdot.pandoc b/docs/pandoc_templates/beforedotdot.pandoc new file mode 100644 index 0000000..3509826 --- /dev/null +++ b/docs/pandoc_templates/beforedotdot.pandoc @@ -0,0 +1 @@ +

To Home page

diff --git a/docs/pandoc_templates/header.pandoc b/docs/pandoc_templates/header.pandoc deleted file mode 100644 index aceb5b1..0000000 --- a/docs/pandoc_templates/header.pandoc +++ /dev/null @@ -1,20 +0,0 @@ - - diff --git a/docs/pandoc_templates/icon.pandoc b/docs/pandoc_templates/icon.pandoc new file mode 100644 index 0000000..f43d0b9 --- /dev/null +++ b/docs/pandoc_templates/icon.pandoc @@ -0,0 +1 @@ + diff --git a/docs/pandoc_templates/icondotdot.pandoc b/docs/pandoc_templates/icondotdot.pandoc new file mode 100644 index 0000000..85695be --- /dev/null +++ b/docs/pandoc_templates/icondotdot.pandoc @@ -0,0 +1 @@ + diff --git a/docs/pandoc_templates/style.css b/docs/pandoc_templates/style.css index ce7d496..95fdd89 100644 --- a/docs/pandoc_templates/style.css +++ b/docs/pandoc_templates/style.css @@ -2,9 +2,6 @@ body { max-width: 30em; margin-left: 1em; } -p.center {text-align:center; - } - table { border-collapse: collapse; } @@ -28,4 +25,16 @@ ul.task-list{list-style: none; .display.math{display: block; text-align: center; margin: 0.5rem auto; } h1.title{text-align: center; font-size: xxx-large; -} \ No newline at end of file + } +div.center {text-align:center; + } +div.centre {text-align:center; + } +table { + border-collapse: collapse; + } +td, th { + border: 1px solid #999; + padding: 0.5rem; + text-align: left; + } diff --git a/pandoc_templates/style.css b/pandoc_templates/style.css index 8276cac..fe553b1 100644 --- a/pandoc_templates/style.css +++ b/pandoc_templates/style.css @@ -1,29 +1 @@ -body { - max-width: 30em; - margin-left: 1em; - } -p.center {text-align:center; - } - -table { - border-collapse: collapse; - } -td, th { - border: 1px solid #999; - padding: 0.5rem; - text-align: left; - } -code{white-space: pre-wrap; - } -span.smallcaps{font-variant: small-caps; - } -span.underline{text-decoration: underline; - } -div.column{display: inline-block; vertical-align: top; width: 50%; - } -div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em; - } -ul.task-list{list-style: none; - } -.display.math{display: block; text-align: center; margin: 0.5rem auto; - } \ No newline at end of file +../docs/pandoc_templates/style.css \ No newline at end of file