33 KiB
This discussion of the lightning layer may well be obsoleted
by the elegant cryptography of Scriptless Scripts using adaptive Schnorr signatures and of Anonymous Multi-Hop Locks.
Contingent payments can reveal a key to an alien blockchain on the bitcoin blockchain, and zero knowledge contingent payments on the bitcoin chain can reveal any arbitrarily complex secret that fulfils any arbitrarily complex condition.
I need to understand the Anonymous Multi-Hop Locks primitive, and rewrite this accordingly.
Scriptless scripts have the huge advantage of being self enforcing – it is impossible to defect from the script, because there is no script – there are just actions that are cryptographically possible or impossible.
But scriptless scripts cannot in themselves solve the hard problem, that all participants in a multilateral transaction need to know in a short time that the whole multilateral transaction has definitely succeeded or definitely failed. This inherently requires a reliable broadcast channel, though if everyone is cooperating, they don’t have to actually put anything on that channel. But they need the capability to resort to that channel if something funny happens, and that capability has to be used or lost within a time limit.
So, the adapter secret not only has to become known to the participants, it has to become known to the participants within timing limits.
Anonymous Multi-Hop Locks can ensure that the lightning network is always in a definite state, and that those parts of it that are undergoing state change are locked, but need to be embedded in a protocol that ensures that those locks always go away in a short time with commitment of the total transaction, or rollback of the total transaction, and that the participants in the transaction always know whether the transaction was committed or rolled back within a short time. Scriptless scripts are timeless, and need to be embedded in scripted scripts that have timing constraints, and which require information to be broadcast over a reliable broadcast channel if the information is available, yet certain time limits are nonetheless exceeded.
My conclusion was that full circle unbreakability of lightning network transactions within time limits needs a reliable broadcast, and I envisaged a hierarchy of reliable broadcasters, (sidechains, with some sidechains representing a group of bilateral lightning network gateways that act as one multilateral lightning network gateway) But this conclusion may be wrong or overly simple – though we are still going to need sidechains and hierarchical reliable broadcasting, because it can do no end of things that are very difficult otherwise.
But reliable broadcast mechanism both supplies and requires a solution to distributed Byzantine fault tolerant consensus, so the problem of getting a lock up and bringing it down is a general distributed Byzantine fault tolerant consensus problem, and perhaps viewing it as a reliable broadcast problem is a misperception and misanalysis.
Rather, the blockdag requires a mechanism to establish a total order of blocks, and the distributed multihop lock requires a mechanism to establish the state of the lock, the classic distributed state machine problem addressed by Practical Byzantine Fault Tolerant distributed consensus. Albeit implementing Practical Byzantine Fault Tolerant distributed consensus as a state machine over a blockdag may well be a simpler and more humanly intelligible form of this algorithm. But a state machine for BEGIN … COMMIT is going to be a fundamentally different state machine to the one that constructs a total order of transactions.
Lightning layer concept
The lightning layer is a design that implements something like full reserve correspondence banking on top of the blockchain layer, thereby solving the problem of the blockchain exposing too much information about everyone’s transactions to everyone, allowing a blockchain solution to scale to completely replacing fiat currency, and allowing instant on the spot transactions.
A lightning gateway allows instant trustless unilateral transactions (Alice pays Bob) between two or more people, but you have to lock up a significant amount of money in each gateway, and if you have more than two participants, and one of the participants goes off the internet, then the protocols necessary for the gateway to keep going between the remaining participants become a bit more complicated. Multiparty gateways will be (eventually) implemented as sidechains.
A lightning gateway consists of coin (unspent transaction output) on the blockchain that need a joint signature to be spent. The parties to coin construct a transaction splitting it up between them before they create that coin on the blockchain, and then do not drop that transaction to the blockchain layer, keeping the transaction for when one them decides to break up the gateway. To change ownership of the coin, to increase the amount of the coin owned by one of them and decrease the amount owned by the other, they generate a new transaction, and then do not drop that transaction to the blockchain layer either.
And the problem is that we don’t want unilateral transactions. We want Alice pays Bob, and Bob gets a proof that he paid Alice, completing the circle.
And, to work like correspondence banking, we want Alice pays Bob, Bob pays Carol, Carol pays Dan, Dan pays Erin, Erin pays Frank, and Frank gets proof that he paid Ann, completing the circle.
And we want every unilateral transaction in the circle to go through, or none of them to go through. We want the whole circle of unilateral transactions making up a complete multilateral transaction to succeed or every unilateral transaction in the whole circle to fail.
And we want it to normally and usually succeed fast, and if it fails, to take a reasonably short time to fail. And if it fails, we want all participants to know it has definitely failed after a reasonably short timeout.
The Bitcoin lightning layer is not
It is not correspondence banking with predictable behavior enforced by cryptography.
It is correspondence banking that uses cryptography, where the participants trustworthy behaviour is enforced from behind the scenes by unclear authority, which is likely to lead to the existing problem where authority is supposed to protect you from the bankers, but who protects you from authority?
For the lightning network to change from a single consistent state of who owns what part of each gateway unspent transaction output to another single consistent state is equivalent to the well known hard problems of a reliable broadcast channel, the Byzantine generals problem, the two generals problem, and acid transactions on a distributed database.
So when I looked at the lightning documents for a lightning layer on top of bitcoin I expected to see discussion of smart contracts on the bitcoin layer or lightning network protocols on the lightning layer for resolving these hard problems and eventually reaching a resolution, so that the network would always eventually reach a consistent and intended state. Not seeing them.
Instead what I do see is that the lightning network is safe because “misconduct” will result in you losing your bitcoins.
It will? Who do you lose them to? How is misconduct defined? Who decides misconduct? The guy who gets your bitcoins? Do we have a central banker of the lightning network somewhere?
We should not be talking about “misconduct”. We should be talking about the lighting network entering an inconsistent state due to lost messages, nodes going down at inopportune moments, and nodes deviating from the protocol, or entering a state that some of the participants in the transaction did not intend or expect.
The cryptographic term for misconduct is “Byzantine failure”, which indeed normally results from wickedness, but can result from bugs or data corruption.
While the usual and archetypal cause of Byzantine failure is that someone wrote or modified software for the purposes of betrayal, lying, cheating, and stealing, it happens often enough as a result of data corruption or running a program that was compiled under a compiler and in an environment different from that it was tested and developed on.
A gateway does a unilateral transaction between the parties on the lightning layer who control controlling the gateway unspent transaction output by generating a new transaction on the bitcoin layer breaking up the gateway between the parties, and the parties refrain from committing that transaction to the blockchain, and instead endless generate new transactions, which do not get played either, thereby changing what part of the gateway unspent transaction output is owned by each party to the gateway.
What happens if someone commits an out of date bitcoin layer transaction to the blockchain?
Does the central banker confiscate the bitcoin of the party who committed the transaction. How does he do that?
Suppose Alice’s gateway to Bob is blocked because she now owns the whole of that gateway unspent transaction output, and her gateway to Frankie is blocked because Frankie owns all of that gateway.
So she organizes a transaction that moves bitcoin from the Ann/Bob gateway to the Ann/Frankie gateway. So what Alice intends is that Alice pays Bob, Bob pays Carol, Carol pays Dan, Dan pays Erin, Erin pays Frank, and Frank pays Alice. Except that in the middle the transaction Carol and Erin ungracefully disconnect from the network, so that either Ann generated a bitcoin layer transaction giving bitcoin to Bob, but Frankie did not generate a bitcoin layer transaction to Ann, or the other way around.
What happens when a transaction fails and leaves the parties in an inconsistent state?
Does the central banker decide that Carol and Erin were engaged in misconduct and confiscate their bitcoin?*
Trustless Unilateral transactions
If you wanted to buy or sell cryptocurrency for cash or gold, you could arrange over the internet to meet someone in person, set up a lightning gateway with him, and then, when you met him in person, pay him instantly, on the spot, in a trustless unilateral transaction without need to wait some considerable time for the payment to clear.
Suppose you wanted to sell bitcoin for gold, in person, both parties are going to meet, probably with a weapon in their pocket. You could create a jointly signed unspent transaction output, after you and the other party jointly sign a bitcoin layer transaction giving the bitcoin back to you. And then, when meeting, create a jointly signed bitcoin layer transaction giving the bitcoin to the man who is giving you the gold. Except what is going to stop you from committing the earlier transaction to the blockchain a few seconds before the meeting?
OK, let us suppose we have supersedeable transactions. They have to be committed to the blockchain for a time before they take effect, and if someone submits a transaction with the same signatures but a higher priority, the one with lower priority fails to take effect. Then you can endlessly generate fresh transactions, each with higher priority than the previous one, and never commit them to the blockchain unless the gateway between the parties is abandoned.
It would take a little while for the gateway to become available, but once it was available, instant irrevocable payments become possible.
And if one has an account with a service provider over the internet, one could set up a gate way with that service provider, and after each session or each service, make a small instant payment, without the cost and delay of making transactions on the blockchain.
It would be possible to do such instant two party transactions with bitcoin today, although the wallets are generally not set up to support it, nor is the way the main blockchain processes transactions, but with BTC blockchain working as it today, such transactions are not trustless. If you want to do a BTC transaction you are trusting the lightning network, which means you are trusting you do not whom.
At the time this is written, the lightning network for bitcoin has not been adequately implemented, or even fully designed. Existing implementations on top of the bitcoin blockchain still require some trust in intermediaries, and thus require trust in some mysterious authority with the mysterious capability to punish the intermediaries, and to have a real lighting network on bitcoin requires changes in bitcoin which the miners are not going along with, and which perhaps are underscoped and not fully thought out, ad hoc changes to fit with what already existed, and what was politically feasible. And it seems that rather less was politically feasible than one might hope.
Cryptographic implementation of trustless unilateral transactions
Ann and Bob create on the blockchain a coin (unspent transaction output, utxo) whose corresponding key is the sum of secret key known only to Ann, and a secret key known only to Bob. Before they create this coin, they create a transaction on it, signed by their joint key, that creates two coins, one with a secret key known only to Bob, and one with a secret key known only to Ann. They keep this transaction to themselves and do not place it on the blockchain.
Multilateral (circle) transactions
The lightning layer will function like correspondence banking, only with good behavior cryptographically enforced on the lightning vertices, rather than by state supervision of the “banks”. This will require a blockchain layer designed to support it.
Correspondence banking merges large numbers of small transactions into a small number of large pooled transactions which are eventually settled on the blockchain in one big transaction, with several parties to the transaction representing a very large number of parties engaged in a very large number of transactions.
But correspondence banking works by trust, thus the intermediaries have to be well known and subject to pressure, which is apt to mean subject to government pressure – and government has interests that are in conflict with those of people attempting to use a medium of exchange and a store of value.
lightning network correspondence banking
merges many small two party transactions into a few large transactions merges many small two party transactions into a few large transactions lightning vertex bank lightning gateway bank account trustless need to trust the banks instant Slow in that you can never be sure if a transaction will be honored for some time Payer id visible to payer vertex, payee id visible to payee vertex Government issued id of payer and payee visible to all intermediaries.
We have a separate blockchain of supersedeable transactions. A transaction gets committed to the primary blockchain after it has been sitting on the supersedable chain for a while, if, at the time it is evaluated for commitment, all of its inputs are valid unspent outputs on the primary blockchain, and none of them are inputs to a currently valid higher priority transaction on the blockchain of supersedable transactions, the priority being an arbitrary length binary value.
If one party tries to break up a gateway with an out of date distribution, the other party notices it sitting on the supersedable blockchain, and issues a more up to date transaction. Normally this should never happen, since when one of the parties wants to break up a gateway, the other party should agree to a regular transaction. However, one party may go offline and stay offline, especially if the last transaction reduced the the value of their interest in the gateway to zero.
A gateway in the lightning network layer is a jointly signed coin on the blockchain on the blockchain layer. Bob’s account with BigCrypto is a coin on the blockchain layer, for which there exists a a jointly signed transaction distributing that block between two blocks, one wholly owned by Bob, and one wholly owned by BigCrypto, thus giving BigCrypto effective offchain control of one part of that block, and Bob effective control of the other part of the block, and by generating new superseding transaction, they can effectively transfer ownership of part of the block instantly without anything going on the blockchain.
But …
Acid
We have to make sure that transactions are acid on the lightning network as a whole, that transactions are atomic, consistent, isolated, and durable.
Atomic and consistent
Ann has an account with Bob, Bob has an account with Carol.
To change what is Ann’s account requires a transaction signed by Bob and Ann, and similarly for Carol.
Ann wants to pay Carol, but does not want to sign a reduction in her account with Bob, unless she is sure that Carol gets the corresponding increase. Bob does not want to sign an increase in Carol’s account, unless it gets Ann’s signature on a decrease in her account. Not to mention that Ann probably does not want to sign a decrease on her account without getting a receipt from Carol. Full circle transaction. We need to guarantee that either the full circle goes through, or none of the separate unilateral transactions in the circle go through.
Reliable broadcast channel
The solution to atomicity and maintaining consistency between different entities on the lightning network is the reliable broadcast channel.
Such as the blockchain itself. Create a special zero value transaction that has no outputs and carries its own signature, but can be a required input to other transactions, and whose construction requires the cooperation of all the parties. Each gateway constructs a transaction only be valid if a code is placed on the blockchain that requires the cooperation of all the gateways within a short time. Once the code exists, and everyone knows it exists, they proceed with bilateral transactions that do not require the code and only potentially go on the blockchain. If not everyone knows it exists, and it does not appear on the blockchain within a short time, then the transaction fails. If everyone knows it exists, the transaction succeeds. If not everyone knows it exists, but it appears on the blockchain within the time limit, the transaction succeeds, and each party could potentially play the transaction, and thus effectively owns the corresponding part of the gateway coin, regardless of whether they play it or not.
A reliable broadcast channel is something that somehow works like a classified ad did back in the days of ink on paper newspapers. The physical process of producing the newspaper guaranteed that every single copy had the exact same classified ad in it, and that ad must have been made public on a certain date. Easy to do this with a printing press that puts ink on paper. Very hard to do this, with electronic point to point communications.
But let us assume we somehow have a reliable broadcast channel:
All the parties agree on a Merkle tree, which binds them if the joint signature to that Merkle tree appears on the reliable broadcast channel within a certain short time period.
And, if some of them have the joint signature, then knowing that they could upload it to the reliable broadcast channel, they each agree to superseding unilateral transactions. If Bob expects payment from Ann and expects to make payment to Carol, and he has the joint signature, and knows Carol has a copy of the authenticated joint signature, because Carol sent him the signature and he sent Ann the signature, of it, then he knows Carol can make him pay her, and knows he can make Ann pay him. So he just goes right ahead with unilateral transactions that supersede the transaction that relies on the reliable broadcast channel. And if every party to the transaction does that, none of them actually broadcast the signature the reliable broadcast channel. Which in consequence, by merely being available enforces correct behaviour, and is seldom likely to need to actually broadcast anything. And when something is actually broadcast on that channel, chances are that all the transactions that that broadcast enables will have been superseded.
Each party, when receives a copy of the joint signature that he could upload to the reliable broadcast channel, sends a copy to the counter party that he expects to pay him, and each party, when he receives a copy from the party he expects to pay, performs the unilateral payment to that party that supersedes and the transaction using the reliable broadcast network.
And if a party has a copy of the joint signature and the document that it signs for the full circle transaction, but finds himself unable to perform the superseding unilateral transactions with his counterparties, (perhaps their internet connection or their computer went down) then he uploads the signature to the reliable broadcast channel.
When the signature is uploaded to reliable broadcast channel, this does not give the reliable broadcast channel any substantial information about the amount of the transaction, and who the parties to the transaction are, but the node of the channel sees IP addresses, and this could frequently be used to reconstruct a pretty good guess about who is transacting with whom and why. As we see with Monaro, a partial information leak can be put together with lots of other sources of information to reconstruct a very large information leak.
But most of the time, the channel is not likely to be used, which means it will have only small fragments of data, not enough to put together to form a meaningful picture, hence the privacy leak is unlikely to be very useful to those snooping on other people’s business.
Other use cases for a reliable broadcast channel
The use case of joint signatures implies an immutable data structure of the tuple oid, hash, public key, and two scalars.
But another use case is to publicly root private immutable data.
If you continually upload the latest version, you wind up uploading most of tree, or all of it, which does not add significantly to the cost of each interaction recorded. The simplest sql friendly data structure is (oid of this item, public key, hash, your index of hash, oids of two child hashes) with the reliable broadcast channel checking that the child hashes do in fact generate the hash, and that the tuple (public key, index of hash) is unique.
If the data is aged out after say, three months, cannot directly check uniqueness and correctness for the nodes that are the roots of big trees. How do you know someone has not made up several different and mutually inconsistent pasts for immutable append only data structure associated with this key?
To work around this problem, allow unbalanced Merkle trees, consisting of (oid of this item, public key, hash, your tree node index, index of the highest index leaf governed by this hash, oids of two child hashes) If an unbalanced node referencing an old tree root is uploaded at intervals of less than three months, it can be used to prove the contents and uniqueness of the old balanced binary tree root, since the most recent unbalanced node must have also proved contents and uniqueness, using a less recent unbalanced node, and unbalanced nodes must also be unique on the tuple (public key, index of node, index of highest leaf node governed) Someone can forget his old past, and, after three months, start making up a new past, but the witness for the new past can only begin on the day he starts the upload. He cannot construct several realities, and six months later, choose which reality he finds convenient. Or rather he can, but he cannot provide a six month earlier witness to it.
You upload several nodes that constitute the unbalanced tree right side path that points at the balanced rest of the old tree every two months, superseding the previous right hand side of the unbalanced tree, and thus maintaining a chain of proofs stretching into the past that proves that throughout this period, there is one and only one immutable data structure associated with this public key.
A common case is that they key certifying the state of the immutable data may change, with the torch being passed from key to key. In which case the upload of total state needs to reference the used key, and an earlier, normally the earliest, signing key, with links in the chain of keys authorizing keys being renewed at less than the timeout interval for data to be immutable, but unavailable from the reliable broadcast network. If the witness asserts that key is authorized by a chain of keys going back to an earlier or the earliest keys, then it relies on its previous witness, rather than re-evaluating the entire, possibly very long, chain of keys every time.
But, if the witness cannot do that, then the entire, possibly very very long, chain of keys and signatures has be uploaded for the witness to record the earlier, or earliest key, as authorizing the current ever changing key. Similarly if the client starts uploading to a new witness. But such very long proofs will only be have to done once in a very long while, and done once for everyone.
Because of Byzantine failure or network failure, such a chain may fork. The protocol has to be such that if a fork develops by network failure, it will be fixed, with one of the forks dying when the network functions better, and if it fails by Byzantine failure, we get two sets of reliable broadcast channels, each testifying that the other reliable broadcast channel is unreliable, and each testifying that a different fork is the valid fork, and which fork you follow depends on which reliable broadcast channel you subscribe to.
Another use case is for wallet recovery, with mutable data structures encrypted by the private key whose primary key is the public key.
implementing and funding a reliable broadcast channel
Tardigrade has a somewhat similar architecture to the proposed Reliable
Broadcast network charges $120 per year for per TB of storage, $45 per
terabyte of download. So for uploading a single signature, and downloading
it six times, which one hash, one elliptic point, and two scalars, one
hundred and twenty eight bytes, so the cost of doing what tardigrade does
with reliable broadcast network operated by a single operator would be
4 × 10^{-7} dollars. Which might as well be free, except we have to charge some tiny amount to prevent DDoS.
But, when the system is operating at scale, will want the reliable broadcast network to have many operators, who synchronize with each other so that the data is available to each of them and all of them and from each of them and all of them, and can testify when the data became available, so the cost will be many times that. Which is still insignificant. If the network is composed of a hundred operators, and there are substantial overheads to maintain synchrony and truth, we are still only looking a cost of $0.0001 per transaction. Maybe we should charge for opening the account, and then every hundredth transaction.
We also want the operators to be genuinely independent and separate from each other. We don’t want a single inherently authorized reliable broadcast channel, because it is inherently a low cost target for the fifty one percent attack. I have been thinking about implementing a reliable broadcast channel as byzantine Paxos protocol, but this gives us a massive low cost fifty one percent attack vulnerability. If the reliable broadcast channel is cheap enough to be useful, it is cheap enough for the fifty one percent attack. We want cheap testimony of valuable facts, which makes consensus mechanisms unlikely to work.
A better way reliably implementing a reliable broadcast channel is as a network of trusted witnesses, each of which keeps an eye on the reliability of other witnesses, because each periodically uploads the unbalanced tree testifying to its total state on several of the others, and each makes it data reciprocally available to several of the others, and each monitors the availability of several of the others, and each provides a signed chain of its total state with each witness, or some witnesses. Because the data is reciprocally available, each can testify to the uptime and reliability of each of the others, and none can hide its state. Each makes each of the nodes in each of its balanced trees available by index, the top node in each of its unbalanced right hand tree is signed, and nodes in the unbalanced tree constitute a separate sql like table, which times out considerably faster than the nodes with a single index.
Downtime, failure to provide oids on request, and losing its recent state, will be detected and will whack its reputation.
And it is up to the individual human to decide which trusted witness to follow, which human decision roots all the automatic computer decisions.
isolated
Isolated means that one transaction cannot mangle another transaction. Easiest way to do this is that one gateway of the lightning network cannot handle a transaction while another is pending. Which is going to be a problem when we have nodes handling a lot of traffic, but a more efficient fix can wait till we have traffic problems.
For a more efficient fix, we will need relative as well as absolute transactions. A relative transaction, on being appended to an absolute transaction, subtracts something from one of the outputs, and adds it to another outputs. If this would produce a negative output, the compound transaction is invalid. But this can wait until we have busy gateways.
durable
Durable means that if something bad happens to your node, you can probably recover, and if you don’t recover, no one else has problems.
If one party to a gateway goes down and does not come up, possibly because he has lost the secret key to the gateway, the other party drops the most recent transaction of the lightning layer to the blockchain layer.
If one party to a gateway goes down, but eventually comes up again, possibly with lost data, they resynchronize.
Suppose you lose your wallet?
At present the standard way of recovering a wallet is that you create a wallet using your BIPS master password, and it scans the blockchain for coins whose public key corresponds to the secret keys it might have generated.
Which works to recover your money, but does not recover your metadata, and will not recover a lightning gateway, since the your public key is not on the blockchain, rather the sum of your public key, and the public key of other parties to the gateway.
Metadata is becoming more important. If you lose your Wasabi wallet metadata, you have a big problem, and if we use the wallet as the basis for private end to end encrypted messages that can carry money, and for uncensorable social networking, the wallet will have a lot of very valuable metadata.
We really need cloud backup of wallets, with the wallet encrypted by a secret key derived from its master key, and cloud backup paid for though a lightning gateway. And this needs to be built into the wallet, the way recovery by scanning the blockchain is built into wallets today.
trustless, fast, and free from blockchain analysis
That full circle transactions are atomic and consistent means that the lightning network can operate without trust or reputation – completely pseudonymous parties with no reputation can become nodes in the network, making it very hard for the state to apply know your customer pressure.
If the full content of each gateway’s ownership never becomes known except to the parties to the gateway, then the effect is to mingle and merge coins on the blockchain, creating fungibility and untraceability, as well as instant transactions.
If, on the other hand, we had a rather small number of trusted nodes that have a special, central, and privileged position on the lightning network, this would recreate the traceability problem of correspondence banking. The lightning network has to trust the reliable broadcast channels with reliability and broadcasting, but the information on that channel is meaningless to anyone other than the parties to the transaction.