BIP101 implementation to be made available for altcoins
Posted: Tue Jan 26, 2016 5:47 pm
Given that I have made significant progress on implementing BIP101/4 in litecoin, I thought that it would be interesting to talk about the impressions I received while doing this work. Then, I'll announce that the BIP101 solution will be made available to scrypt-based altcoins along with documentation. Finally, I'll talk about the implications of what I've learned so far for the feasibility of different solutions to the blocksize problem.
First, here's an update on what I actually have accomplished. Last night, I was able to modify the existing unit tests to work with BIP101/4. I had already gotten the implementation itself to compile earlier, but I was further able to finish resolving the compiler errors in the test code yesterday. Then, I triple-checked to ensure that all of Mike Hearn's code was correctly translated. I made sure that everything that needed to be included was there, and that other things that did not need to be included were omitted. Tonight will see the beginning of actually running the tests to figure out what doesn't work. Since it is the end of the month and it's time for "use or lose" vacation days, I'll have 4 days this weekend to devote to the task. We keep time at our business to ensure that it is being spent wisely, and my timesheet records about 36 hours spent so far.
While it may not sound like a great accomplishment to resolve compiler errors, I've come to appreciate the power of C++ over weakly-typed languages like PHP or Python, the latter of which seems to be spreading everywhere recently. I know already that when I run this code, I will not encouter subtle errors caused by conversions of numbers from Strings to Integers. With the exception of Litecoin's 3000+ "main.cpp" file, true inherited classes in C++ mean that code starts the testing period at a higher level of correctness than does Python code. I had been imagining C++ from when I last used it during a class in 2000, and the libraries available to C++ developers have changed significantly since then. Anyone who hasn't used C++ for a long time should definitely look at it more closely for their next projects.
I've taken a "minimalist" approach to this implementation - that is, I included the absolute minimum number of changes necessary to implement BIP101/4 and to test it, without anything else modified. If Litecoin were exactly like bitcoin, that means that I only kept about half the changes from Andresen's pull requests. In reality, Litecoin differs significantly from bitcoin because there are many code refactorings and features that were pulled into bitcoin but which never made it to Litecoin. Coins Andresen's unit tests rely on code that apparently had been added to bitcoin after Litecoin forked from it. Therefore, I had to move some of the dependent code for these tests over to Litecoin to have a high level of confidence that the implementation works as designed. However, I strove to reduce the chance for bugs by changing only what was needed. In some instances (like adding one header file), that means that there is a small deviation from convention. I thought it better to have a small deviation than to include large refactoring that could introduce bugs.
One of the reasons I chose this minimalist design is because coinaday, the developer of Nyancoins, stated that he was interested in using this code for his network. After reading his statement and having learned more about the Litecoin codebase, I decided to expand the scope of this project somewhat. The goal is not only to produce a working and tested implementation of BIP101/4 for Litecoin, but also to generalize this implementation so that it can be merged into Litecoin-derived altcoins with minimal effort. This is possible because the differences between bitcoin and litecoin are far greater than those between litecoin and almost all altcoins. To that end, I have also started writing in-depth documentation that describes the reason for the changes to the files, and also makes it as easy as possible to modify the parameters like initial blocksize and doubling period. I will also make myself available to answer questions on the implementation.
Examination of the source trees of other coins shows that there are more coins derived from Litecoin and its derivations than any other coin, except for Bitcoin. As stated before, I believe the SHA-256 miners are one of the most culpable parties responsible for the current impasse in Bitcoin, and I also believe that BIP101 is the best solution to this problem. By encouraging the spread of BIP101 to as many scrypt-based altcoins like Dogecoin, Novacoin, and Gamecredit as possible, I hope that altcoins will start to become vastly more capable than Bitcoin very rapidly. At the same time, implementation of variations of BIP101 on many networks will prove to the cryptocurrency industry that adopting a simple, permanent solution is possible. Finally, when implementation of BIP101 with parameters set no higher than the technological limit fails to destroy coins, that will make a statement that the Bitcoin Core developers are holding back progress unnecessarily while other networks are quickly leaving them behind. They will be forced to innovate or the movement of value into the more capable networks will be accelerated. In either case, cryptocurrency will advance more quickly than it would were every network constrained - which accomplishes our main goal of keeping ourselves and others in business despite the Core developers' indifference to companies' long-term viability.
However, one of the most critical observations I've made about this implementation is the simple experience that it is very difficult. Understanding the codebase alone requires hours of reading. There are users on the Internet who are claiming that it is easy to write working code, and therefore an alternate implementation like Segregated Witness or the Lightning Network will work just as well. It won't. There's a reason that Segregated Witness has no release candidates despite having been talked about for months; it's because testing network code is extremely difficult. There's a reason why the Lightning Network hasn't been implemented despite years of talk; it's because an entire new system of software needs to be developed to interact with it just to prove that it works. Both of these, along with the Bitpay "adaptive" solution and "miner voting," require significant software upgrades.
That's why Bitcoin Classic is the only (unlikely) solution up for consideration on Bitcoin. It is essentially a version of BIP101 that stops partway, and it requires no significant upgrades to external software. The requirement that all the existing software be upgraded to support the other solutions means that they may be dead on arrival, just like ipV6 has been languishing for years.
In conclusion, software is hard to write, and even harder to test. The opportunity cost I'm losing by writing this code is enormous, because open source development is not a paid job. My experience with modifying the litecoin codebase solidifies my opinion that solutions that require changes to how transactions are created or stored will never be adopted in any existing coin. The end solution to bitcoin, if the parties involved ever reach one at all, will be a variation on BIP101. If Segregated Witness is adopted, most people simply won't encode transactions with it, and the promised "60% increase" will probably turn out to be 10%. If the Lightning Network is pushed out, most people will use the simpler solution of altcoins. If Litecoin adopts a solution with miner voting, the blocksize will never increase because most miners simply won't vote. There is no malice involved, just a recognition that the risk and cost of modifying already complex transaction creation code (to each individual user) is too great to justify the minor benefits that mostly accrue to other people.
I hope that developers of altcoins will continue to follow us as this implementation nears completion. We plan to release detailed documentation and assist anyone who needs help to encourage widespread adoption of BIP101 variants. I'll make another post on Monday with an update about what we accomplished over the weekend.
First, here's an update on what I actually have accomplished. Last night, I was able to modify the existing unit tests to work with BIP101/4. I had already gotten the implementation itself to compile earlier, but I was further able to finish resolving the compiler errors in the test code yesterday. Then, I triple-checked to ensure that all of Mike Hearn's code was correctly translated. I made sure that everything that needed to be included was there, and that other things that did not need to be included were omitted. Tonight will see the beginning of actually running the tests to figure out what doesn't work. Since it is the end of the month and it's time for "use or lose" vacation days, I'll have 4 days this weekend to devote to the task. We keep time at our business to ensure that it is being spent wisely, and my timesheet records about 36 hours spent so far.
While it may not sound like a great accomplishment to resolve compiler errors, I've come to appreciate the power of C++ over weakly-typed languages like PHP or Python, the latter of which seems to be spreading everywhere recently. I know already that when I run this code, I will not encouter subtle errors caused by conversions of numbers from Strings to Integers. With the exception of Litecoin's 3000+ "main.cpp" file, true inherited classes in C++ mean that code starts the testing period at a higher level of correctness than does Python code. I had been imagining C++ from when I last used it during a class in 2000, and the libraries available to C++ developers have changed significantly since then. Anyone who hasn't used C++ for a long time should definitely look at it more closely for their next projects.
I've taken a "minimalist" approach to this implementation - that is, I included the absolute minimum number of changes necessary to implement BIP101/4 and to test it, without anything else modified. If Litecoin were exactly like bitcoin, that means that I only kept about half the changes from Andresen's pull requests. In reality, Litecoin differs significantly from bitcoin because there are many code refactorings and features that were pulled into bitcoin but which never made it to Litecoin. Coins Andresen's unit tests rely on code that apparently had been added to bitcoin after Litecoin forked from it. Therefore, I had to move some of the dependent code for these tests over to Litecoin to have a high level of confidence that the implementation works as designed. However, I strove to reduce the chance for bugs by changing only what was needed. In some instances (like adding one header file), that means that there is a small deviation from convention. I thought it better to have a small deviation than to include large refactoring that could introduce bugs.
One of the reasons I chose this minimalist design is because coinaday, the developer of Nyancoins, stated that he was interested in using this code for his network. After reading his statement and having learned more about the Litecoin codebase, I decided to expand the scope of this project somewhat. The goal is not only to produce a working and tested implementation of BIP101/4 for Litecoin, but also to generalize this implementation so that it can be merged into Litecoin-derived altcoins with minimal effort. This is possible because the differences between bitcoin and litecoin are far greater than those between litecoin and almost all altcoins. To that end, I have also started writing in-depth documentation that describes the reason for the changes to the files, and also makes it as easy as possible to modify the parameters like initial blocksize and doubling period. I will also make myself available to answer questions on the implementation.
Examination of the source trees of other coins shows that there are more coins derived from Litecoin and its derivations than any other coin, except for Bitcoin. As stated before, I believe the SHA-256 miners are one of the most culpable parties responsible for the current impasse in Bitcoin, and I also believe that BIP101 is the best solution to this problem. By encouraging the spread of BIP101 to as many scrypt-based altcoins like Dogecoin, Novacoin, and Gamecredit as possible, I hope that altcoins will start to become vastly more capable than Bitcoin very rapidly. At the same time, implementation of variations of BIP101 on many networks will prove to the cryptocurrency industry that adopting a simple, permanent solution is possible. Finally, when implementation of BIP101 with parameters set no higher than the technological limit fails to destroy coins, that will make a statement that the Bitcoin Core developers are holding back progress unnecessarily while other networks are quickly leaving them behind. They will be forced to innovate or the movement of value into the more capable networks will be accelerated. In either case, cryptocurrency will advance more quickly than it would were every network constrained - which accomplishes our main goal of keeping ourselves and others in business despite the Core developers' indifference to companies' long-term viability.
However, one of the most critical observations I've made about this implementation is the simple experience that it is very difficult. Understanding the codebase alone requires hours of reading. There are users on the Internet who are claiming that it is easy to write working code, and therefore an alternate implementation like Segregated Witness or the Lightning Network will work just as well. It won't. There's a reason that Segregated Witness has no release candidates despite having been talked about for months; it's because testing network code is extremely difficult. There's a reason why the Lightning Network hasn't been implemented despite years of talk; it's because an entire new system of software needs to be developed to interact with it just to prove that it works. Both of these, along with the Bitpay "adaptive" solution and "miner voting," require significant software upgrades.
That's why Bitcoin Classic is the only (unlikely) solution up for consideration on Bitcoin. It is essentially a version of BIP101 that stops partway, and it requires no significant upgrades to external software. The requirement that all the existing software be upgraded to support the other solutions means that they may be dead on arrival, just like ipV6 has been languishing for years.
In conclusion, software is hard to write, and even harder to test. The opportunity cost I'm losing by writing this code is enormous, because open source development is not a paid job. My experience with modifying the litecoin codebase solidifies my opinion that solutions that require changes to how transactions are created or stored will never be adopted in any existing coin. The end solution to bitcoin, if the parties involved ever reach one at all, will be a variation on BIP101. If Segregated Witness is adopted, most people simply won't encode transactions with it, and the promised "60% increase" will probably turn out to be 10%. If the Lightning Network is pushed out, most people will use the simpler solution of altcoins. If Litecoin adopts a solution with miner voting, the blocksize will never increase because most miners simply won't vote. There is no malice involved, just a recognition that the risk and cost of modifying already complex transaction creation code (to each individual user) is too great to justify the minor benefits that mostly accrue to other people.
I hope that developers of altcoins will continue to follow us as this implementation nears completion. We plan to release detailed documentation and assist anyone who needs help to encourage widespread adoption of BIP101 variants. I'll make another post on Monday with an update about what we accomplished over the weekend.