Three challenges to running a daemon server
Posted: Wed Oct 14, 2015 5:58 am
I recently have had a few customers ask me some questions about our supported coins, which reminded me of how far my knowledge has come since I started dealing with coin daemons about two years ago. When I first installed the Litecoin client in December 2013, I never envisioned how effort is needed to keep coins online. From researching new coins to add to keeping existing coins up to date and correcting I spend about 10 hours per week adding new coins as well as updating and fixing coin daemons.
The first challenge I face on a daily basis is that most coins are poorly documented. As the altcoin market has become saturated, new coins focus disproportionately on flashy websites and catchy marketing while excluding basic information. Sometimes even tracking down essential information such as the algorithm of a given coin can require searching through multiple websites and forums. I hold an axiom – the flashier a coin’s website, the less likely it is to have any technical information. I’ve had to learn to read the source code to determine coin parameters. Even then, it isn’t simple to determine what parameter affects each. For example, Litecoin forks use the constant COINBASE_MATURITY to specify the number of confirmations before a newly found block can be spent, while Peercoin forks use nCoinbaseMaturity, and these constants as specified in different files for each fork.
This lack of documentation frustrates me every day. A few hours ago, I had a user contact me asking what the reward for Neucoin was because a competing pool was promising 100x the reward we state on our site. He couldn’t find any information on this, and even with my own searching on the web, there was no official information on the proof of work block reward. I am still trying to determine the actual block reward by investigating the source code. For all I know we may be paying out 1% of the actual reward without knowing it.
The second challenge I face is in the number of branches of code. Nearly every single coin we support is forked off of Litecoin, but there are dozens of different versions of the Litecoin client from which they are forked, each with its own quirks. For example, older forks use makefiles, and newer forks use automake. Some forks exclude critical directories such as the src/obj directory; others have file permissions that prevent compilation of the source. New coins have unique dependencies to support their unique features. The newest Litecoin forks only accepts commands from an executable separate from the daemon, while older forks accept commands directly through the daemon’s executable.
I have attempted to create a script that compiles coins automatically, which takes into account over 20 different possible forks and common coding mistakes, but even then that script still only works about half of the time. The rest of the time I have to manually diagnose the issue, and the lack of documentation compounds the problem. This is the primary reason why adding a new coin is difficult, because a new coin brings with it new dependencies and new quirks for which we have to account. I fully anticipate that by the end of next year my compilation script will have over 50 workarounds for broken coins.
The third challenge is in actually keeping the coins running on our server. Due to numerous bugs, coins crash frequently. Others have memory leaks that could cause a whole daemon server to crash. I have had to program a script that runs every hour, checking all coins to ensure they are running and cycling through restarting coins to work around memory leaks.
About 10 of our coins are in error for not having enough connections. Without enough connections, our found blocks have a high probability of being orphaned. Of course, the best scenario is to have our coin daemons connect with our exchange’s coin daemons, yet exchanges do not provide any connection information. One would think that exchanges would have an incentive to be the canonical chain for each coin by connecting as many peers as possible. They do not host their coin daemons on the same IP address as their main site, so attempting to connect to their websites is not possible. Even after contacting exchanges, they decline to provide this information, which baffles me.
Then, there’s the issue of keeping coins up to date. There’s no automated way that I am notified of forks, so the only way I can tell is by monitoring forums such as Bitcointalk for updates on each of our coins. Of course, we compare our chain heights with our exchanges’ chain heights and disable coins whose chain heights deviate too much, but just keeping up to date with the latest happenings in the altcoin world is the most time-consuming task I face.
While I try my best to support all coins on Prohashing, you will still see about 10 in error at any given time. I may be searching for a resolution to an issue running the coin, contacting the developer about a fix, or even just waiting to see if anyone else has the problem. Unfortunately, most of the issues that cause a coin to go into error are never fixed because the coin’s developer has disappeared, and we are forced to discontinue the coin. If you are ever wondering why a coin is in error and would like it fixed, contact me and I can provide you more explanation as to why it is offline. Maybe you will have the information that I need to get it back online.
A centralized system that tracks all coins and provides information on their networks would be ideal, and that is what we strive to be through our block explorer at Prohashing. I sometimes wish that the altcoin market would consolidate, reducing my workload, but I don’t hold any hope of that occurring. As long as there is money to be made you can be sure that the future will be more coins and more work to maintain them, and we will be there every step of the way.
The first challenge I face on a daily basis is that most coins are poorly documented. As the altcoin market has become saturated, new coins focus disproportionately on flashy websites and catchy marketing while excluding basic information. Sometimes even tracking down essential information such as the algorithm of a given coin can require searching through multiple websites and forums. I hold an axiom – the flashier a coin’s website, the less likely it is to have any technical information. I’ve had to learn to read the source code to determine coin parameters. Even then, it isn’t simple to determine what parameter affects each. For example, Litecoin forks use the constant COINBASE_MATURITY to specify the number of confirmations before a newly found block can be spent, while Peercoin forks use nCoinbaseMaturity, and these constants as specified in different files for each fork.
This lack of documentation frustrates me every day. A few hours ago, I had a user contact me asking what the reward for Neucoin was because a competing pool was promising 100x the reward we state on our site. He couldn’t find any information on this, and even with my own searching on the web, there was no official information on the proof of work block reward. I am still trying to determine the actual block reward by investigating the source code. For all I know we may be paying out 1% of the actual reward without knowing it.
The second challenge I face is in the number of branches of code. Nearly every single coin we support is forked off of Litecoin, but there are dozens of different versions of the Litecoin client from which they are forked, each with its own quirks. For example, older forks use makefiles, and newer forks use automake. Some forks exclude critical directories such as the src/obj directory; others have file permissions that prevent compilation of the source. New coins have unique dependencies to support their unique features. The newest Litecoin forks only accepts commands from an executable separate from the daemon, while older forks accept commands directly through the daemon’s executable.
I have attempted to create a script that compiles coins automatically, which takes into account over 20 different possible forks and common coding mistakes, but even then that script still only works about half of the time. The rest of the time I have to manually diagnose the issue, and the lack of documentation compounds the problem. This is the primary reason why adding a new coin is difficult, because a new coin brings with it new dependencies and new quirks for which we have to account. I fully anticipate that by the end of next year my compilation script will have over 50 workarounds for broken coins.
The third challenge is in actually keeping the coins running on our server. Due to numerous bugs, coins crash frequently. Others have memory leaks that could cause a whole daemon server to crash. I have had to program a script that runs every hour, checking all coins to ensure they are running and cycling through restarting coins to work around memory leaks.
About 10 of our coins are in error for not having enough connections. Without enough connections, our found blocks have a high probability of being orphaned. Of course, the best scenario is to have our coin daemons connect with our exchange’s coin daemons, yet exchanges do not provide any connection information. One would think that exchanges would have an incentive to be the canonical chain for each coin by connecting as many peers as possible. They do not host their coin daemons on the same IP address as their main site, so attempting to connect to their websites is not possible. Even after contacting exchanges, they decline to provide this information, which baffles me.
Then, there’s the issue of keeping coins up to date. There’s no automated way that I am notified of forks, so the only way I can tell is by monitoring forums such as Bitcointalk for updates on each of our coins. Of course, we compare our chain heights with our exchanges’ chain heights and disable coins whose chain heights deviate too much, but just keeping up to date with the latest happenings in the altcoin world is the most time-consuming task I face.
While I try my best to support all coins on Prohashing, you will still see about 10 in error at any given time. I may be searching for a resolution to an issue running the coin, contacting the developer about a fix, or even just waiting to see if anyone else has the problem. Unfortunately, most of the issues that cause a coin to go into error are never fixed because the coin’s developer has disappeared, and we are forced to discontinue the coin. If you are ever wondering why a coin is in error and would like it fixed, contact me and I can provide you more explanation as to why it is offline. Maybe you will have the information that I need to get it back online.
A centralized system that tracks all coins and provides information on their networks would be ideal, and that is what we strive to be through our block explorer at Prohashing. I sometimes wish that the altcoin market would consolidate, reducing my workload, but I don’t hold any hope of that occurring. As long as there is money to be made you can be sure that the future will be more coins and more work to maintain them, and we will be there every step of the way.