Status as of Monday, September 24, 2018
Posted: Mon Sep 24, 2018 8:30 am
Good morning!
- I made a breakthrough yesterday in identifying some of the network connectivity issues being encountered by the system, especially those involving Ethash mining. These problems had hindered Ethereum for about 6 weeks. It turns out the problem was that there was insufficient bandwidth for the coin daemon servers. Because bandwidth was too limited, some coins would fail to respond to RPC calls in a timely fashion, causing coins to go into error frequently and blocks to be orphaned more often than necessary.
- Bandwidth is managed by a program that restricts coin bandwidth most of the time, and then opens up the restrictions when a block has been found so that the daemon can upload the block as quickly as possible to the rest of the network. There were two problems. First, due to an arithmetic error, each daemon server had 1Mbps of upload bandwidth available to it for all coins when no coins were ready for block submission - I resolved that by increasing the bandwidth to 7Mbps per daemon server. Second, the bandwidth is managed on a per-server basis, rather than on a per-coin basis, which means that one coin on a server can starve all the other coins for bandwidth. Today, I'll be modifying the daemon-network-management program to create new tc and iptables classes so that there is one bandwidth class per coin, rather than one for all coins.
- However, despite there still being more work to be done to make the bandwidth management perfect, simply increasing the amount of bandwidth allocated to the coin servers has so far eliminated all of the service outages from those servers; there hasn't been a single case where the time got out of sync or where a block failed submission since yesterday.
- Later today, expect a new website release. This release will add a new chart to the "Earnings" page that compares actual vs target earnings over the past day. The chart will allow customers to more easily identify when errors have caused earnings adjustments, and will complete our work on making that data visible to customers.