Page 1 of 1
Exchanges went into error last night
Posted: Mon Feb 17, 2020 8:43 am
by Steve Sokolowski
Last night, all exchanges went into error for an unknown reason. The system responded as designed to the lack of prices and miners who were mining the algorithms that did not have prices.
We're investigating the cause of the issue and will report when we know more.
Re: Exchanges went into error last night
Posted: Mon Feb 17, 2020 11:05 am
by Steve Sokolowski
After some investigation, I determined the general cause of the problem.
We added the concept of a "price timeout," where if it takes too long for a single HTTP operation to an exchange to complete, the market is placed into error and the next price is queried. There is also existing code where the mining server has a failsafe to place coins in error if no price data is received within 3600.0 seconds. The failsafe will disconnect all miners to prevent potential losses due to price volatility.
It was never considered that some exchanges may take a very long time to respond to every single call, rather than just one call. If an exchange requires 100 tiny HTTP requests, and each request has a timeout of 10 seconds, then that one exchange alone would require 1000 seconds just to timeout all of its markets. In this case, it appears that one or two exchanges because extremely slow for an unknown reason, and the cumulative time the trader needed to wait before timing out over and over again was longer than the mining server's failsafe.
The solution to this problem is to add an "overall timeout," which is much lower than how long it would take to wait for many individual timeouts. If it takes longer than the overall timeout to process all the prices from an exchange, then that exchange's data will be marked as invalid and the program will move onto the next exchange's prices, allowing price updates to be received before the mining server goes into failsafe mode. We plan to work on that and get it implemented before February 29.
The purpose of the failsafe was to prevent losses of money by disconnecting miners instead of allowing them to mine due to invalid price data, and it worked as designed. No share corrections are necessary in this instance because, as the documentation indicates, compensation is only available when miners are connected but mining incorrectly.
Thanks for your patience!