Last night, all exchanges went into error for an unknown reason. The system responded as designed to the lack of prices and miners who were mining the algorithms that did not have prices.
We're investigating the cause of the issue and will report when we know more.
Exchanges went into error last night
Forum rules
The News forum is only for updates about the Prohashing pool.
Replies to posts in this forum should be related to the news being announced. If you need support on another issue, please post in the forum related to that topic or seek one of the official support options listed in the top right corner of the forums page or on prohashing.com/about.
For the full list of PROHASHING forums rules, please visit https://prohashing.com/help/prohashing- ... rms-forums.
The News forum is only for updates about the Prohashing pool.
Replies to posts in this forum should be related to the news being announced. If you need support on another issue, please post in the forum related to that topic or seek one of the official support options listed in the top right corner of the forums page or on prohashing.com/about.
For the full list of PROHASHING forums rules, please visit https://prohashing.com/help/prohashing- ... rms-forums.
-
Steve Sokolowski
- Posts: 4585
- Joined: Wed Aug 27, 2014 3:27 pm
- Location: State College, PA
-
Steve Sokolowski
- Posts: 4585
- Joined: Wed Aug 27, 2014 3:27 pm
- Location: State College, PA
Re: Exchanges went into error last night
After some investigation, I determined the general cause of the problem.
We added the concept of a "price timeout," where if it takes too long for a single HTTP operation to an exchange to complete, the market is placed into error and the next price is queried. There is also existing code where the mining server has a failsafe to place coins in error if no price data is received within 3600.0 seconds. The failsafe will disconnect all miners to prevent potential losses due to price volatility.
It was never considered that some exchanges may take a very long time to respond to every single call, rather than just one call. If an exchange requires 100 tiny HTTP requests, and each request has a timeout of 10 seconds, then that one exchange alone would require 1000 seconds just to timeout all of its markets. In this case, it appears that one or two exchanges because extremely slow for an unknown reason, and the cumulative time the trader needed to wait before timing out over and over again was longer than the mining server's failsafe.
The solution to this problem is to add an "overall timeout," which is much lower than how long it would take to wait for many individual timeouts. If it takes longer than the overall timeout to process all the prices from an exchange, then that exchange's data will be marked as invalid and the program will move onto the next exchange's prices, allowing price updates to be received before the mining server goes into failsafe mode. We plan to work on that and get it implemented before February 29.
The purpose of the failsafe was to prevent losses of money by disconnecting miners instead of allowing them to mine due to invalid price data, and it worked as designed. No share corrections are necessary in this instance because, as the documentation indicates, compensation is only available when miners are connected but mining incorrectly.
Thanks for your patience!
We added the concept of a "price timeout," where if it takes too long for a single HTTP operation to an exchange to complete, the market is placed into error and the next price is queried. There is also existing code where the mining server has a failsafe to place coins in error if no price data is received within 3600.0 seconds. The failsafe will disconnect all miners to prevent potential losses due to price volatility.
It was never considered that some exchanges may take a very long time to respond to every single call, rather than just one call. If an exchange requires 100 tiny HTTP requests, and each request has a timeout of 10 seconds, then that one exchange alone would require 1000 seconds just to timeout all of its markets. In this case, it appears that one or two exchanges because extremely slow for an unknown reason, and the cumulative time the trader needed to wait before timing out over and over again was longer than the mining server's failsafe.
The solution to this problem is to add an "overall timeout," which is much lower than how long it would take to wait for many individual timeouts. If it takes longer than the overall timeout to process all the prices from an exchange, then that exchange's data will be marked as invalid and the program will move onto the next exchange's prices, allowing price updates to be received before the mining server goes into failsafe mode. We plan to work on that and get it implemented before February 29.
The purpose of the failsafe was to prevent losses of money by disconnecting miners instead of allowing them to mine due to invalid price data, and it worked as designed. No share corrections are necessary in this instance because, as the documentation indicates, compensation is only available when miners are connected but mining incorrectly.
Thanks for your patience!