The difficult task of comparing mining pools

[Steve Sokolowski]

Having taken a break from adding new features to focus on stability and marketing, Chris began to investigate competitors in the scrypt mining industry. At first, he thought that evaluating pools would be as simple as signing up, seeing if they offer a lot of features, and then seeing how much he would get paid. Alas, he quickly discovered that obtaining accurate profitability metrics for pools is extremely difficult. The purpose of this post is to review some of the difficulties we've encountered and to explore some of the ways that comparing pools could be made more accurate.

Mining is not as simple as starting up an ASIC and telling it to mine at a pool, as not all ASICs are the same. Because scrypt requires many intermediate calculations that must be stored in memory before hashing begins at full speed, there is a brief period at the beginning of each work unit when these calculations are performed. Some miners, like KNC Titans, are very inefficient at performing these calculations, so they take a second or more to even start hashing at all when a new block is found. Such miners perform poorly on pools that mine coins with low block times, and therefore earn less money despite their high manufacturer-advertised hashrate. This brings us to the first lesson:

Lesson #1: "1 MH/s" on a miner that has a low work restart time is not the same as "1 MH/s" on a miner that has a high work restart time.

This lesson alone significantly reduces the usefulness of sites like Poolpicker (http://poolpicker.eu), a site that compares the profitability of mining pools. The site lists profitability metrics for a number of pools side by side. However, since Poolpicker doesn't publish the average block times for each of the pools, the numbers are not comparable for any given miner. For example, pools that send the entire pool's hashrate to a single coin to overload the network often result in high profitability metrics, but most customers are not able to obtain those because their miners never have enough time to get started with their work before the next block arrives.

Comparing profitability metrics on sites like Poolpicker is flawed in many other ways. Another lesson follows:

Lesson #2: Profitability calculations are not comparable unless they are computed at the exact same time.

There is no standard reporting period for mining pools, so every pool reports a different period. A visit to one site might display one daily profitability value calculated in UTC, another in Eastern European Standard TIme, and another in Pacific Daylight Time. While it might sound reasonable to extrapolate profitability across an entire day or to extend yesterday's earnings, during most of the time mining is spent on low-profitability coins like litecoins. Every hour or so, there is a minute when profitability spikes to 30x or more than what it was before, which means that half the pool's profits are made during that minute. The other half is made mining the low profitability base coins like litecoins. There is no known way to predict when those periods are going to occur, so the only way to earn such rewards is to keep miners earning a low amount of money until a profitable coin comes along. Mining for 15 minutes and then multiplying by 96 will not yield an accurate estimate of daily profitability.

But even if it were possible to standardize reporting periods, comparing pools fails because there is no standard in accounting for pools' fees. Some pools, like Clevermining, do not even publish what their fees are, instead suggesting a wide range:

Lesson #3: Reported profitability can be presented before fees are subtracted or after, if the fees are even published at all.

It was recently pointed out here that one pool reports their earnings before fees are subtracted, so actual payouts differ from what is published. To complicate matters, some pools charge static withdrawal fees that do not scale linearly with what is earned. These fees eat into profits that actually reach customers' wallets and disproportionately affect small miners. Finally, some pools pay out in the coins that were mined, rather than in the coins that users want (like bitcoins), so a straight comparison of profitability ignores both exchange trade fees and the exchange withdrawal fees. In fact, bitcoins are often the most expensive coins to withdrawal from exchanges.

Complexity in trading often affect pool payouts. Some pools report the value of earnings now, rather than what the mined coins will be worth when the coins are sold:

Lesson #4: A pool that reports profits based on the value of unsold coins will almost always overreport profits.

Since coins often have periods of high profitability after a difficulty adjustment, it's possible for a pool to mine hundreds of blocks in a row before that coin's difficulty changes again. In many cases, the coin's market isn't deep enough to sustain the sale of all the blocks without a fall in price. Many pools manually sell coins all at once, but report profits by multiplying the current sell price of all coins on hand times the number of coins held. Unless a bubble is forming in one of these coins, this practice almost always results in the pool operator paying out less than promised.

Another factor can play into profit reporting: opportunity cost. In December, the Infernopool was paying extremely high profitability because it was assuming that its balances would be honored by Cryptsy. The owners acknowledged on the website that they were fronting the money from their own accounts.

Lesson #5: The risk of failure should be subtracted from the reported figure.

While most pools are not at risk of failure, operators who take exceptional risks to pay out more should be accounted for by miners. If a pool is using an exchange that has a 10% risk of failure, but is only paying 5% more money, then given enough time the customers can expect to average a loss. Arguably, payout delays should also be considered - if a pool has a 2% chance of going out of business every week, and payouts don't occur for a week, then the risk is greater at that pool.

After all that research, Chris concluded that the only true way to determine how much can be earned by mining at a pool is to actually mine there and count balances only when they arrive in the payout wallet. If it were profitable to do so, we would like to set up a system where identical miners are periodically connected to pools at the same time and the amount of money paid to wallets by each of the pools was listed. Unfortunately, the cost of buying the multiple types of mining equipment necessary to accurately test miners with different characteristics makes the venture unprofitable, but such a system would be the ultimate arbiter that would be much more valid than self-reporting of pools.

Until such a system is available, the only reliable way to determine which pool is the most profitable is to try them all out with your own miners.