Explanation of earnings from Dec 11-13

[Steve Sokolowski]

We recently discovered an issue with earnings for Dec 11-13, which was brought to our attention as a result of many support tickets submitted on Dec 14 that drew attention to lower profits that day compared to the previous days.

Chris spent over 12 hours investigating this problem. It turns out that during the period of Dec 11-13, exchanges with low volume did not see its litecoin price track the price of the other exchanges. This was a period of an extreme increase in litecoin prices, up to over $400. The data below demonstrates the discrepancies. The first two columns are the day of the month and the hour of day. The third and fourth columns are the Poloniex LTC/BTC price, and the fourth column is the price at the lowest-priced exchange. The ratio of the two prices is in the fifth column.


day hour poloniex_price outlier_price ratio_of_outlier_to_poloniex
9 0 0.009782696 0.009466673 0.967695698
9 1 0.009589155 0.009415575 0.981898354
9 2 0.009477958 0.009391412 0.990868638
9 3 0.010008159 0.009896414 0.988834557
9 4 0.010014572 0.00997425 0.995973662
9 5 0.010022874 0.009930385 0.990772179
9 6 0.009646416 0.009552601 0.99027464
9 7 0.009699054 0.009689729 0.99903861
9 8 0.009874545 0.009839075 0.996407955
9 9 0.010159194 0.010121642 0.996303628
9 10 0.010154612 0.010119304 0.996523024
9 11 0.010196556 0.010157176 0.996137942
9 12 0.01013047 0.010063301 0.993369671
9 13 0.01003676 0.009775917 0.974011146
9 14 0.01003002 0.009896656 0.986703544
9 15 0.010016726 0.009987674 0.997099651
9 16 0.010106209 0.010026333 0.99209634
9 17 0.010151133 0.010065681 0.991581995
9 18 0.010111847 0.01010398 0.999222003
9 19 0.010157163 0.010099486 0.994321535
9 20 0.010138628 0.009984377 0.98478589
9 21 0.010084344 0.009810804 0.972874853
9 22 0.009936678 0.009858729 0.992155454
9 23 0.009846391 0.009735912 0.988779702
10 0 0.009901798 0.009897684 0.999584502
10 1 0.00986684 0.009893148 1.002666256
10 2 0.00995562 0.009922419 0.996665016
10 3 0.009970084 0.009921371 0.995114066
10 4 0.009833823 0.009880609 1.004757651
10 5 0.009754749 0.009789207 1.003532479
10 6 0.00969487 0.00971957 1.002547715
10 7 0.009726326 0.009764681 1.003943456
10 8 0.009792292 0.009807416 1.00154447
10 9 0.00939144 0.009335078 0.993998518
10 10 0.009173528 0.009184427 1.001188101
10 11 0.009193513 0.009162812 0.996660558
10 12 0.00962281 0.009579762 0.995526477
10 13 0.0098509 0.009864184 1.001348512
10 14 0.009882647 0.009893534 1.001101552
10 15 0.009802563 0.009755142 0.995162394
10 16 0.009708812 0.009756616 1.004923823
10 17 0.009700294 0.009740592 1.004154275
10 18 0.009560566 0.009544129 0.998280715
10 19 0.009580186 0.009534353 0.995215808
10 20 0.009509071 0.009525142 1.001690076
10 21 0.00940565 0.009257362 0.984234193
10 22 0.00940565 0.009134416 0.97116267
10 23 0.00940565 0.009172771 0.975240483
11 0 0.009479664 0.009384498 0.989961029
11 1 0.009753564 0.009688587 0.993338102
11 2 0.009515552 0.009565896 1.005290718
11 3 0.009554868 0.009558331 1.000362403
11 4 0.009674401 0.009696491 1.002283329
11 5 0.009767942 0.00970445 0.993499968
11 6 0.009685061 0.0097 1.001542481
11 7 0.009828467 0.0097 0.986929088
11 8 0.010346858 0.0097 0.937482643
11 9 0.010953289 0.0097 0.885578724
11 10 0.010992889 0.0097 0.882388579
11 11 0.01093475 0.0097 0.887080164
11 12 0.011400066 0.0097 0.850872252
11 13 0.011742657 0.0097 0.826048157
11 14 0.011635209 0.0097 0.833676491
11 15 0.010983364 0.0097 0.883153862
11 16 0.011255672 0.0097 0.861787717
11 17 0.012403952 0.011050051 0.890849224
11 18 0.012797908 0.010807347 0.844462015
11 19 0.012581696 0.0097 0.770961245
11 20 0.013174643 0.0097 0.736262846
11 21 0.014281162 0.0097 0.679216457
11 22 0.014099521 0.0097 0.687966646
11 23 0.014422838 0.0097 0.672544481
12 0 0.014513866 0.0097 0.668326397
12 1 0.013992004 0.0097 0.693253093
12 2 0.013566193 0.0097 0.715012694
12 3 0.013994231 0.0097 0.69314275
12 4 0.014266388 0.010153771 0.711726802
12 5 0.014189908 0.0097 0.683584404
12 6 0.014972842 0.0097 0.6478396
12 7 0.016772367 0.0097 0.57833219
12 8 0.016490732 0.0097 0.588209199
12 9 0.018509711 0.0097 0.524049244
12 10 0.017947357 0.0097 0.540469545
12 11 0.018032625 0.0097 0.537913929
12 12 0.018242366 0.010417435 0.571057218
12 13 0.018451966 0.0097 0.525689248
12 14 0.01822378 0.0097 0.532271583
12 15 0.01858688 0.0097 0.521873498
12 16 0.018692429 0.0097 0.518926661
12 17 0.017624562 0.0097 0.550368286
12 18 0.017312481 0.0097 0.560289435
12 19 0.017784813 0.0097 0.545409178
12 20 0.01778565 0.0097 0.545383497
12 21 0.017749377 0.0097 0.546498062
12 22 0.017643157 0.0097 0.549788235
12 23 0.017557074 0.0097 0.552483866
13 0 0.018032605 0.011337665 0.628731396
13 1 0.018220947 0.0097 0.532354342
13 2 0.017984421 0.0097 0.539355709
13 3 0.017858137 0.0097 0.543169775
13 4 0.017756707 0.011717544 0.659893976
13 5 0.017662683 0.016738506 0.947676307
13 6 0.017581802 0.0097 0.551706829
13 7 0.018173932 0.0097 0.533731507
13 8 0.019404351 0.0097 0.499887891
13 9 0.019094611 0.0097 0.507996725
13 10 0.018622946 0.0097 0.520862805
13 11 0.018642836 0.0097 0.520307112
13 12 0.018249918 0.0097 0.531509226
13 13 0.018027596 0.0097 0.538063989
13 14 0.018240914 0.0097 0.5317716
13 15 0.017916934 0.0097 0.541387265
13 16 0.018063287 0.0097 0.537000815
13 17 0.01828951 0.015820533 0.865005886
13 18 0.018211909 0.018219305 1.000406109
13 19 0.018113646 0.018140526 1.001483972
13 20 0.018316162 0.018333814 1.000963717
13 21 0.0185736 0.018484683 0.995212685
13 22 0.018527631 0.018505473 0.998804043
13 23 0.018561171 0.018535414 0.998612325
14 0 0.018445996 0.018417976 0.998481003
14 1 0.018305062 0.018279205 0.998587402
14 2 0.018327354 0.018297003 0.998343971
14 3 0.017875151 0.017783944 0.994897543
14 4 0.01776668 0.017782626 1.000897513
14 5 0.017878657 0.017879134 1.000026643
14 6 0.01789963 0.017898786 0.999952867
14 7 0.01772104 0.017692035 0.998363266
14 8 0.017397729 0.017277846 0.993109259
14 9 0.017068026 0.016980412 0.994866773
14 10 0.016493799 0.016383273 0.993298963
14 11 0.017117219 0.017008784 0.99366513
14 12 0.017022163 0.017024169 1.000117872
14 13 0.016387525 0.016351973 0.997830556
14 14 0.016235473 0.016228674 0.999581229
14 15 0.016508412 0.016500408 0.999515161
14 16 0.016674859 0.016685069 1.00061235
14 17 0.016566461 0.01656444 0.999878008
14 18 0.016436872 0.016457069 1.001228728
14 19 0.016541031 0.01657557 1.002088079
14 20 0.015947788 0.01582927 0.992568396
14 21 0.014870482 0.014769196 0.993188831
14 22 0.015030073 0.014921239 0.992758896
14 23 0.015905438 0.015756781 0.990653679
15 0 0.015556504 0.015538616 0.998850117
15 1 0.015627198 0.015603145 0.998460828
15 2 0.015650137 0.015624321 0.998350454
15 3 0.015663773 0.015662701 0.999931557
15 4 0.015261283 0.015279209 1.001174587
15 5 0.014555103 0.014486663 0.995297886
15 6 0.01458068 0.014607529 1.001841404
15 7 0.014602197 0.01462302 1.001425989
15 8 0.016013318 0.015822721 0.988097569
15 9 0.016267617 0.016191977 0.995350317
15 10 0.017121171 0.016952683 0.990159073
15 11 0.017387858 0.017332687 0.996827079
15 12 0.017519658 0.017434133 0.995118331
15 13 0.016998107 0.016955418 0.997488588
15 14 0.016834257 0.016815099 0.99886196
15 15 0.017241564 0.017219685 0.998731006
15 16 0.017252827 0.017283437 1.001774156
15 17 0.017153128 0.017126938 0.998473128
15 18 0.017022402 0.016859461 0.990427897
15 19 0.016925558 0.016717003 0.987678107
15 20 0.017007542 0.016973797 0.998015909
15 21 0.017208563 0.017225882 1.00100643
15 22 0.017090554 0.017154707 1.00375374
15 23 0.016985074 0.016974657 0.999386682
16 0 0.01696603 0.016981236 1.000896239
16 1 0.016866242 0.016818023 0.997141097
16 2 0.016568316 0.016575505 1.000433891
16 3 0.01665539 0.01668989 1.002071385
16 4 0.016936251 0.016634471 0.982181438
16 5 0.016721438 0.016602252 0.992872256
16 6 0.016656748 0.016668366 1.00069749
16 7 0.016862942 0.016818788 0.997381579
16 8 0.016869742 0.016850476 0.99885794

As you can see, at about 7:00am EST on Dec 11, the price of LTC/BTC on Poloniex started to dramatically climb, but the lower volume exchanges did not immediately respond to the sudden jump. Once the LTC/BTC price fell after the price of LTC declined on Dec 13 at 6:00pm EST, all the exchanges returned to around the same price. Between those two times, you can see where the ratio is as low as 0.5, meaning that LTC on Poloniex was twice as valuable as that on the other exchanges. It's strange to me that prices became this far out of sync, as normally people seem to sell litecoins on the higher priced exchange when this happens. However, we double checked this data and determined that it is correct.

During that time, we used the lowest buy price in calculating earnings, as we always do, and therefore customers who requested to be paid in litecoins made (1/column 5) times what they should have earned. When column 5 returned to normal in the later rows, many customers who were overpaid in previous days noticed that their next day's profits had declined to the correct amount and submitted support tickets.

About $100,000 was overpaid in this incident. While our policy is that we may request all but 15% of the overpayment back, we decided not to execute any clawbacks. If you requested litecoins for payment during those days, enjoy the Christmas bonus!

As an aside, I troubleshot the issue on the morning of Dec 15 and because I thought that the problem was occurring then, rather than already having been fixed, I messed up the hashrate charts for that day. Chris is fixing them, but no money was affected that day.

Chris will be directing all support tickets about the issue to this post. In the future, we'll add a new feature to cease trading and using price data from exchanges where the price is such a dramatic outlier from all the others like this.

[Ewil]

Yikes!
That does explain a fair bit...

Question - is this why the "Total Earnings" just went ultra-wonky in my history today? I wasn't keeping precise records, but it appears that LTC earnings (across several accounts, not just mine) both went up & down overall each day, as if the present USD value of LTC got retro-actively assigned to payouts, rather than snapshots for each days payouts.

[AppleMiner]

Well that sucks. I only came back over to the pool after I had seen those profits on those days.
So if the norm is returning to a lower level, miners are going back to another pool with a higher payout then.

[Steve Sokolowski]

Yikes!
That does explain a fair bit...

Question - is this why the "Total Earnings" just went ultra-wonky in my history today? I wasn't keeping precise records, but it appears that LTC earnings (across several accounts, not just mine) both went up & down overall each day, as if the present USD value of LTC got retro-actively assigned to payouts, rather than snapshots for each days payouts.

The USD values for those days were calculated based upon the prices of the lowest exchange, which wasn't correct. Chris retroactively corrected the USD values to be correct. The amount of money actually paid never changed, and the number of litecoins was always correct.

[gnikminer]

Call this a dumb question, but you say that people were overpaid, but you were using the lowest buy price on LTC. If you were purchasing LTC at these prices, sure, they may have received more LTC than with the other exchanges, but you still distributed the same amount of BTC value, which is how everything on your site is valued. How is that overpaying if you are still spending the same amount as you stated that you owed?

Example:
Miner owed .5 BTC
Exchange 1 LTC price - .09 BTC
Exchange 2 LTC price - .05 BTC
Pool buys 10 LTC at exchange 2 to pay miner - .5 BTC
Just because the miner received more LTC and could profit by flipping it at exchange 1 doesn't mean that the pool is out any extra money.

I don't see where it would have been fair if you were to have recalled payouts, yet you're claiming it as a bonus. Keep in mind, I've been solo mining lately, so I'm not effected by this one way or another. I'm just trying to wrap my head around your business practices. I get more and more confused by them as you handle situations. Like the solo mining merge mined blocks being PPS? I looked through the website after you replied to my ticket, and there was no reference to PPS for merge mining while solo. That section from the help page that I sent you a screenshot of was the only reference to the subject which states that the blocks go to the miners.

[NYCrypto1]

I would be interested in reading a reply to gnikminers statement.

[Steve Sokolowski]

Call this a dumb question, but you say that people were overpaid, but you were using the lowest buy price on LTC. If you were purchasing LTC at these prices, sure, they may have received more LTC than with the other exchanges, but you still distributed the same amount of BTC value, which is how everything on your site is valued. How is that overpaying if you are still spending the same amount as you stated that you owed?

Example:
Miner owed .5 BTC
Exchange 1 LTC price - .09 BTC
Exchange 2 LTC price - .05 BTC
Pool buys 10 LTC at exchange 2 to pay miner - .5 BTC
Just because the miner received more LTC and could profit by flipping it at exchange 1 doesn't mean that the pool is out any extra money.

I don't see where it would have been fair if you were to have recalled payouts, yet you're claiming it as a bonus. Keep in mind, I've been solo mining lately, so I'm not effected by this one way or another. I'm just trying to wrap my head around your business practices. I get more and more confused by them as you handle situations. Like the solo mining merge mined blocks being PPS? I looked through the website after you replied to my ticket, and there was no reference to PPS for merge mining while solo. That section from the help page that I sent you a screenshot of was the only reference to the subject which states that the blocks go to the miners.

That's pretty simple. We didn't need to purchase any litecoins, because we had already mined enough, so we didn't buy litecoins at that price.

[gnikminer]

So, you're saying that because you already had them that the value was higher than what you would have purchased them for, therefore a loss? This would put purchasing the LTC a better scenario. To distribute the coin with out replacing it indicates that the value is even or lower than the available cost. This makes the distribution that the miners received fair value, not a bonus.

[AppleMiner]

So if you didn't need to purchase LITECOINS, then people who got payouts in LITECOINS would also not be affected by the prices on the exchange?

So how were the profits of the people who were autopool mining and taking payouts in LTC which didnt need to be bought or sold to be paid on an exchange with incorrect pricing data affected also by this error?

[Ewil]

So if you didn't need to purchase LITECOINS, then people who got payouts in LITECOINS would also not be affected by the prices on the exchange?

So how were the profits of the people who were autopool mining and taking payouts in LTC which didnt need to be bought or sold to be paid on an exchange with incorrect pricing data affected also by this error?

I, too, am interested in the response to this question. If you mined the litecoins, then you're only out the equipment percentage/electricity cost and not the exchange cost. If there's one thing I've been told over and over again while dealing in crypto, it's "don't count value of a coin until it's exchanged".
Honestly the statement is now looking quite, um, "hollow", when you say 'While our policy is that we may request all but 15% of the overpayment back, we decided not to execute any clawbacks. If you requested litecoins for payment during those days, enjoy the Christmas bonus!' only for us to find out that it didn't cost *you* what you're saying it would on the exchanges.
These 'bugs' and 'discrepancies' are getting unusually numerous as of late...