Investigating hashrate issue

[Steve Sokolowski]

We're currently investigating a hashrate issue that affects a limited number of customers. These customers have a small decline of about 10% of hashrate. We think that the issue is caused by rejected shares that aren't recorded in the database. The issue only seems to occur with SHA-256 and scrypt.

We hope to have a solution within a few hours.

[Steve Sokolowski]

Chris will be restarting the mining servers over the next hour to try to get this set of changes out.

[GenTarkin]

Awesome, ill report back here or in chat if my reject rate doesnt improve after that.

[Steve Sokolowski]

Awesome, ill report back here or in chat if my reject rate doesnt improve after that.

This release has just been issued, and we believe the problem has been resolved, because we identified a specific line that could have caused it.

Let me know if that's not the case.

[GenTarkin]

Looks like still a problem somewhere. The reject reason seems to have changed the error output reason/message but Im still getting prety high 10-20% (prior to the 15th I was gettin 1-5% )reject rates =(
Example:
May 18 19:32:20 (none) local0.notice cgminer[10152]: Rejected 8c6545ec Diff 119K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:32:23 (none) local0.notice cgminer[10152]: Rejected 3d9254d2 Diff 272K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:32:27 (none) local0.notice cgminer[10152]: Rejected 011f03e0 Diff 58.5K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:32:35 (none) local0.notice cgminer[10152]: Rejected 28e534f9 Diff 410K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:32:41 (none) local0.notice cgminer[10152]: Rejected 011132d1 Diff 61.4K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:32:42 (none) local0.notice cgminer[10152]: Rejected ad87635d Diff 96.7K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:32:57 (none) local0.notice cgminer[10152]: Rejected 17e17045 Diff 703K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:33:02 (none) local0.notice cgminer[10152]: Rejected 75f061ea Diff 142K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:33:07 (none) local0.notice cgminer[10152]: Rejected 01f76ed6 Diff 33.3K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:33:07 (none) local0.notice cgminer[10152]: Rejected 01f29e2e Diff 33.6K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:33:16 (none) local0.notice cgminer[10152]: Rejected 01de7095 Diff 35.1K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:33:17 (none) local0.notice cgminer[10152]: Rejected 0170883c Diff 45.5K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:33:18 (none) local0.notice cgminer[10152]: Rejected 1a409e43 Diff 639K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:33:19 (none) local0.notice cgminer[10152]: Rejected 2295f415 Diff 485K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:33:21 (none) local0.notice cgminer[10152]: Rejected 3a818af6 Diff 287K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:33:25 (none) local0.notice cgminer[10152]: Rejected 978416b5 Diff 111K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:33:26 (none) local0.notice cgminer[10152]: Rejected 01864db6 Diff 43K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:33:28 (none) local0.notice cgminer[10152]: Rejected 01a36670 Diff 40K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:33:29 (none) local0.notice cgminer[10152]: Rejected 01ab3338 Diff 39.3K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:34:16 (none) local0.notice cgminer[10152]: Rejected 015fe712 Diff 47.7K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:34:16 (none) local0.notice cgminer[10152]: Rejected 019c82ec Diff 40.7K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:34:17 (none) local0.notice cgminer[10152]: Rejected 01a7c135 Diff 39.6K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:34:25 (none) local0.notice cgminer[10152]: Rejected 018e1974 Diff 42.1K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:34:26 (none) local0.notice cgminer[10152]: Rejected 9bf8a065 Diff 108K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:34:26 (none) local0.notice cgminer[10152]: Rejected 743ae34f Diff 144K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:34:27 (none) local0.notice cgminer[10152]: Rejected 32a1be2e Diff 331K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:34:29 (none) local0.notice cgminer[10152]: Rejected 16d0680a Diff 735K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:34:35 (none) local0.notice cgminer[10152]: Rejected 0174a67a Diff 45K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:34:36 (none) local0.notice cgminer[10152]: Rejected 016308c1 Diff 47.3K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:34:36 (none) local0.notice cgminer[10152]: Rejected 0175129e Diff 45K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:34:37 (none) local0.notice cgminer[10152]: Rejected a45b0421 Diff 102K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:34:42 (none) local0.notice cgminer[10152]: Rejected 01738c2d Diff 11.6M/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:34:44 (none) local0.notice cgminer[10152]: Rejected 01ea7c18 Diff 34.2K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:34:50 (none) local0.notice cgminer[10152]: Rejected ffcb1203 Diff 65.6K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:34:53 (none) local0.notice cgminer[10152]: Rejected 56e93a92 Diff 193K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:35:01 (none) local0.notice cgminer[10152]: Rejected 0eebaaca Diff 1.12M/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:35:04 (none) local0.notice cgminer[10152]: Rejected b237a1a5 Diff 94.1K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:35:13 (none) local0.notice cgminer[10152]: Rejected 019dbbe1 Diff 40.5K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:35:19 (none) local0.notice cgminer[10152]: Rejected 3d1c6b11 Diff 275K/32768 L3 0 pool 0 (The submitted share's diffic
May 18 19:35:36 (none) local0.notice cgminer[10152]: Rejected 01d5d611 Diff 35.7K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:35:38 (none) local0.notice cgminer[10152]: Rejected 01bd97ff Diff 37.7K/32768 L3 0 pool 0 (A bug in the mining server p
May 18 19:35:39 (none) local0.notice cgminer[10152]: Rejected 69aa448b Diff 159K/32768 L3 0 pool 0 (A bug in the mining server p

[GenTarkin]

I think the latest change may have fixed the problem. Mining for 17mins now w/o rejects .. will keep an eye on it and inform if regresses

[Steve Sokolowski]

I think the latest change may have fixed the problem. Mining for 17mins now w/o rejects .. will keep an eye on it and inform if regresses

Great - I think so too, because I was able to reproduce the issue finally in the development environment, and then I was able to confirm that it didn't happen after I applied the fix.

[GenTarkin]

Looks like as of this morning, the rejects have started creeping back in .. but not nearly at the horrible rate they were yesterday. But they do have similar error / reason messge .. Ill keep an eye on it on my end, maybe it was just some connection variability idk:
May 19 07:56:35 (none) local0.notice cgminer[14239]: Rejected 7ba94248 Diff 136K/32768 L3 0 pool 0 (This job was assigned to a d
May 19 07:56:35 (none) local0.notice cgminer[14239]: Rejected a2e07078 Diff 103K/32768 L3 0 pool 0 (This job was assigned to a d
May 19 07:56:35 (none) local0.notice cgminer[14239]: Rejected 0192cf81 Diff 41.6K/32768 L3 0 pool 0 (This job was assigned to a d
May 19 07:56:35 (none) local0.notice cgminer[14239]: Rejected 06069c02 Diff 2.78M/32768 L3 0 pool 0 (This job was assigned to a d
May 19 07:57:11 (none) local0.notice cgminer[14239]: Rejected 27725a76 Diff 425K/131072 L3 0 pool 0 (A bug in the mining server p
May 19 07:57:33 (none) local0.notice cgminer[14239]: Rejected 7cee9526 Diff 134K/131072 L3 0 pool 0 (A bug in the mining server p
May 19 07:57:33 (none) local0.notice cgminer[14239]: Rejected 1de94ded Diff 561K/131072 L3 0 pool 0 (A bug in the mining server p
May 19 07:57:33 (none) local0.notice cgminer[14239]: Rejected 54362c24 Diff 199K/131072 L3 0 pool 0 (A bug in the mining server p
May 19 07:59:02 (none) local0.notice cgminer[14239]: Rejected 2bb047c0 Diff 384K/131072 L3 0 pool 0 (A bug in the mining server p
May 19 07:59:17 (none) local0.notice cgminer[14239]: Rejected 1d5bc38c Diff 571K/131072 L3 0 pool 0 (A bug in the mining server p
May 19 07:59:32 (none) local0.notice cgminer[14239]: Rejected 2d400655 Diff 371K/131072 L3 0 pool 0 (A bug in the mining server p
May 19 07:59:41 (none) local0.notice cgminer[14239]: Rejected 053a1ac9 Diff 3.21M/131072 L3 0 pool 0 (A bug in the mining server p
May 19 07:59:51 (none) local0.notice cgminer[14239]: Rejected 2d0839cc Diff 373K/131072 L3 0 pool 0 (A bug in the mining server p
May 19 08:00:00 (none) local0.notice cgminer[14239]: Rejected 0a31f2e8 Diff 1.65M/131072 L3 0 pool 0 (A bug in the mining server p
May 19 08:00:36 (none) local0.notice cgminer[14239]: Rejected 5a71749b Diff 185K/131072 L3 0 pool 0 (A bug in the mining server p
May 19 08:00:50 (none) local0.notice cgminer[14239]: Rejected 06f140f1 Diff 2.42M/131072 L3 0 pool 0 (A bug in the mining server p
May 19 08:01:31 (none) local0.notice cgminer[14239]: Rejected 07f88d49 Diff 2.1M/131072 L3 0 pool 0 (A bug in the mining server p
May 19 08:01:52 (none) local0.notice cgminer[14239]: Rejected 1707ca42 Diff 728K/131072 L3 0 pool 0 (A bug in the mining server p
May 19 08:02:22 (none) local0.notice cgminer[14239]: Rejected 171d850b Diff 726K/131072 L3 0 pool 0 (A bug in the mining server p
May 19 08:02:23 (none) local0.notice cgminer[14239]: Rejected 4d9fbbe7 Diff 216K/131072 L3 0 pool 0 (A bug in the mining server p

[Steve Sokolowski]

I'll take a look and get back to you shortly. What timezone are these logs in, so I can investigate the problem?

[Steve Sokolowski]

After some investigation, I determined that these errors are being caused by the mining servers disconnecting from WAMP.

I'll look into these problems and hope to have a fix by tomorrow.