System reached capacity

[Steve Sokolowski]

Today, the system reached capacity, with CPU load close to 100%, as the predicted maximum worker count of 11,000 was approached. Therefore, we closed registrations.

Unfortunately, stale shares are expected to be high because the system cannot process coin assignment in real time anymore, and therefore miners continue to submit old work because they aren't being told about new work once it becomes available.
The fix for this problem is not easy, and it involves parallelizing the system to allow coin assignment to occur on multiple computers.

Thanks for your patience as we continue to try to get advice on how to move forward. We determined today that we will be able to make an announcement on November 17 as to the future of the pool, at which time we may reopen registrations.

[GregoryGHarding]

why wait so long to close registrations? you waited until the very last second before closing.. the point where cpu is 100%... 11000 workers.. and now shares or blocks will orphan because inserter will not be able to process real time.. lost profits for everyone. whereas it should have been closed last month, duringt he rise of x11 sales, system would be more stable capped off NOT as limit, and not as much stales/orphans/lost profit

[micca410evo]

why wait so long to close registrations? you waited until the very last second before closing.. the point where cpu is 100%... 11000 workers.. and now shares or blocks will orphan because inserter will not be able to process real time.. lost profits for everyone. whereas it should have been closed last month, duringt he rise of x11 sales, system would be more stable capped off NOT as limit, and not as much stales/orphans/lost profit

well fuck me, there is someone who thinks

[holygoof]

So, does this mean no new accounts only, or for those of us here already we will not be able to add more miners?
Thank you for the clarification and all the hard work you do there, I have been very happy with everything these past 21 days.
Tom

[centar]

why wait so long to close registrations? you waited until the very last second before closing.. the point where cpu is 100%... 11000 workers.. and now shares or blocks will orphan because inserter will not be able to process real time.. lost profits for everyone. whereas it should have been closed last month, duringt he rise of x11 sales, system would be more stable capped off NOT as limit, and not as much stales/orphans/lost profit

I have to agree here, especially as you guys have repeatedly stated you have no plans to improve the situation for several months at minimum. I don't see the logic in waiting until it starts affecting performance...I mean I understand the more miners you have the more profit you make, but doesn't the overall decreased performance (not to mention user dissatisfaction) outweigh those profits?

[Darklyspectre]

why wait so long to close registrations? you waited until the very last second before closing.. the point where cpu is 100%... 11000 workers.. and now shares or blocks will orphan because inserter will not be able to process real time.. lost profits for everyone. whereas it should have been closed last month, duringt he rise of x11 sales, system would be more stable capped off NOT as limit, and not as much stales/orphans/lost profit

It's called greed and trying to get that 5% off as many people as the system physically can handle.

The brothers doing good logical work/decisions that benefit the stability of the pool?

They never listen to us and now we just have to damn sit here with lost shares and instability while they KNEW very well when there would be problems.

[Mrrt]

Miss Cleo sez the announcement on November 17th will be that no decision has been made because reasons but we will have an answer after New Years.

[Steve Sokolowski]

why wait so long to close registrations? you waited until the very last second before closing.. the point where cpu is 100%... 11000 workers.. and now shares or blocks will orphan because inserter will not be able to process real time.. lost profits for everyone. whereas it should have been closed last month, duringt he rise of x11 sales, system would be more stable capped off NOT as limit, and not as much stales/orphans/lost profit

I have to agree here, especially as you guys have repeatedly stated you have no plans to improve the situation for several months at minimum. I don't see the logic in waiting until it starts affecting performance...I mean I understand the more miners you have the more profit you make, but doesn't the overall decreased performance (not to mention user dissatisfaction) outweigh those profits?

We're not making much profit. Most of the profit is being lost to bugs and issues. For example, the system stopped accepting shares this morning, causing us to have to pay customers for blocks that weren't submitted. There are also other issues that need to be investigated, but they require someone to be able to monitor the system at all times. While we were making money in the past, at this point we're just trying to keep the business operating until we have enough information to determine the future of the system.

I agree with you that customer dissatisfaction will start to eat into profits, which is why the November 17 deadline is set. We're not going to expand after that date, because we figure that so much negative publicity will have occurred by then that it will have harmed the business too much in the interim. It's do or die time and a decision has to be made at some point.

For the next two weeks, though, we're going to see if there are some changes we can make, like decreasing the number of coins, which ordinarily would decrease profitability but in this case may increase it by reducing the number of stale shares.

As to centaur's point, we do know exactly what needs to be done, and we actually already have code in basic testing to resolve this issue. The reason why we need a few months is because even if a positive decision is reached, we can't just wrap up our current commitments instantly. That's going to take several additional weeks, before we can then focus fully on this project and buy the equipment needed for full testing.

[ZiGen]

Today, the system reached capacity, with CPU load close to 100%, as the predicted maximum worker count of 11,000 was approached. Therefore, we closed registrations.

Unfortunately, stale shares are expected to be high because the system cannot process coin assignment in real time anymore, and therefore miners continue to submit old work because they aren't being told about new work once it becomes available.
The fix for this problem is not easy, and it involves parallelizing the system to allow coin assignment to occur on multiple computers.

Thanks for your patience as we continue to try to get advice on how to move forward. We determined today that we will be able to make an announcement on November 17 as to the future of the pool, at which time we may reopen registrations.

Hi Steve and Chris,

This is my second attempt to reach you in order to try to help with your amazing system and it's performance problems ...

For your information I am not only a highly skilled software developer and consultant, but also infrastructure expert - hardware, networking, systems engineering and integration etc., with vast knowledge and many years of experience. My "forte" is as a system architect, spent my last 15+ years in financial industry with heavy enterprise client/server systems for high frequency trading, real time data feeds, Forex markets etc.

This brief introduction was not for my own highlight, but to help you understand better where I am coming from and my point of view.

IMHO the problem(s) you are facing is not new or unknown. This is the price of business or project success, leading to fast growth, but without or very little preparation for scaling up or plan to address this growth/capacity issues. Many other have been facing such fast success/growth, but failing to resolve the Scalability Challenge quick is a recipy for failure.

The question is: what CAN be DONE NOW, right?

If I am in your shoes I would :

1. BRUTE force - 100% CPU utilization ...OK ...What are the specs of the server CPU -brand, model, # of cores, threads etc.
Could you swap the existing CPU with BEST performing CPU on the market right now, which is compatible with current socket/motherboard/system?
Or even better - I would BUILD immediately completely NEW server with highest possible CPU performance and other specs for the reasonable price:

Like - AMD Threadripper 1950X -16cores/32threads, 32/64GB memory, PCI/Nvme high speed 1/2TB SSD first level storage and large 10/60 RAID second level etc, redundant PSUs etc. I don't think that brand names are ready with such(AMD) systems yet and the price probably will be double, so best would be to be build. If you don't have such ability, I would be glad to build for you a super server, contributing my free labor. Have been doing this the last 40 years. If necessary would come to Pennsylvania too.

2. System SPLIT and duplication - load the system on the second machine AND split processing load between two of them : like half of the customers or relatively independent sub-parts of the whole system etc. Here the MOST obvious DIVISION line is the mining algorithm : X11 or SCRYPT . They are mutually exclusive even by the stratum servers, mining, coins exchange etc. Just load X11 components on one server; SCRYPT on another, apply necessary changes to front end to redirect miners/customers to the X11 or SCRYPT servers, which would be not much I guess. This would split the load in half or whatever % of miners are on X11 and Srrypt.

3 MULTIPLY and DISTRIBUTE geographically in different locations: Build 2 or more ADDITIONAL servers or PAIR X11/SCRYPT and send them to be hosted somewhere else, like I would be glad to host one in Florida, another to West Coast like San Francisco/LA or Seattle and direct most local traffic to them - South/Latin America to FL, Asia to SF/LA, Europe to you in Phily, etc. The GranatGas guy from Indonesia has less resources and abilities, but is running 3 servers US, Europe and Asia. You/we could probably do much better than him, I guess.

This is not exactly a rocket science and I believe you could avoid bigger changes like "parallelizing the system to allow coin assignment" etc , right ?

These possible steps would give Prohashing pool(s) at least 2 to 5-6 times capacity scaling right now, like days to couple weeks max, not until Nov 17 which is about 20 days from now. IMHO current situation is coming to critical level of NOT acceptable and need to be resolved ASAP. Losing 10% of 120-130K daily income is like 10-12K per day for miners and 5% of this is pool's loss = $500-600 for you. $10,000 till Nov 17 is ~ cost of 2 new servers.

That said is only MY personal views and opinions in regards to the performance problems and possible solutions.

This awesome system that you, Sokolowski Brothers, created is YOURS, your product, your business and ultimately subject of your technical and business decisions.

I still don't know why do you need a lawyer, because your technical expertise and common business sense should be enough for some/all of these possible steps. His fees are maybe worth equivalent of another 2-3 servers. I understand that government involvement or regulation(s) could effect the crypto scene, but the government is usually very slow, more over now with Trump's situation. I would NOT expect this to change any time soon by the next year or two.

Meantime if your business fail to scale quickly and soon, you might NOT NEED a lawyer at all. I hope NOT.
Just trying to be helpfull.

Thank you for you time and efforts, working on your system in behalf of all of us!

I am hoping that you will take this writing under serious consideration,

Best regards,
Zigen

PS. I am sending you PM with my name and phone#, in case you need it.

[rulovico]

I know it's none of my business , but I would give ZiGen proposal a chance