Status as of Sunday, December 17, 2017

[Steve Sokolowski]

Good morning!

• I finally feel like we're making progress in getting the system towards the premier experience we want it to be. The support ticket queue reached zero yesterday. Thanks to Chris and Constance for helping resolve all the open support tickets! While we can't promise that tickets will be resolved promptly during the holiday week, and the holidays will prevent us from hiring a support representative immediately, we hope that we will have someone whose sole job is to answer tickets trained by February. We believe that support will set us apart from other companies in this industry, and I myself stopped using Coinbase to sell bitcoins because I couldn't get in contact with a customer service representative.
• We also plan to issue a release this afternoon to address some stability and network connectivity issues. I think we'll be able to reduce the WAMP bandwidth usage by 80% with this release, and we're hoping that the large number of channel publications when a block on a difficult network like litecoins is found will have been the cause of the connectivity issues.
• The current plan is to spend next week continuing to focus on the mining server, improving stability, connectivity, and reliability; and preventing bugs like the one that cost all that money last week.
• During the holiday week, I think we'll have reached a point where we'll be satisfied the mining server is providing the experience we want it to and will then move our focus onto profitability. One of the ways we can improve profitability is by adding new coins, and adding new exchanges that offer those coins. We also think that we can spin up more mining servers and let each server be more computationally intensive in assigning coins to miners. We had to make some concessions earlier that we can now reverse.
• After the holidays, we may need two more positions - a full-time engineer, and a part-time accountant. The engineer will be tasked with some of the improvements that customers have been asking for but which have not been bugs, like implementing payout coins like XRP that don't follow the bitcoin API specification.
• By the way, for those interested, the ETH blockchain is now about 400GB. We're going to have to take it offline during the holiday week to replace the 512GB solid state disk with a 128GB SSD on top of a CacheCade WD Red. The entire setup will cost about $90, and with compression, will probably last for another 2 years. That should wake people up to the idiocy of the Core's argument that the bitcoin blockchain, which is smaller than ETH's, will somehow be too expensive to store for an ordinary person.

[Jamin]

Sounds like great news on all fronts.
As a brief aside, you mention payout for coins that are somewhat troublesome. Could I request NEO as part of that?
Cheers

[Aura89]

Good morning!

• We also plan to issue a release this afternoon to address some stability and network connectivity issues. I think we'll be able to reduce the WAMP bandwidth usage by 80% with this release, and we're hoping that the large number of channel publications when a block on a difficult network like litecoins is found will have been the cause of the connectivity issues.

Have you guys thought about, to reduce bandwidth on the website anyways (if needed), to not plaster everything onto the main page?

Maybe have the live-profitability and twitter feed on there, but have separate tabs (clearly labeled, in your face as to not cause people to not understand how to get somewhere) for all of the other areas of the website? That way when people are constantly loading up the website, even just to get to the forums or their own login page, it doesn't constantly send data people aren't looking at, taking up bandwidth? As well, people accidently leaving the website open in a tab, using up bandwidth for the "live coin status" and "live found blocks" etc.?

If bandwidth isn't an issue in any way for the website, then disregard my comment, but if it is, might be something to look into.

[Steve Sokolowski]

Good morning!

• We also plan to issue a release this afternoon to address some stability and network connectivity issues. I think we'll be able to reduce the WAMP bandwidth usage by 80% with this release, and we're hoping that the large number of channel publications when a block on a difficult network like litecoins is found will have been the cause of the connectivity issues.

Have you guys thought about, to reduce bandwidth on the website anyways (if needed), to not plaster everything onto the main page?

Maybe have the live-profitability and twitter feed on there, but have separate tabs (clearly labeled, in your face as to not cause people to not understand how to get somewhere) for all of the other areas of the website? That way when people are constantly loading up the website, even just to get to the forums or their own login page, it doesn't constantly send data people aren't looking at, taking up bandwidth? As well, people accidently leaving the website open in a tab, using up bandwidth for the "live coin status" and "live found blocks" etc.?

If bandwidth isn't an issue in any way for the website, then disregard my comment, but if it is, might be something to look into.

The issue isn't bandwidth, it's CPU usage. In general, everything in our system is CPU-limited.

In this case, I think what is happening is that the systems reach 100% CPU usage and drop packets, when a new block arrives for a large coin and a lot of work needs to be sent to miners. My theory is that if we reduce the size of the messages, then CPU usage will drop, but we don't know if that's true until we actually deploy it.

[greaseman316]

Steve i dont know if this will help or not but i did a little testing last night to do some comparisons with my miners and i noticed something that maybe might free a little bottle neck. this is only for x11. when new miners are connecting with no static difficulty set the server starts them out at a difficulty of 4. and slowly progresses upward. heres what i found and it seems a bit alarming seeing the overall hash rate for x11

Difficulty of 4. my miners were submitting a stupid amount of shares and server accepting between 300 to 400 average per minute.
Difficulty of 8 submitted shares decrease a bit but not by a lot, only a few hundred per minute. accepted shares decreased slightly to around 150 to 200spm.
Difficulty of 16 things seemed to mellow out a bit and come more inline with normal rates here of 50 to 75 per minute accepted shares.
Difficulty of 32 looked normal and accepted shares where more inline at about 2 to 3 every 15 secs.

I can go on as i tested for about 4 hrs last night. but point i am trying to make maybe the starting difficulty needs to be raised. if for some reason you have to restart servers there is no way it can keep up with roughly 5k d3s all coming online and just totally swamping the servers. even a few th/s can swamp everything, if some one with a lot of hash power has an issue and restarts their miners.

just something to think about
G

[Steve Sokolowski]

Steve i dont know if this will help or not but i did a little testing last night to do some comparisons with my miners and i noticed something that maybe might free a little bottle neck. this is only for x11. when new miners are connecting with no static difficulty set the server starts them out at a difficulty of 4. and slowly progresses upward. heres what i found and it seems a bit alarming seeing the overall hash rate for x11

Difficulty of 4. my miners were submitting a stupid amount of shares and server accepting between 300 to 400 average per minute.
Difficulty of 8 submitted shares decrease a bit but not by a lot, only a few hundred per minute. accepted shares decreased slightly to around 150 to 200spm.
Difficulty of 16 things seemed to mellow out a bit and come more inline with normal rates here of 50 to 75 per minute accepted shares.
Difficulty of 32 looked normal and accepted shares where more inline at about 2 to 3 every 15 secs.

I can go on as i tested for about 4 hrs last night. but point i am trying to make maybe the starting difficulty needs to be raised. if for some reason you have to restart servers there is no way it can keep up with roughly 5k d3s all coming online and just totally swamping the servers. even a few th/s can swamp everything, if some one with a lot of hash power has an issue and restarts their miners.

just something to think about
G

Hi greaseman,

OK. I'll increase the starting difficulty to 16 and see if that makes any difference. Thanks for the tip!

-Steve