Upcoming system downtime

News updates about the Prohashing pool
Forum rules
The News forum is only for updates about the Prohashing pool.

Replies to posts in this forum should be related to the news being announced. If you need support on another issue, please post in the forum related to that topic or seek one of the official support options listed in the top right corner of the forums page or on prohashing.com/about.

For the full list of PROHASHING forums rules, please visit https://prohashing.com/help/prohashing- ... rms-forums.
User avatar
-MaVerick-
Posts: 137
Joined: Wed May 12, 2021 3:12 pm
Location: Germany

Re: Upcoming system downtime

Post by -MaVerick- » Tue Jun 01, 2021 5:06 am

IT can be a beast sometime, so give them some room.
User avatar
Steve Sokolowski
Posts: 4585
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Upcoming system downtime

Post by Steve Sokolowski » Tue Jun 01, 2021 7:15 am

As Chris posted in the other thread, copying between disks is taking longer than estimated. The 2014 disks are much slower than he thought, and he expects them to be empty by 6 minutes from now, rather than by 5 hours ago, as was originally thought.

Once the disks have been copied, he expects it will be less than an hour before the system can be brought back online.
User avatar
-MaVerick-
Posts: 137
Joined: Wed May 12, 2021 3:12 pm
Location: Germany

Re: Upcoming system downtime

Post by -MaVerick- » Tue Jun 01, 2021 7:23 am

Thanks for the info!
futurebit
Posts: 11
Joined: Sat May 18, 2019 1:42 pm

Re: Upcoming system downtime

Post by futurebit » Tue Jun 01, 2021 8:16 am

Thank you a lot for your Infomation...
And specially for the details...

Yes chia pushed up the prices vor HDD an NVMe extrem.
User avatar
Steve Sokolowski
Posts: 4585
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Upcoming system downtime

Post by Steve Sokolowski » Tue Jun 01, 2021 8:30 am

The system is now online.

The cause of the delay was that the 2014 disks were much slower than anticipated. SSDs have improved more dramatically than we thought since the old server was created on system launch day.

We will provide a full overview of how things went in about 12 hours, after we get the old equipment removed and go to sleep. Good night!
User avatar
-MaVerick-
Posts: 137
Joined: Wed May 12, 2021 3:12 pm
Location: Germany

Re: Upcoming system downtime

Post by -MaVerick- » Tue Jun 01, 2021 4:15 pm

Hey Prohashing Team,

gratz on the job well done. Miners are running as they should for a couple of hours now, website functionality is pretty much back with a few quirks here and there. I guess there will be some bugs, cause there are always some bugs, but I have no doubt that you will smash them when they arise.

Hope you got some great sleep.
FreeFelix
Posts: 13
Joined: Mon Apr 12, 2021 5:24 am

Re: Upcoming system downtime

Post by FreeFelix » Tue Jun 01, 2021 6:42 pm

Website seems unobtainable - just blank white page - miners seems to be working OK though
User avatar
Steven Godshall
Posts: 94
Joined: Tue Mar 03, 2020 11:04 am

Re: Upcoming system downtime

Post by Steven Godshall » Tue Jun 01, 2021 7:07 pm

There was an issue with accessing the website around 5p-7p EST today that has since been fixed. Mining and earnings were not affected.
Philipdz
Posts: 6
Joined: Tue Jun 01, 2021 4:41 am

Re: Upcoming system downtime

Post by Philipdz » Tue Jun 01, 2021 8:45 pm

It’s still not working..
User avatar
Steve Sokolowski
Posts: 4585
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Upcoming system downtime

Post by Steve Sokolowski » Wed Jun 02, 2021 9:15 am

Now that this downtime has been completed, here is a final report of the progress.

There were no major issues during the upgrade except for the slowness of the 2013 solid state disks causing delays in copying, which Chris didn't anticipate in his estimates. The slow disks have been removed from the system and will be listed on eBay tomorrow.

All of the changes in this upgrade involved the database server. For customers, the biggest perceived differences will be in the response time of the website. Some queries are now returning 10 times faster than they did before, because there are more cores to process them, because the new disks are six times faster than the old ones, and because PostgreSQL 13 has optimizations that allow some queries to be processed in parallel.

There are two tasks remaining that Chris is addressing. First, Chris is reindexing the entire database. Version 13 has a new index format, which dramatically reduces disk usage and improves lookup performance in queries that use indexes. The reindexing will take a day or so and proceed in parallel to normal operations, so the full performance gain from the upgrade won't be available until tomorrow or Friday.

Second, Chris is creating a fully offline backup onto a hard drive he just purchased. While Prohashing already has offsite backups in the case of fire, those backups are connected to the Internet so they can be updated. This other backup drive will be plugged in onsite once per week, and then Chris will disconnect the USB cable once it is finished until the next backup. That way, we are completely protected from the Russian ransomware attacks that have plagued infrastructure providers in the past few weeks.

There was an unrelated issue that involved the website. We attempted to upgrade the website to the latest version of Angular - 12 - but the release failed and we rolled it back, causing a brief interruption in website service but not in mining. The issue is not reproducible in development and we're trying to figure out how to prevent website releases that fail due to build issues that don't occur in the test environments.

As always, thanks for mining with us!
Locked