Upcoming system downtime

[-MaVerick-]

IT can be a beast sometime, so give them some room.

[Steve Sokolowski]

As Chris posted in the other thread, copying between disks is taking longer than estimated. The 2014 disks are much slower than he thought, and he expects them to be empty by 6 minutes from now, rather than by 5 hours ago, as was originally thought.

Once the disks have been copied, he expects it will be less than an hour before the system can be brought back online.

[-MaVerick-]

Thanks for the info!

[futurebit]

Thank you a lot for your Infomation...
And specially for the details...

Yes chia pushed up the prices vor HDD an NVMe extrem.

[Steve Sokolowski]

The system is now online.

The cause of the delay was that the 2014 disks were much slower than anticipated. SSDs have improved more dramatically than we thought since the old server was created on system launch day.

We will provide a full overview of how things went in about 12 hours, after we get the old equipment removed and go to sleep. Good night!

[-MaVerick-]

Hey Prohashing Team,

gratz on the job well done. Miners are running as they should for a couple of hours now, website functionality is pretty much back with a few quirks here and there. I guess there will be some bugs, cause there are always some bugs, but I have no doubt that you will smash them when they arise.

Hope you got some great sleep.

[FreeFelix]

Website seems unobtainable - just blank white page - miners seems to be working OK though

[Steven Godshall]

There was an issue with accessing the website around 5p-7p EST today that has since been fixed. Mining and earnings were not affected.

[Philipdz]

It’s still not working..

[Steve Sokolowski]

Now that this downtime has been completed, here is a final report of the progress.

There were no major issues during the upgrade except for the slowness of the 2013 solid state disks causing delays in copying, which Chris didn't anticipate in his estimates. The slow disks have been removed from the system and will be listed on eBay tomorrow.

All of the changes in this upgrade involved the database server. For customers, the biggest perceived differences will be in the response time of the website. Some queries are now returning 10 times faster than they did before, because there are more cores to process them, because the new disks are six times faster than the old ones, and because PostgreSQL 13 has optimizations that allow some queries to be processed in parallel.

There are two tasks remaining that Chris is addressing. First, Chris is reindexing the entire database. Version 13 has a new index format, which dramatically reduces disk usage and improves lookup performance in queries that use indexes. The reindexing will take a day or so and proceed in parallel to normal operations, so the full performance gain from the upgrade won't be available until tomorrow or Friday.

Second, Chris is creating a fully offline backup onto a hard drive he just purchased. While Prohashing already has offsite backups in the case of fire, those backups are connected to the Internet so they can be updated. This other backup drive will be plugged in onsite once per week, and then Chris will disconnect the USB cable once it is finished until the next backup. That way, we are completely protected from the Russian ransomware attacks that have plagued infrastructure providers in the past few weeks.

There was an unrelated issue that involved the website. We attempted to upgrade the website to the latest version of Angular - 12 - but the release failed and we rolled it back, causing a brief interruption in website service but not in mining. The issue is not reproducible in development and we're trying to figure out how to prevent website releases that fail due to build issues that don't occur in the test environments.

As always, thanks for mining with us!