Server downtime next week

[Chris Sokolowski]

Hi everyone,

Early next week, we will be taking the server down for an extended period of time to transfer our entire database to solid state storage. Until now, we have been running our database on a hybrid SSD/HDD arrangement, but with the new block explorer and the increase in hashrate, the database can no longer sustain the I/O needed for reliable performance. This manifests itself in long website load times and random bugs.

The maintenance will take between 12-24 hours. The reason so much time is needed is because the database is extremely fragmented on the hard drive and copying the data will be slow. We expect to perform the transfer on Monday, when the solid state disks are expected to arrive, but if their arrival is delayed, then we will perform the maintenance on Tuesday. I will provide more specific estimates when I get more details on the shipping.

Thanks again to rootdude for all his advice on solid state disks. We will be running a RAID 10 of 1 TB Samsung SSD 850 Pro disks, which I estimate will offer at least a 10x performance improvement over the current arrangement. With the new, faster database, we look forward to the many possibilities that lie ahead.

Thanks,

-Chris

[kires]

That's great; Hooray for SSDs! Just out of curiosity, why not use dd to copy the drive over, fragmentation and all, then tweak the partition size to match the available space if needs be? From what I think I know, SSDs give no f---s about fragmentation, and garbage collection and/or trim will take care of whatever needs handling, anyway. Just for the record, I'm pretty sure I'm missing something, just not sure what it is.

[Steve Sokolowski]

Unfortunately, we can't do that. I'll give an explanation of the arrangements.

Right now, everything is on a CacheCade setup of 256MB flash memory -> 2x 240GB RAID 1 SSDs -> 4x 4TB RAID 1+0 HDDs.

The new setup is: 256MB flash memory -> 4x 1TB RAID 1+0 SSDs for the database, plus 2x 240GB RAID 1 SSDs for operating system and other servers, plus 1 4TB hard drive for running backups.

Most of the existing 8TB array was unused, and the storage structure is different. Rather than everything being on one disk, the new system will have the operating system and virtualization on one set of slow SSDs, and the database with SSDs directly connected to a virtual machine to avoid the 3% overhead of virtualization. While it may be possible to shrink the partitions, we want to reconfigure the system entirely to put the database on the different, faster storage.

In the end, we'll have the OS and other VMs operating off the small drives, the database on the fast drives, and the write ahead logs being compressed and written out every five minutes to the hard drive, so that we can restore to any point in time if the database fails or there is a "hacking."

[Chris Sokolowski]

Due to a delay in receiving the SSDs, the server maintenance will begin on Tuesday at 10 AM EDT. I have used the extra time to prepare the server for the transfer, so the actual downtime should be shorter than predicted. I hope to have the server back online by 8 PM EDT.

[kires]

Good luck, sir! I hope you have a boring day.

[Chris Sokolowski]

Everything is proceeding as planned, and the site should be back up around 9PM EDT.

[Chris Sokolowski]

Copying the database back is taking longer than anticipated. I will have the server online as soon as possible.

[tucsondirect]

Better Grab some coffee Chris, Did you mirror the drives before the migration attempt?

[kires]

Well, so much for boring.

[Chris Sokolowski]

The server is coming back online right now. I only got about 2 hours of sleep last night. The database backup I made on-site was corrupt and I had to restore from a secondary backup, which needed to be downloaded through the internet from another server. No data was lost, it just took a long time.