Virtually Amazing: Playing with my new server
You may or may not remember a couple of months ago we had a power outage where I work. Through a series of events, that power outage took down our main server and had me scrambling for a day or so to rebuild some of our infrastructure.
We have battery backups, but those only last an hour or so. If power doesn't come back on before then, the servers shut off hard. This can be damaging for the old style mechanical drives and that is exactly what happened in our case. I was able to recover some of the drives, but basically I had to just start over again.
After we notice a couple of other things that had been affected by the power outage, we placed a call to our insurance company and after some back and forth with them and a forensic interview, they paid for my brand new server.
I decided to go a bit big with this server because quite honestly, I kind of hope it is the last one I ever have to buy while I am working here. The last server was at least ten years old, so if they can get that kind of life out of this, they aren't doing too bad.
I've mentioned this before, but total cost was about $14,000 USD. It came with dual Xeon Silver processors, 128 GB of RAM, and solid state hard drives.
The solid state drives were a bit more, but I liked the idea that we might not have to worry about mechanical failures should the power go out for an extended period of time again like it did before. In the best of circumstances, once the power went out I would have shut down the server manually, but it was after hours and I was half an hour away from work at home when the power failed.
On my last server, I used a RAID configuration that allowed me to lose one drive, but if two drives went down I would be hosed. For this server I decided to go with a RAID configuration that would allow for two drives to fail without data loss.
The server also comes with plenty of room for expansion if I decided to add some more drives in the future. Additionally, on the back side of the server, there are a couple of M2 drives that allow you to install your host software and keep it off your data drives.
I was on the fence about buying this server. I had been looking at Amazon Web Services and getting some pricing for that. Ultimately I decided I wanted to have my own hardware. There is something about being able to put your hands on the machine and being able to spin up a server whenever you want to. All without having to worry about pricing and bandwidth and all of that.
If you haven't figured it out, this server is acting as virtual host for a wide number of individual servers that I use on my network. My next big decision was whether or not I wanted to go with VMWare or the Microsoft Hyper V package for my virtualization environment.
I already have a license for Hyper V, but I ultimately ended up going with VMWare because the education pricing for that was extremely cheap. In addition to that, a good number of my other servers that were still running are hosted on a VMWare ESXi host, so I wanted to be able to move them easily between hardware should the need arise.
It took me less than a day to get ESXi installed on the new hardware and a couple of new servers up and running. Now, I am working getting an SCCM server up and running so I can use it for imaging of the end user desktop computers in my environment.
I plan on writing more about that in a future post.
So far, I am really happy with this new hardware. It is super easy to manage and at least for now not having to worry about running out of space is a huge weight off my shoulders.
I already have four servers running on this hardware and just the other day I used one of the tools from VMWare to move a virtual server from one of my old pieces of hardware to this one. The process was really smooth.
I just had to shut down the server instance on the old hardware, launch the software on my desktop. Point to the old server, pick the virtual machine that I wanted to move, and then tell it the address for the new piece of hardware I wanted to move it to. In about five to ten minutes, I was booting up that same server on the new hardware.
Talk about slick!
I've been working in IT for close to 25 years now and I have to say, virtualization is probably one of the coolest developments I have seen in that time. It's totally game changing from a hardware management standpoint. It gives businesses a huge amount of flexibility and cost savings over the old way of doing things.
No longer do you need to have twenty different services running on a single server, or twenty separate servers running a single service with the rest of the system resources going to waste.
It's pretty amazing how far we have come!
Sports Talk Social - @bozz.sports
I don't deal with servers directly, but we all rely on them. I expect there's lots of cool development going on. Do you run Linux on your servers?
I do. I just spun up an Ubuntu Server 22 client the other day. I am using it to run a piece of software called Cacti. It is supposed to help me monitor my network devices like switches and UPS's using SNMP. I haven't gotten it fully configured yet though.
I expect you use a lot of monitoring apps. Can you check the status on your phone?
Haha, not as many as I should. No, I don't have anything like that. I don't think Cacti has a native app. It runs on Apache, so I could probably access the website via my phone, but then I would have to get them to NAT a public address through the firewall so I could access it offsite and I don't really like doing that any more than I have to.
Oh I remember that day very well!
I bet you are like a wee boy Christmas excited to play with his new bike!
On the plus side, it looks like it really is a great upgrade and should last till your retirement.
Yeah, it is pretty sweet. I think I have four virtual servers running on it right now and I am not even using 20% of the total resources. It's crazy!
Awesome, should make things easier now then and more reliable!
For sure!
That looks a great piece of kit. I am surprised you didn't go for AWS though, its all the rage and pricing is supposed to be competitive.
The pricing I saw wasn't competitive at all. It was pretty expensive even for a small instance. Maybe I was reading it wrong. Their pricing is not easy to decipher at all. So many tiers and so many options.
I guess they are mostly interested in the larger customers with a larger spend
Ain't that always the way...
I mean mechanical hard drives shouldn't fail just because of a power failure. Unless the drive was just one power cycle away from a failure anyway. Or there was an associated power surge (but that could affect a lot more than the drives).
I think the biggest limitation for the number of VMs you can run has been the amount of memory you have for a long time now. But with 128GB you should be able to run a bunch. Not all that many years ago when limited to a 32-bit OS and 4GB of RAM it was a lot more limiting. But so much has changed since then.
I looked at the server again. It came with 256GB of RAM. My bad :) I'm pretty sure what happened is the disk was spinning when the power failed. The two drives plattered and then started throwing errors. I was in the server room earlier that week and I took a picture and all of my drives were green, so it was either the hard shutdown or they were close or a combination of the two. I'd probably lean towards a combination of the two.
Oh the bitter sweetness of a server crash. Dark side, you have (or had) a lot of work to do to get everything back up and running, lot's of stress and people breathing down your neck.. Bright side, NEW! FAST and FUN. hehe. Enjoy playing around and configuring everything.
This one was actually not too bad. I expected to have a lot more people bugging me. If we had our old superintendent, he would have been hounding me non stop. Our new one just let me do my job and get things back up and running. It was amazing.