« The hip bone has millions of transistors in it | Main | License to Compute »
Virtual Machine’s are not the answer to everything
By farseeker | May 15, 2009
I love Virtual Machines. Consolidating servers, saving space, saving electricity, saving emissions, saving the world.
There are dozens of virtualisation products at the server end of the market. SuSE and Novell have one, there’s the venerable VMWare ESX, Microsoft Virtual PC, HP and Dell even ship servers with VMWare ESXi HyperVisor installed on a flash chip. There’s VirtualBox, Parallels, Virtuozzo, the list goes on.
At our datacenter, our sysems administrator started making the move to virtualisation 12 months ago. He moved four terminal servers and two web servers onto his first application server (A dual-quad-core Xeon with 24gb of RAM for those who were wondering). Since then we’ve moved countless terminal servers, web servers, backup domain controllers, certificate authorities, DNS servers, nameservers (yeah they’re the same thing I know), and backup databases onto Virtual Machines, over about 6 application servers (all similar specs to the one above).
From four fully-packed racks down to less than one rack. Which has meant that he’s been able to afford the rack space in a better co-location faculity and thus we’ve gone from a 2Mb copper link to a 1000Mb fiber link directly into a major backbone and MAN network in Sydney. Brilliant.
When I was on holidays recently (see my post about my trip to Adelaide a few weeks ago), our primary database server (which was not a virtual machine, it was a dedicated dual-quad-core Xeon with 4Gb of RAM and RAID10 SAS drives) went to hell in a handbasket, and it took everything with it.
Now, we’re not stupid. We had live mirroring to our backup servers, and it only took them a few minutes to take the backup servers out of recovery mode and flick the switch to turn them into production machines (we don’t have the licenses for the version of MSSQL that does this for you, as a few mins of downtime is not the end of the world).
Boy though, were we in for a surprise. Even though it was only for a few days, and technically the application servers were superior in specs than the database servers, we fielded countless complains about the speed of EVERYTHING. And of course with the database server under load, it stole resourceses from the other machines on that server.
After the replacement server arrived, we moved everything back to that server (except for one database) and everyone was happy again. Why did we leave one database? I don’t know – I think we just forgot. It’s not a heavily used database, however yesterday someone who was using it complained of incredible slowness when saving records. According to the profiler it was taking FOUR SECONDS to execute a single “update” command.
We thought, rather than take it offline and move it in the middle of the day, I’ll see if I can bandaid it and find the query that’s obviously screwing it up. Well, I found the query, analysed it, and it was perfect. Indexes were OK, it wasn’t waiting on any locks, SQL analyser found nothing wrong with it.
So, after 20 minutes of farting around with the query, I decided to just bite the bullet, kick everyone off, and move the database to where it was meant to be in the first place. So I did, people were grumpy while it was offline (it may have been slow but at least it worked), but I think we’ll survive.
The result? The statement that took 4143 milliseconds to run on the virtualised server – took 45 milliseconds on the proper server. Thats almost a 100x (or 10,000%) increase in speed. On a machine with less ram (4gb vs 24gb) and slower processors.
Topics: General | 1 Comment »
June 6th, 2009 at 11:08 pm
That was one reason we never seriously looked into virtualisation at my previous employer; For a lot of things, yes it can work: But databases, and various other mission-critical services, it just doesn’t cut it.
Besides: We seem to be going round in circles with the whole mainframe shenanigans again