Archive
Backing up your Hyper-V cluster with Trilead
It’s been a while since I wrote about any VM backup software (or for that matter ANYTHING) but I wanted to take some time to update my readers on a great little software from the folks at Trilead. I have been using this software since the early version 3 releases and they lost my attention when they began to require the VMware guests be shutdown before they were backed up. Now in version 4.0.37 I noticed that they have added that functionality back along with Hyper-V support! (actually Hyper-v had been added earlier than 4.0.37 but now that I have a Windows 2012 test machine I noticed that it was supported).
They have added a two server restriction so you will only be able to connect to a maximum of two servers but as long as you don’t have more than two nodes in your cluster you can use the Free version.
You can backup all of your virtual machines running on your Hyper-V cluster and you can even move them by backing it up and then restoring it to another host (unfortunately you cannot use the replication feature in the free version although who would want to – migrating from one hyper-v to another in a cluster is easy).
You can only perform a full backup on any of these vms but since this is a free product and you can do it while the guest is running I think this software is a great plus you can check them out at http://www.trilead.com/Editions. Where else can you find a software that will allow you to backup a vm – convert it and then import it to a hyper-v cluster that is free?
How to extend your NTFS partition with a vmdk file on ESXi
If you are using a large windows file server on your VMware server you will probably end up with a low disk space warning one day. You could choose to resize a partition in-place there but if something goes wrong you are kinda hosed unless you have a backup. You could copy it but not everyone has that kind of disk space so I like to add an ISCSI datastore to my ESXi server and choose to copy the VM to another datastore first. Then I can try to extend the copied vmdk file using a helper vm. It required a combination of stopping the vm, copying the file, renaming the original vm and bringing the new extended vmdk file online to extend it. Once I verify that everything was working our plan was to then remove the old VM once we were completely sure it was working and to free up the space for other projects.
In ESXi vsphere Shutdown vm
Login to ESXi server using putty
cp /vmfs/volumes/datastore1/WindowsXP /vmfs/volumes/datastore2
Here we see the copied file is exactly the same at the original
In ESXi vsphere
Rename Old VM to .old (this changes the VM name not the directory name)
Open helper vm and edit settings, close
Open settings again and increase the hard drive to desired size
Attach copied vmdk file from new datastore directory to existing VM as another drive
In helper OS
Open command prompt and type diskpart
List volume
Select volume (new volume number)
extend
quit
In ESXi vsphere
Create new VM (custom) with old name and do not attach hard drive
Add hard drive and use existing vmdk from new datastore directory
When you boot the machine it might do a check disk to resize the partition or after it boots it might need to restart with the new device added.
Verify IP address assignments make changes after initial boot if you do not assign IP automatically using static DHCP addressing.
Reboot and verify operation and once you are satisfied to can delete the old VM.
Backing up your Hyper-V cluster isn’t expensive
It’s becoming very easy to backup virtual machines these days, there are even free versions becoming available (http://www.trilead.com/ – for VMware and http://www.altaro.com/ for Hyper-V – there is also script for XenServer available from Jeff Riechers) but one of the problems with using a high availability cluster is affordable backup. Most companies take the approach that only large companies are using cluster file systems and so they can afford the cost of an enterprise solution.
Well I am here to tell you about a software that is coming from the UK called Hyperoo (www.hyperoo.net) that is as little as 79 bucks! The software is a rsync client/server model and it works with VSS snapshots so there is no downtime. First you setup a server that will receive and house your backups. These are basically the VHD disk files and a Virtual Machine subdirectory that contains the configuration. (Most people agree that it is easier to restore a VM by recreating the configuration and using the VHD file so this is all that is really needed.)
We used a local USB drive on our workstation to save the backups. This make it easier to deploy to other VM servers like XenServer.
We also find that it’s easier to install VMs on our Hyper-V servers by making an ISO directory on the shared filesystem for installations.
You can see that we have several VM directories on our shared filesystem that we will backup along with our ISO directory. Since this system only deals with a delta backup (files that change) it will not take up very much space on the backup drive.
Now we will install and setup the client portion on one of our Hyper-V servers. (Fortunately we will only need one license and one server to contain the backup software because the shared file system contains all of the VMs whether they are running on this particular Hyper-V server or not) We will need to add a few Server-Core components in order to install our software. From a command prompt on the Hyper-V server type the following;
dism /online /enable-feature /featurename:NetFx2-ServerCore
dism /online /enable-feature /featurename:NetFx2-ServerCore-WOW64
dism /online /enable-feature /featurename:NetFx3-ServerCore
dism /online /enable-feature /featurename:NetFx3-ServerCore-WOW64
(I can’t remember which one I used and I do remember something about the WOW64 is for 32bit machine but maybe someone else can elaborate?)
Once you have updated your server you can run the Hyperoo client from the command prompt. (BTW there is a great project running around that gives you an applet to add remove applications, run updates, etc. called CoreConfig from codeplex at http://coreconfig.codeplex.com/). When you have finished the install you can open the app by typing ‘hyperooconfig’ from the Program files (x86)\Hyperoo directory. You should get an app that looks like this (without the configurations).
We have configured our storage root and all sub directories and setup a schedule to run at midnight every night. We have also setup email notification so our backup status report is emailed to us (see below).
Backup Profile: Backups
Machine name: HYPERV2
Remote Backup completed successfully.
Number of files transferred: 7
Number of files skipped: 20
Matched data: 26.70 GB
Skipped data: 6.22 GB
Changed data: 1.33 GB
Total sent: 1.33 GB
Total received: 23.11 MB
Remote backup start time: 7:07:06 AM
Remote backup end time: 7:43:24 AM
Backup duration : 00:36:17
As you can see on our Gigabyte network using a USB 2.0 backup drive we manage to syncronize our backup copies of almost 27Gig of data in just over 35 minutes! This custom rsync algorithm is very quick. We could even run this job every hour resulting in very little lost data if disaster ever occurs.
For the cost of a few Core 2 Duo or Core 2 Quad computers and 79 bucks you can have the benefits of High availability between two live VM servers and back them up in the event of disaster recovery. All this for under 1000.00 makes Hyper-V a great solution for any business that needs 5 9s (99.999%) of uptime.
Deploy Hyper-V Cluster image using PlateSpin Migrate 9.1
The new platespin migrate has come out last August and we thought it was time to test drive it on the Hyper-V SP2 cluster we have created. Unfortunately you cannot directly connect to a cluster or a Hyper-v host so you have to setup your new environment in what PlateSpin calls a semi-automatic mode. This requires that you boot an ISO image on the target virtual machine making sure to set it up with as little or more resources as your source (hard drive, space, RAM, etc.)
One of the nice things we notice right away is the use of a debugger for determining why the boot loading/registering process won’t work. It also has the ability to inject drivers after a failed migration which is very handy too! 
(As an aside – we noticed that our XP test failed to boot stating that there was not enough memory when in fact we were using 512MB and platespin only requires 384MB. They do have a smaller memory boot ISO available if you need it but in our test platform this issue was resolved by using the new dynamic memory feature of Hyper-V SP2.)
After the controller is downloaded and started we wait a few minutes for the discovery process to finish and we have a new target that shows up in our Platespin migrate client.
You can see the new target appearing on the bottom of our display above. The boot ISO images register with the PlateSpin Migrate server using a randomly generated host name.
We can then deploy our Windows XP image using an image that we have captured to our new Hyper-V target.
Want to recover some space on your ESXi server–read on
I wanted to tell my readers about a great tool I found from those creative people at Quest Software called vOptimizer. This tool is an essential part of any virtual administrators toolbox.
You start by connecting to your ESXi server and create a Project. This project will use a boot ISO uploaded by the windows client wizard during initialization and it will allow you to run a disk optimization that can do things like removing old temp files all the way to resizing your VM disk!
We started by connecting to a Windows 2003 VM on our ESXi 5 server that was grossly oversized at 40G. This project was setup to remove every excess file (including removing the swap file) and once we had successfully ran the disk optimizer we were left with a running server that was only about 11G in size. The entire process only took about 40 minutes of downtime.
We checked the ESXi summary before it was completed (left) and after (right).
Now I used to run a gparted iso and change the size of the disk and then expand the NTFS partition to make a disk larger but this trick allows you to make it bigger or smaller.
Unfortunately is does not work for LVM disks and with most of the Linux distros moving towards this default installation we are still looking to resolve this issue (gparted still doesn’t support it either).
(BTW this tool cannot optimize a VM if there is a snapshot so you will need to back it up first and then remove the snapshot before you can run the optimizer. This is true for windows vms as well).
After 30 days this tool will run as a free tool and only optimize 2 VMs but it is still a handy tool to have in your toolbox.
Want to do more with just a single phone line–ask us how.
Some of my readers may already be familiar with Marc Spencer and his creation of Asterisk (for instance anyone who has called some of the Canadian Government services) and may know about Alison and her beautiful voice but for the rest of you, I want to talk about phone systems today.
I have been working with Digium and Asterisk products for more than a decade now and I am hear to tell you that you can run a small business with just one phone line and have all the benefits of voicemail, IVR and faxing for free* (this assumes that you know how to program and maintain your system).
AsteriskNOW is a CentOS Distribution that is already configured with a web based front end that allows you to create a phone system using any PC based hardware. (We even run it in a virtual environment on a Windows HYPER-V Cluster using a Linksys SPA3102 – see our article here).
It can act as a host for so many applications;
| VoIP Gateway | Skype Gateway |
| IP PBX | Call Center ACD |
| Conference Bridge | IVR Server |
| Voicemail System | Call Recorder |
| Fax Server | Speech Server |
One of the best reasons I found to run it at my home office was to force Telemarketers to press a digit to speak to me. I haven’t been interrupted at dinner in more than a decade thanks to one of these servers.
If you need to press a button when someone calls you to receive a fax then you can use of these machines. It can be setup to email you a fax instead of wasting paper when you receive those fax advertisements. No more waiting for someone to send a fax to you. You can use the same number you use for your phone calls. The system can recognize that a fax is coming in and automatically send it to you. You can even set your system send your received faxes and voicemails to your inbox as attachments.
Hardware is available to have multiple phone lines and multiple telephones connected so you can create a call center complete with screen pops for inbound and outbound calling.
Setup an outbound dialer to make telemarketing calls to your potential clients and play them a message informing them about your specials.
Maybe you need to record all of your calls for quality purposes or just to ensure accuracy while you or your employees are doing business.
The system can be programmed to work with almost any Contact manager or database you currently have. Imagine creating a phone system for your clients that call often for information. They can use your new Interactive Voice Response system to make inquiries, place orders or even make payments.
All of these features can be setup on just one one phone line today and your system can grow with you. Comes complete with updates and add-on modules to enable all of this functionality so what are you waiting for?
With more than 15 years in the communications industry working with IVR/Fax/Data I can tell you that this phone system can be setup to do almost anything. More information is available by sending an email to sales@the-techy.com to arrange for a free consultation. We can speak on the phone or visit you at your office and discuss how this phone system can help you.
Migrating AsteriskNOW VMs from ESXi to Hyper-V
I wanted to touch base with some of you and share my experiences with virtualization. For those of you who have followed some of my posts you may be asking ‘how can I export/import’ between the different virtual hosts. (I have been asking myself this question for a few years now). As some of you are no doubt aware there are a number of tools available for a price that will enable the user to make this migration with as little time as a can of coke and some tim-bits but for the average tech out there these tools are expensive. Some of them work great between one platform or another but not all of them work perfectly. As I am a man on a budget and choose only to spend money when I have to I tend to lean towards tools that are free that, although they may take time and expertise to operate, make my bottom line, and that of my clients, easier to swallow.
Part of the problem with changing hypervisors is that apps like Xen and Hyper-V use vhd files while vendors like VMware have made their own type called vmk. There are some tools that will convert from one file type to another but they don’t take into consideration the different architectures that you made the VM image with and plan to run it on. Then there are drivers – those can blue screen a windows system or hang a Linux based system if you aren’t careful.
Well I recently tried to migrate some vms from my ESXi 5 machines onto a Hyper-V cluster and made some interesting observations.
OpenSUSE has a hard time converting from vmk to vhd because of the sixe of the resulting vhd reporting a superblock size difference. If you run resize2fs on the disk you can actually get the vm to boot on the Hyper-v system.
(Might require the Integration tools – more on that later)
AsteriskNOW runs on CentOS and has no problem running on Hyper-V clusters but I warn you – you must set the MAC address to be static on a Virtual Network Adapter in order to have the IP come up on the other Hyper-V hosts in the cluster. (Incidently I have used VMs to run asterisk in my home office for almost a year now without any issues).
We need to do a few things to add the Hyper-V Integration tools for Linux to make this complete. You need to install the GCC compiler and any dependancies to be able to compile the Hyper-V tools (storage, VM bus, synthetic NICs, etc.) Once installed you should see the boot screen resemble something like this
Now that we have confirmed that the guest is back online let’s shut it down with the tools.
Next we can add the VM to the list of HA apps on the cluster. Once we have done that let’s start it up again to confirm it can boot.
Now if one of my vm hosts goes offline (loss of power, network, etc.) then my PBX is dynamically moved to a new host and brought online again with a short downtime (about 15 seconds) shown below.
It sure is nice to have AsteriskNOW running on a free cluster now (without the cost of running enterprise software 
.
A look at Windows Hyper-V as a Cluster
I decided it was time to look at the free windows version of Virtual Machine software that comes with Windows Server 2008 R2 called Hyper-V. The platform has a little larger of a footprint compared to to others but the administration tools should make it easier to manage from our new Windows 7 platform. This proved to be challenging but now that it is setup I am quite happy with the results.
Through the Hyper-V Configuration panel (blue window), I did the following:
- Changed the default Host Name (I changed my hostname several times*)
- Restarted server to apply the computer name settings
- Changed the IP to static addresses
- Enabled RDP support
- Allow MMC Remote management
- Enable Windows Powershell
- Restarted server to apply changes
- Configured Remote Management to allow WinRM and relax Firewall settings
- Enabled an extra firewall setting (through the command Netsh advfirewall firewall set rule group=“Remote Volume management” new enable=yes) on my administrative machine for managing the disks through a remote MMC snap-in
- Joined the domain (Windows 2008 R2 Domain created on a separate server on the network)
- Rebooted without changing the machine name again.
- Added the domain Administrator to the local Administrators group (option 4 of the Hyper-V Configuration tool).
Next we are going to use iscsicpl (the control panel added to R2) in order to add our iSCSI target volumes to our first hyper-v cluster server. When we type this command into the hyper-v machine we are asked if we want the Microsoft iSCSI service to start automatically so please say yes.
(Now we used a free version of DSS to configure our third PC as a storage server running iSCSI RAID 5. There are a number of software/hardware options and a multitude of configuration methods but we chose this because it supported multipathing – it should also be noted that we previously connected and formatted both of these disks as NTFS)
Enter the IP address of our target and click ‘QuickConnect’ Once you are in the quick connect window click on each of the disks and click connect. They should read Connected before we proceed. Press ‘Done’ and then press Ok to close the applet.
You should now be able to connect to the Disk Management of that server from your management system and configure the drive letters. I changed my quorum drive (the Witness disk) to use letter Q and the main storage cluster volume to use S.
I then ran the command "start /w ocsetup MultipathIo" to setup multipathing and rebooted.
I then repeated the process (all except step 9) on the second virtual server and so on (if you had more than 2 machines considered for your cluster).
After I finished the setup on the second server I connected back to the first server with Server Manager and verified that Multipath was installed while I waited for the second machine to restart.
![clip_image001[4]](https://mobilesecurityanalyst.com/wp-content/uploads/2012/01/clip_image0014.png "clip_image001[4]")
Once I verified that the second machine had reconnected to the same drive letters and that it too had Multipathing enabled we could turn on the Failover Clustering Feature (option 11). You should also see that feature listed in the server manager when you connect to both.
Next step was to connect to each of the servers and enable an External (Virtual) Network for the VMs to use. I just used the default settings and enabled the same NIC that the management interface was using.
Next step was to run the Failover Cluster Manager and Validate our two servers for a cluster. (* We found that this method would not allow us to connect and I suspected that it was a name resolution issue that is caused when the machine name is changed without joining a domain. I went back to the first step above and removed the computer from the domain and rebooted. Then I joined the domain AND changed the name of the computer before I restarted to fix it).
After validating that the two servers could be used in a cluster (*) we created the cluster.
(* When using only one set of network cards I found that there was a warning stating that the network was a single point of failure but I choose to disregard that)
Once we have the cluster created we can turn on the Cluster Storage Volumes and add the CSV that we created earlier.
Now Click on Services and Applications in the Failover Cluster Manager and look on the right side for an option to create a Virtual Machine and make sure to use the ‘Store the Virtual Machine in a different location’ check box and enter ‘C:\ClusterStorage\Volume1’ as your path to store the Virtual Machine. The configuration will automatically use the Witness Disk and the vhd will reside on the Clustered Volume.
Building a Network File System for use with Virtual Machines
One of the most important parts to a computer and one that is the hardest to maintain is the file system or disk. Most systems use a local file system and that is fine for the majority of applications but inevitable the disk fails and you lose your data or OS.
RAID (redundant Array of Inexpensive Disks) has helped prolong the local file system and some people have great success with various hardware and software versions of this approach but now that we have LAN speeds of 10G the advent of using a remote file system that is shared amongst our servers and workstations seems our best solution.
I wanted to use the new iSCSI approach to communicating with a remote array but it seems more and more efficient and popular to use an almost 30 year old approach called NFS.
There are a few file system operating systems available for free to use that allow the user to setup a computer with hard drives and some type of RAID configuration so I took an old P4 X86 computer I had to use for my new (old) file server.
I had run Windows 2003 on this but I found that the OS overhead coupled with the poor NFS/iSCSI support forced my to run it with Linux/BSD variant instead. I know that Open Filer would run on a small partition on the hardware RAID controller I had but the Adaptec 2100 was not supported in the 32 bit version of OpenFiler 2.3. (I have used it for clients with an HP rack mount server without problems and it runs surprisingly stable too). I choose to grab the new version of FreeNAS that uses the FreeBSD kernel and it would prove a surprise to be able to run with all my old hardware. The only caveat is that I could not run it on a USB/Compact Flash drive and had to settle for an old IDE hard drive to run the OS. 
It comes as no surprise that I can setup a RAID5 drive of approx. 100G using some old 36G SCSI drives and put in a Broadcom NIC to be able to run VMs using this remote NFS server.
With an old 2.4G Generic motherboard and less that 1G of RAM anyone can have safe online storage that can enable your VM servers to run more efficiently with less resources.
A look at the new Xenserver 6
After we rebuilt a new workstation for the office we were able to start experimenting with some new disaster recovery ideas. These tests would involve some of the new VM server configurations and Xenserver 6 has made it to the top of my list.
Once of the issues with VMservers that everyone has is updates. It’s quite a benefit running more than one server on a single piece of hardware but it’s quite another when you have to patch the VMserver and take them all offline too!
One of the nice features of the new Xenserver 6 is that upgrades to the pool of VMservers can be done using the XenCenter console with the push of a button. Everything from uploading the patches to rebooting the VMservers is done for you.
We tested our first patch on two fresh installs of XenServer 6 and the process took about 8 minutes from the start to the finish.
We then imported a VM into the master and updated the pool again with two patches in succession to see the time it would take to migrate the VM back and forth to maintain the uptime. Unfortunately this product does not allow migrations when using local storage. Even when we took into consideration the time to suspend the VM and spin it up again the patches only took 15 minutes.
Lastly we moved the storage to an NFS host (FreeNAS) and created shared storage. We added another patch and started the process while we left the lab. The updates took less than 10 to reboot the same machines AND migrate the working XP machine from one server to the other. Best of all is the uptime – there was no appreciable lost of available IP as shown in this simple ping test. 
(Notice the point when we began the migration from one server to another in the pool is shown by the increase in time to approx. 10ms)
I really like the ability to run VMs with maximum uptime and that is good news for small business owners and support personnel like myself. That means we don’t have to patch at 4 in the morning anymore! 
Security Analyst - Jeff Soehner CISSP, GXPN,GPYC,GMOB,GCIH 