As per Gea's ideas about using an all in one ESXi box I wanted to test a setup here at work. We already have an existing NAS here at based on Nexenta Community edition 3.0.4 and I'm thinking about converting it to an ESXi host/NAS/internal SAN. Thus the test setup. I get to the point of installing vmware tools on the fresh nexenta install and reboot only to be greeted by this unpleasant message: vmxnet3s0: macalloc failed A brief bit of searching seems to indicate that there although vmware tools can be installed, there is no working vmxnet3 driver for current versions of Nexenta. This makes me sad. Anyone seen this or know a solution?. VMXNET 3 — The VMXNET 3 adapter is the next generation of a paravirtualized NIC designed for performance, and is not related to VMXNET or VMXNET 2.
It offers all the features available in VMXNET 2, and adds several new features like multiqueue support (also known as Receive Side Scaling in Windows), IPv6 offloads, and MSI/MSI-X interrupt delivery. VMXNET 3 is supported only for virtual machines version 7 and later, with a limited set of guest operating systems: o 32 and 64bit versions of Microsoft Windows XP, 2003, 2003 R2, 2008,and 2008 R2. O 32 and 64bit versions of Red Hat Enterprise Linux 5.0 and later o 32 and 64bit versions of SUSE Linux Enterprise Server 10 and later o 32 and 64bit versions of Asianux 3 and later o 32 and 64bit versions of Debian 4 o 32 and 64bit versions of Ubuntu 7.04 and later o 32 and 64bit versions of Sun Solaris 10 U4 and later Notes:.
Jumbo frames are not supported in Solaris Guest OS with VMXNET 2 or VMXNET 3. Fault Tolerance is not supported on a virtual machine configured with a VMXNET 3 vNIC in vShpere 4.0, but is fully supported on vSphere 4.1. Update; I have setup two test all-in-one systems at work just to try out the idea. They are both just some simple setups we had on the bench, nothing fancy.
Q9550 CPU's with 8GB ram on a desktop board. One has SE 11 and the other has Open Indiana 148, both with Napp-It. Both systems seem to be working with the VMXNET3 driver after installing vmware tools on each.
I will do some more testing with tomorrow and try to figure out how it performs. NOTE: I only gave the internal SAN VM's on each machine 4GB as I was limited in ram and wanted to load another VM or 2 on each box.Each has an Intel SASUC8I pass though to the storage VM. Some quick initial thoughts and/or problems I ran into: 1) I could NOT create the NFS datastore on the ESXi host until after I had add a VMkernal port to the vSwitch in ESXi. I just stumbled across this since I am not very familar with it. It also was not mentioned in your all-in-one guide Gea. Is this what I was supposed to do or did I miss something somewhere?
I had already configured the NFS sharing for the folder on each ZFS setup so it wasn't that. 2) After I finally did create the NFS datastore on the OpenIndiana machine I could not then create a VM with it's disc on the NFS store. It would give an error when trying to create it. However, the SE 11 setup worked right away. After comparing the two in napp-it I saw that the se 11 setup had it's shared folder permissions at '777+'.
The OI machine had it's folder at '775.' I changed that to 777 and creating a virtual machine on the datastore then worked fine! Not sure what the '+' means on the 777 though or why the two setups ended up a bit different. 3) I only got the point of creating a win 7 vm on the SE 11 box. This box has 4x500GB WD RE3 drives in RAID 10 (two mirrored vdevs). I ran a CrystalDisk Mark bench inside of windows on the system drive just to see what it did.
Right away I noticed the sequential writes seemed slow. I got ok reads at about 170MB/s but both Seq Write and 512K random write was something like 30MB/s. Wondering why that might be? Could it be the CPU, or the only 4GB ram? Or do I have some tuning or fiddling to do? (I don't know how to do any ZFS tuning yet) I will be doing more testing tomorrow. Also I will be trying the same type of all-in-one setup on a 'real' server tomorrow.
Supermicro board, X3430 and 16GB ram. PS!- My eventual plan if this works out well is to convert our two production machines over to this type of setup. I would like to set them up where I can replicate the contents of the SAN in the primary all-in-one over the the SAN in the secondary/backup all-in-one.at very regular intervals. It doesn't actually have to syncronous, but short term async would be great (like syncing every minute or so). Question is what is the best way to do this with the above software? An auto service to run rsync? I have no experience with ZFS-send and only have just recently dipped my toes into rsync (using auto service in Nexenta).
2) After I finally did create the NFS datastore on the OpenIndiana machine I could not then create a VM with it's disc on the NFS store. It would give an error when trying to create it. However, the SE 11 setup worked right away. After comparing the two in napp-it I saw that the se 11 setup had it's shared folder permissions at '777+'.
The OI machine had it's folder at '775.' I changed that to 777 and creating a virtual machine on the datastore then worked fine! Not sure what the '+' means on the 777 though or why the two setups ended up a bit different. 3) I only got the point of creating a win 7 vm on the SE 11 box. This box has 4x500GB WD RE3 drives in RAID 10 (two mirrored vdevs). I ran a CrystalDisk Mark bench inside of windows on the system drive just to see what it did. Right away I noticed the sequential writes seemed slow.
I got ok reads at about 170MB/s but both Seq Write and 512K random write was something like 30MB/s. Wondering why that might be? Could it be the CPU, or the only 4GB ram? Or do I have some tuning or fiddling to do? (I don't know how to do any ZFS tuning yet).
I will be doing more testing tomorrow. Also I will be trying the same type of all-in-one setup on a 'real' server tomorrow. Supermicro board, X3430 and 16GB ram. PS!- My eventual plan if this works out well is to convert our two production machines over to this type of setup. I would like to set them up where I can replicate the contents of the SAN in the primary all-in-one over the the SAN in the secondary/backup all-in-one.at very regular intervals. It doesn't actually have to syncronous, but short term async would be great (like syncing every minute or so). Question is what is the best way to do this with the above software?
An auto service to run rsync? Http tunnel client keygen for mac pro. I have no experience with ZFS-send and only have just recently dipped my toes into rsync (using auto service in Nexenta). Thanks Gea for those updates to the all-in-one guides!
Ok more testing today. All is fine with the original vmxnet interfaces. Those are all up and running fine. Internal virtual network is 10Gb between all VM's.
That is nice but on to the next thing. I am testing the ZFS san today with a pool made of just a single Intel 160GB SSD.
I share it NFS and add it as a datastore to the ESXi box. This is an all-in-one test box running on Xeon X3460 CPU and 10GB ram for the OpenIndiana VM. With one single Win 7 VM running off the SSD datastore I have run the same Crystal Disk bench mark as yesterday. Everything except the small random writes looks fine. 4K write and 4K(QD32) write is terrible! I get about 3.8 MB/s for both. The SSD is on a passedthrough 1068 card so ZFS has native control of it.
Why such terrible random write speed? This is a much faster CPU, more ram, and a faster (SSD) drive than I tested yesterday.