Tuesday 30 September 2008

Wake on LAN problems - Dell 2950 servers and Broadcom NICS

Although I've been using vmware for over 3 years through various incarnations, I've only just been in a position to exploit the power-saving features in v3.5. 

At first glance it looks pretty straight forward - click the "enter standby" option and sit back and watch. No.

First off, you need to have Wake on LAN (WOL) supported on your vmotion NIC.   This isn't necessarily something you think of when setting up the patching of the virtual server - Intel PT1000 Quad Port NICs only support WOL on the first port (Dual ports also).  Others in the range might be different, but the VT1000 NICs have their own issues in vmware depending on the version of vmware installed.

Most of the servers I work on are Dell x9xx series which have Broadcom 5708 onboard NICs.  Looking in Virtual Center at the NIC configuration shows whether VMware thinks WOL is available.  I was a bit confused when I found for some servers it was available and for others it wasn't.  Dell support went through the usual steps - update the firmware, update the NIC firmware, etc. - to no avail.

I tested the WOL feature anyway with the AMD Magic Packet utility and the MAC address of the NIC and lo! it woke up.  Curiouser and curiouser.

After more research by Dell, one of their engineers discovered source of the issue (hurrah!) in the version of the Ethernet Controller Hardware.  an lspci command (run as root) reveals the version number:

05:00.0 Ethernet controller: Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T (rev 11)

Rev 11 is the culprit and Rev 12 servers correctly show WOL available in vmware.  Dell have escalated the issue to vmware so hopefully there will be some kind of resolution soon.

So in summary:
  1. Make sure vmotion is enabled on a NIC that supports WOL
  2. Check your broadcom NICs are Rev12 and above.
  3. Put your Host in a Cluster
  4. At least one other Host in the Cluster must be on (this does the wakeup on the others that are in standby)
  5. Enable WOL in the NIC boot bios

Blogged with the Flock Browser