ESX and virtual machines losing time and hanging

I ran into a nasty bug between VMware ESX and VMWare Fusion. The
VMWare guys had no idea about it and after 3 hours on the phone with
them with little progress I started experimenting on my own and found
the solution.

Remember that I migrated my VMWare Fusion hosted VMs to ESX using the
VMWare converter tool. Prior to conversion I removed all snapshots
created by VMWare fusion.

The symptoms were that the machines under ESX were losing time and
network connectivity intermittently with no log entries on ESX and no
errors on the machines other than the massive time jumps. You could
see the machines "disappear" for 2+ minutes at a shot.

Turns out the issue was VMWare Fusion's autoprotect feature. I didn't
disable that on the VMs prior to migration and while ESX doesnt
support that feature it appears to break it. ESX was creating some
kind of snapshots frequently and there was no way at ESX to disable
this functionality.

My solution was to use VMware converter to go back from ESX, load the
machine into VMware fusion, turn off the Autoprotect feature, then re-
convert the machine back to ESX. Since then everything has been perfect.

The VMWare support people were friendly but not helpful and despite
the obvious client hangs and lots of snapshots getting created by ESX
were unwilling to admit it was an ESX issue. Obviously if my fix above
fixed the issue then it was a VMWare issue. I made no changes to the
guest operating systems.

VMWare Fusion -> ESX

Finally made the leap at work and got an ESX setup. Then had to work
through converting 5 VMs from an OSX VMware fusion environment to ESX
server. A few observations along the way:

1) VMware needs a TON of help on marketing. Their product names,
portfolio etc are confusing as all get out.

2) Get the "VMWare vCenter converter standalone" app (Windows/Linux
only) to do the conversions

3) Since you can only run on Windows or Linux you'll have to move your
VMs to somewhere that the app can access them for import

4) The VMs must be shut down. Also the snapshots don't seem to import
so remove them first (saves space)

5) As you go through the wizard for the converter, make sure you
choose to make your disks "thin" versus "flat" in the last step. The
default is flat which will eat your ESX disk space.

6) For the ones that wont convert with a "the object or item referred
to could not be found" error, downgrade the image in VMWare fusion and
then repeat the process.

7) If you're moving around Linux machines this way you may need to fix
the networking config. For us this was:
rm /etc/udev/rules.d/70-persistent-net.rules*

8) Their web access for ESX just gives a "503 Service Unavailable". It
seems out of the box this doesnt work and ssh into the console doesnt
work.

In the end everything moved and we're up and running on a real
environment. The tired desktop with flaky fusion is now retired.

Yet another OSX Server'ism

Today our Open Directory service lost its mind. At least thats what it
seemed like. Turns out what started it was a configuration change we
made a couple days ago. We enabled SSL for LDAP via the Server Admin
tool. Turns out SSL auth is broken in OSX server without some fixes.
It also turns out that once you click this button, it appears that
local client apps like Workgroup Manager start using SSL to
communicate and when you turn this back off they don't stop using SSL.
In other words, once you go SSL you can't go back just by shutting it
back off, so you're forced to fix the SSL issue.

Anyway, here's what you need to do if you're having SSL issues with
Open Directory on OSX server (10.5):

Add:
olcDisallows: bind_anon
to:
/etc/openldap/slapd.d/cn\=config.ldif

then: sudo killall slapd

Also see: http://www.afp548.com/article.php?story=20071203011158936

To test on a client:
ldapsearch -v -x -W -D
"uid=<auser>,cn=users,dc=<host>,dc=<domain>,dc=<com>" -H ldaps://
<host>.<domain>.<com> -b "dc=<host>,dc=<domain>,dc=<com>"

Replacing the things in <>'s with your appropriate information. You
can test this with and without the user ID section to see if anonymous
access is allowed.

If you're using a self-signed cert you may need to do this on the
client:

Edit:
/etc/openldap/ldap.conf

Change:
TLS_REQCERT demand
To:
TLS_REQCERT allow

From there you should be able to ping your LDAP server (make sure you
allow access through firewalls etc).

Another OSX Server-ism

I was making a minor change on OSX server today, just adding another
machine to DNS. I found out the hard way that if you insert a comment
with parenthesis "()" in it in the description fields OSX server DNS
crashes in interesting ways. The GUI crashes and eventually all of DNS
comes down, specifically the line that crashed this piece of OSX server:

machinename IN HINFO "Virtual Machine running on anothermachine
(description)" "more text"

The same line without parenthesis works just fine.

Makes me wonder if anyone is using OSX Server for real prime time
operations.

Cisco ASA and ASDM

Cisco has a nice client side java app, ASDM, for managing their
firewalls. Unfortunately if you go to the web page for the firewall (https://
<ip address of firewall>) and click "Run ASDM" then you just end up
with a downloaded file "asdm.jnlp". If you have the less-secure
setting of open files after download it will likely open up an XML
document in Xcode or something like that. So how do you run the ASDM
on the Mac?

Do this at the command line:

javaws asdm.jnlp

From there it will ask you where to save the .app bundle and then
next time you can just run that application.

OSX Server - What works and what Doesnt

So after 2 months of using OSX server, here's the update:

What we're using it for:

1) Central file server - I often have to fix permissions on the shares as someone will copy things into a share and the file will be read only to others in the same group. Since the UI is messed in Leopard (see previous post) you have to fix this through a VNC connection on the server which means you need to be an admin. Apple really needs to fix that.

2) DHCP - this is working, although some of the engineers have complained that the OSX DHCP server seems slow.

3) DNS - no real complaints here

4) Mail - The UI around the mail setup was weak and broken. Has anyone tested that? Some things we had to do:

    /etc/postfix/master.cf -> uncomment submission inet n       -       n       -       -       smtpd
        enables port 587

    /etc/postfix/main.cf -> add tls_random_source = dev:/dev/urandom (gets rid of some error messages)
        "no entropy source specified with parameter tls_random_source" 

    /etc/imapd.conf -> add tls_ca_file: /etc/certificates/wasabi.nasuni.net.crt (gets rid of more error messages)
        "TLS server engine: No CA file specified. Client side certs may not work"
  
        http://www.corpmac.co.uk/2008/09/30/tls-no-ca-file-specified-reason-and-solution/

They also don't let you add aliases other than Groups so you need to:

sudo vi /etc/aliases
sudo newaliases

Also note that when you stop the mail server and restart it through their UI (i.e. after you changed some settings) you often lose the first email that gets sent after restart. Nice.

Generally i'd recommend using someone else's mail server. The experience was not an "apple" one.

5) Open Directory - Apple uses this for all the user/group management and I havent had to touch it.

6) Radius - We use this for Cisco VPN authentication. Note that OSX server didn't really support this in the UI, so you need to follow some instructions.

7) Software update - saves multiple macs downloading to the same spot. Note that the first time you turn this on its really painful as it brings in a lot of updates.

8) Backups - the server time machine's itself and presents itself as a time machine target for any clients to use.

What we're NOT using it for:

1) Wiki - I used the Apple Wiki quite a bit. Its VERY limited. The WYSIWYG is nice, but you don't always get what you see and it can do some odd formatting stuff. If you want to do more advanced stuff you get thrown into HTML and the HTML is cluttered and hard to manage. You can't do a lot of basic stuff and eventually I punted and went with the tried-and-true MediaWiki which i'm very happy with. Its running as its own VM.

2) Web Hosting - I didn't really even try this. I wanted a server that wasn't hosting all the stuff above to be our external web host. So I created a standard Linux VM and am hosting the site via Apache.

3) Firewall - We have a decent Cisco firewall so I didn't try the OSX server firewall.

4) Our source code control/bug tracking etc is off on Linux VMs.

OSX Server bonding issues

We had some crazy issues with our OSX server setup. When we had clients join the server we had odd permission problems, synchronization issues with passwords etc. After much mucking around it turns out it was mostly our fault. Our OSX server is also a target for SSH from the outside (not on the normal port 22 and requiring certificates). To support logging in I had created some local user accounts on the OSX server machine, and for convenience I had given them the same names as the server accounts.

DONT DO THAT

OSX server gets very confused if you have local accounts with the same names as the server accounts. And you don't need the local accounts anyway as you can enable the server accounts to login/get a shell through the admin tools. Deleting all the local user accounts and enabling the ones we needed to login helped a lot of the issues.

The Leopard client still does odd things when you're looking at group/user permissions (and ACLs) on network shares. Seems that Apple messed this up in the UI in Leopard so that if you look at them you see things that just don't make sense as the users/groups are not getting copied over from the server and displayed correctly. Under the covers of the UI the right things seem to be happening but it can be confusing for the users.

OSX Environment and IE

In our vision for an All-Mac office we sort of expected to be able to pull that off without any bumps. I mentioned previously that I ran into early bumps of some office environment management apps that required windows. Later I found out that our bank (Silicon Valley Bank) and our payroll group (ADP) both have web sites that are not friendly to non-Windows/Internet Explorer environments. I proceeded to setup a new VM with Vista Home ($199 for the full install) for our office admin to run just for IE for these sites but otherwise to use the Mac side of things. When you run into mainstream companies that lack support for non-Windows platforms with their websites you really get a sense of the size of the Linux/OSX market on the business side of things.

Another thing i'm amazed at is that Vista Business sells for $300, but I can buy a full Netbook with Vista Home for $300, or a Dell Inspirion 531 with Vista Home for $249. Vista Home sells for $199 for a fresh install which is required for VM use. So the hardware is worth $49. You can setup a VM for $199 or you can have dedicated hardware for $249. While I went the VM approach mostly because I dont want more hardware to deal with (power, cooling, all that) the economics just don't make sense.