It has been a long a frustrating couple of days.
On Monday afternoon a bit after 2pm, three emails turned up in my mail box with the ip address of this server as the subject. This made me instantly suspicious, so I ssh’d into the server and found it had been hacked. I downed it straight away to avoid further problems, but I already had problems enough. Someone using ManiaC r00tkit had got in and defaced the main web page among other things.
We ended up rebuilding the server Monday night, and got all the we pages back up, but there was a problem with email until Wednesday lunchtime. Much more work needs to be done on hardeneing the system. The new server is already more secure, and it will get better.