Well, it finally happened to me...I guess it was only a matter of time.  Thankfully it wasn't a full blown take your system down type hack, more or less it was an inconvenient defacement type.  I thought I had the system pretty well locked down too, and actually I had, they got in via a program I use to track website statistics for all our customer sites, Awstats.  I was one version out of date and took a hit for it.  Some punk bastards in Brazil basically passed a bogus command to this program running on a website, then followed it with the system commands they wanted to have run.  Since the webserver runs as a powerless user anyway they couldn't do much more then deface a few files, but that yanked me back into the reality of how easy it is to be hacked.  I've since upgraded the application to the newest version, but I still haven't enabled it to the general public and probably won't.  Instead I'll make them use their email username and password to see the statistics of their website. 

I now have an in depth knowledge of how the Awstats hack works and I would be happy to help anyone make sense of it, see it in action and perform cleanup and damage assesment.  Our biggest saving grace is a restrictive set of firewall rules and that I run the services as low permission users.  I think much more damage could have been done had I not had them in place.  I have some basic information on these creeps, but I don't want to break the law to get back at them as tempting as it is....and since they are in Brazil it wouldn't do me much good to call the cops.  Learn from my mistake...if you're running Awstats <=6.3 UPGRADE!