Madder than Heck

[rank1st]

:shock:
As, I sit here running FTP - reloading my server - after having fought with 1and1 support on the phone all day, I thought I would recount my experiences for all of you.

I arrived at work this morning to find my server down. After an hour of trying to figure out what was wrong (no network), I call 1and1 support to be told that my server is "locked" - blocked at the switch.

The claim was that my server was the source of a DOS attack, however I was not able to see any such information in logs anywhere and 1and1 provided absolutely no proof. The response was that my server was sending blocks of UDP requests to a particular IP address. Log analysis reveals absolutely no trace of that IP address in any logs. Based on the information that I got from support, I still think they are full of **.

Just so everyone knows, they were pretty much Nazi's about this whole thing. The only way to get the server unblocked was to re-image, absolutely no other option. They did finally allow it to be unlocked in recovery mode so that I could do an up to the minute backup. So, here I sit 8 hours from problem discovery until I can actually get to the point where I can do a restore.

-Every client on the server has called - two cancelled, all are pissed.
-In spite of the fact that they offer server re-image online you apparently still have to fax a form in if its locked (That was a 6 mile round trip - twice) The first fax apparently didn't go through, although I received a receipt from the first fax saying it did and it mysteriously showed up after I made the second trip( I know because the lady at the copy shop called to say they had just sent my second fax about 15 minutes after I returned - the second time).
-Every step of this process that involved 1and1 required calling them and prodding them along.

Personally I am planning on dropping them as soon as possible for another provider.

[sholzy]

When you've been "rooted" by someone who is "good" at what they do, they will "clean up" after themselves by cleaning up the logs, deleteing files, etc... and do it quickly. Be very cautious of any restore you do, it may contain the trojan that was used to carry out the attack.

Do you allow any shell access by clients? If they used an easy-to-remember password that could have been an entry point. I have been receiving about 1000 ssh dictionary break-in attempts daily, some of them from other 1and1 IP's. I've noticed they concentrate the attempts around the admin/password combination. I turn in the 1and1 IP's immediately with a copy of my LogWatch report, you may very well have one of the IP's I've turned in. If it was, then I apologise. There was no intent to cause problems to an innocent victim. One thing I've learned is 1and1 does not fool around when it comes to shutting down a server when they feel there has been a compromise.

Here are a few links that might help...
http://www.linuxsecurity.com/content/view/118211/49/
http://www.chkrootkit.org/
http://www.rootkit.nl/

If you were compromised, you may be a target again in the future if they think you were easy enough to break into the last time. Good luck in securing your server and keeping it secure.

[paul]

Oh trust me. Logs don't contain everything. If you're not familiar with advanced security techniques, you server probably got exploited or "rooted". When the "hacker" is in, he can delete all his traces and do watever he wants with your server. Rooted boxes are very common in datacenters.

Now, I think that 1and1 support did their best because they allowed you to access your server in recovery mode.

Otherwise, remember that you get what you pay for. Get a better datacenter that has more experienced personnel. Might I add, www.ev1servers.net.