Locking yourself out of the server

Ok, this is one of the reason for the previous rant

Now one of the most dreaded things that could happen is doing something so stupid that you lock yourself out of the server... obviously I did.

The server's been running for a couple of days while I got things set up. But somewhere along the lines, don't ask me when where or why, one of the installation scripts must have edited the firewall/iptables settings. Or perhaps the control panel did for some reason.

After rebooting the server remotely for the first time, I found the server ignoring web connections. Panic!!!!

A quick check with ping and SSH revealed that the server did succeed in rebooting and was working. Just not accepting requests. Hmm apache's running. Then I discovered named/bind wasn't responding to DNS requests either. Essentially, the server's not going to listen to the world except through the primitives of ICMP and fortunately SSH.

On a hunch, from many a times my Windows browers just mysteriously could not access any websites, I did a /sbin/service iptables stop. Voila, webpages were served and pop3 access granted!

I turned it back on immediately however. The idea of leaving my server naked without a firewall is just scary.

So now I know, it's the iptables table settings. So briliantly, I googled for iptables firewall howtos on the web. See previous post for why I didn't bother with man pages on this, it just stare at me when I tried.

So wonderful, I found a forum posting on a site that says "Linux Firewall (iptables)

There it was the first glorious instruction "iptables -F" and the nice explanation that this flushes all current rules. Then "iptables -p INPUT DROP", this blocks all incoming connection. Nice since we don't want people trying to connect where they shouldn't.

See, what's so hard about writing documentation this way from a logical progression? People look at it, they know exactly what it does so they can figure out for themselves when they need to use it this.

But of course, after God knows how many hours without sleep, with bleary eyes and half braindead, I was so happy to find the guide that I started typing things in, step by step without reading ahead.

The astute readers will realize what that means when I hit the enter button on INPUT DROP.

SSH stopped responding. Basically the server was dead to the world because The Noob Admin has stupidly blocked all incoming connections to a server that wasn't physically next to him. Fucked.

It was a panicked half an hour trying to figure out if I could get access to the datacenter in the middle of the night. Finally cursing and swearing at the writer for not putting a BIG WARNING about not using them on a remote machine, I saw the small print that such command line changes are NOT saved to disk.

15 minutes later, after the wonderful IDC staff, God bless them, rebooted the machine, I was back in business.

Note to self: Never fuck with iptables in realtime. Always use the conf file and make sure the first line is to allow SSH so that whatever fucks up, you can always get back in.