IP Blocked by stateful Firewall, now what?

I was working on a clients server who hired me to optimize it for the client-side by making the site fast to load and render in visitors browsers, and also to optimize the server itself to reduce the amount of CPU and Memory being used, as well as reduce the amount of disk reads/writes and network (like mysql) connections. So I log in via ssh for the first time to drop off some authorization_keys and set up shop.. I'm not new to this, just used to logging into servers hired to find ways to bypass the security, hackfest in a shell. It's still kind of the same deal for me, I use a lot of the exact same techniques and commands to understand the system, but instead of security issues I'm searching for ways to turbocharge the system for performance.

I feel out the system and map out the network, looking for bottlenecks and that sort of thing. Once I understood the system environment and machine I needed to learn more about the internal networking that was set up. Like what nameservers were being used, how many network interfaces were available to work with, how fast could the main server TX/RX to a 2nd server on the internal network, what kernel settings were in place for the stack, what are the most efficient routes internally, on and on and on with that lovely sort of thing..

Network Inventory

I need to work as fast as possible at mapping the network, as it can become as complex as you have time and passion for, without much benefit to show for the effort. After saving to pdf the the output of the trust commands lsof, netstat, ss, ip, ipconfig, route, arp, host, traceroute, and whois, (few greps, many seds, 1 or 2 awks) I decided to fire up Nmap from my backtrack to quickly learn what applications were running and listening on the server. I first tried a simple $ nmap -sS 64.111.114.111 (syn scan) which I kind of do from habit to make sure I can reach the host, ending the scan with a ctrl-c as soon I see output. I was planning on then running a version detection scan to give me a quick list of whats running where.

Trouble with Portscanning

This clients server had a great firewall built-in that detected the SYN portscan and added my IP address to a temporary bad list for 3600 seconds.

/var/log/lfd.log: lfd[2242]: *Port Scan* detected from 208.113.134.190. 11 hits in the last 296 seconds - *Blocked in csf* for 3600 secs
/etc/csf/csf.tempban: 1250398105:208.113.134.190::in:3600:lfd - *Port Scan* detected from 208.113.134.190. 11 hits in the last 296 seconds
/etc/csf/csf.tempip: 208.113.134.190:0:1250398105
/etc/csf/stats/iptables_log:| kernel: Firewall: *TCP_IN Blocked* IN=eth1 OUT= MAC=### SRC=208.113.134.190 DST=64.111.114.111
/etc/csf/stats/iptables_log:| kernel: Firewall: *SYNFLOOD Blocked* IN=eth1 OUT= MAC=### SRC=208.113.134.190 DST=64.111.114.111

/etc/csf/csf.tempban:1250398105:24.123.215.58::in:3600:lfd - *Port Scan* detected from 24.123.215.58. 11 hits in the last 296 seconds
/etc/csf/csf.tempip:24.123.215.58:0:1250398105


# Added to /etc/csf/csf.allow
24.123.215.58 # Manually allowed by askapache - Sun Aug 16 01:26:22 2009

# Added to /etc/csf/csf.ignore
24.123.215.58 # Manually allowed by askapache - Sun Aug 16 01:26:22 2009

CSF ConfigServer Firewall and Security

A Stateful Packet Inspection (SPI) firewall, Login/Intrusion Detection and Security application for Linux servers. Learn More..

• Straight-forward SPI iptables firewall script
• Daemon process that checks for login authentication failures for:
  • Courier imap, Dovecot, uw-imap, Kerio
  • openSSH
  • cPanel, WHM, Webmail (cPanel servers only)
  • Pure-pftd, vsftpd, Proftpd
  • Password protected web pages (htpasswd)
  • Mod_security failures (v1 and v2)
  • Suhosin failures
  • Exim SMTP AUTH
  • Custom login failures with separate log file and regular expression matching
• POP3/IMAP login tracking to enforce logins per hour
• SSH login notification
• SU login notification
• Excessive connection blocking
• UI Integration for cPanel, DirectAdmin and Webmin
• Easy upgrade between versions from within cPanel/WHM, DirectAdmin or Webmin
• Easy upgrade between versions from shell
• Pre-configured to work on a cPanel server with all the standard cPanel ports open
• Pre-configured to work on a DirectAdmin server with all the standard DirectAdmin ports open
• Auto-configures the SSH port if it's non-standard on installation
• Block traffic on unused server IP addresses - helps reduce the risk to your server
• Alert when end-user scripts sending excessive emails per hour - for identifying spamming scripts
• Suspicious process reporting - reports potential exploits running on the server
• Excessive user processes reporting
• Excessive user process usage reporting and optional termination
• Suspicious file reporting - reports potential exploit files in /tmp and similar directories
• Directory and file watching - reports if a watched directory or a file changes
• Block traffic on the DShield Block List and the Spamhaus DROP List
• BOGON packet protection
• Pre-configured settings for Low, Medium or High firewall security (cPanel servers only)
• Works with multiple ethernet devices
• Server Security Check - Performs a basic security and settings check on the server (via cPanel/DirectAdmin/Webmin UI)
• Allow Dynamic DNS IP addresses - always allow your IP address even if it changes whenever you connect to the internet
• Alert sent if server load average remains high for a specified length of time
• mod_security log reporting (if installed)
• Email relay tracking - tracks all email sent through the server and issues alerts for excessive usage (cPanel servers only)
• IDS (Intrusion Detection System) - the last line of detection alerts you to changes to system and application binaries
• SYN Flood protection
• Ping of death protection
• Port Scan tracking and blocking
• Permanent and Temporary (with TTL) IP blocking
• Exploit checks
• Account modification tracking - sends alerts if an account entry is modified, e.g. if the password is changed or the login shell
• Shared syslog aware
• Messenger Service - Allows you to redirect connection requests from blocked IP addresses to preconfigured text and html pages to inform the visitor that they have been blocked in the firewall. This can be particularly useful for those with a large user base and help process support requests more efficiently
• Country Code blocking - Allows you to deny or allow access by ISO Country Code
• Port Flooding Detection - Per IP, per Port connection flooding detection and mitigation to help block DOS attacks
• DirectAdmin UI integration
• Updated Webmin UI integration
• WHM root access notification (cPanel servers only)