I was working on a clients server who hired me to optimize it for the client-side by making the site fast to load and render in visitors browsers, and also to optimize the server itself to reduce the amount of CPU and Memory being used, as well as reduce the amount of disk reads/writes and network (like mysql) connections. So I log in via ssh for the first time to drop off some authorization_keys and set up shop.. I'm not new to this, just used to logging into servers hired to find ways to bypass the security, hackfest in a shell. It's still kind of the same deal for me, I use a lot of the exact same techniques and commands to understand the system, but instead of security issues I'm searching for ways to turbocharge the system for performance.
I feel out the system and map out the network, looking for bottlenecks and that sort of thing. Once I understood the system environment and machine I needed to learn more about the internal networking that was set up. Like what nameservers were being used, how many network interfaces were available to work with, how fast could the main server TX/RX to a 2nd server on the internal network, what kernel settings were in place for the stack, what are the most efficient routes internally, on and on and on with that lovely sort of thing..
Network Inventory
I need to work as fast as possible at mapping the network, as it can become as complex as you have time and passion for, without much benefit to show for the effort. After saving to pdf the the output of the trust commands lsof, netstat, ss, ip, ipconfig, route, arp, host, traceroute, and whois, (few greps, many seds, 1 or 2 awks) I decided to fire up Nmap from my backtrack to quickly learn what applications were running and listening on the server. I first tried a simple $ nmap -sS 64.111.114.111 (syn scan) which I kind of do from habit to make sure I can reach the host, ending the scan with a ctrl-c as soon I see output. I was planning on then running a version detection scan to give me a quick list of whats running where.
Trouble with Portscanning
This clients server had a great firewall built-in that detected the SYN portscan and added my IP address to a temporary bad list for 3600 seconds.
/var/log/lfd.log: lfd[2242]: *Port Scan* detected from 208.113.134.190. 11 hits in the last 296 seconds - *Blocked in csf* for 3600 secs
/etc/csf/csf.tempban: 1250398105:208.113.134.190::in:3600:lfd - *Port Scan* detected from 208.113.134.190. 11 hits in the last 296 seconds
/etc/csf/csf.tempip: 208.113.134.190:0:1250398105
/etc/csf/stats/iptables_log:| kernel: Firewall: *TCP_IN Blocked* IN=eth1 OUT= MAC=### SRC=208.113.134.190 DST=64.111.114.111
/etc/csf/stats/iptables_log:| kernel: Firewall: *SYNFLOOD Blocked* IN=eth1 OUT= MAC=### SRC=208.113.134.190 DST=64.111.114.111
/etc/csf/csf.tempban:1250398105:24.123.215.58::in:3600:lfd - *Port Scan* detected from 24.123.215.58. 11 hits in the last 296 seconds
/etc/csf/csf.tempip:24.123.215.58:0:1250398105
# Added to /etc/csf/csf.allow
24.123.215.58 # Manually allowed by askapache - Sun Aug 16 01:26:22 2009
# Added to /etc/csf/csf.ignore
24.123.215.58 # Manually allowed by askapache - Sun Aug 16 01:26:22 2009
CSF ConfigServer Firewall and Security
A Stateful Packet Inspection (SPI) firewall, Login/Intrusion Detection and Security application for Linux servers.Learn More..
Straight-forward SPI iptables firewall script
Daemon process that checks for login authentication failures for:
Courier imap, Dovecot, uw-imap, Kerio
openSSH
cPanel, WHM, Webmail (cPanel servers only)
Pure-pftd, vsftpd, Proftpd
Password protected web pages (htpasswd)
Mod_security failures (v1 and v2)
Suhosin failures
Exim SMTP AUTH
Custom login failures with separate log file and regular expression matching
POP3/IMAP login tracking to enforce logins per hour
SSH login notification
SU login notification
Excessive connection blocking
UI Integration for cPanel, DirectAdmin and Webmin
Easy upgrade between versions from within cPanel/WHM, DirectAdmin or Webmin
Easy upgrade between versions from shell
Pre-configured to work on a cPanel server with all the standard cPanel ports open
Pre-configured to work on a DirectAdmin server with all the standard DirectAdmin ports open
Auto-configures the SSH port if it's non-standard on installation
Block traffic on unused server IP addresses - helps reduce the risk to your server
Alert when end-user scripts sending excessive emails per hour - for identifying spamming scripts
Suspicious process reporting - reports potential exploits running on the server
Excessive user processes reporting
Excessive user process usage reporting and optional termination
Suspicious file reporting - reports potential exploit files in /tmp and similar directories
Directory and file watching - reports if a watched directory or a file changes
Pre-configured settings for Low, Medium or High firewall security (cPanel servers only)
Works with multiple ethernet devices
Server Security Check - Performs a basic security and settings check on the server (via cPanel/DirectAdmin/Webmin UI)
Allow Dynamic DNS IP addresses - always allow your IP address even if it changes whenever you connect to the internet
Alert sent if server load average remains high for a specified length of time
mod_security log reporting (if installed)
Email relay tracking - tracks all email sent through the server and issues alerts for excessive usage (cPanel servers only)
IDS (Intrusion Detection System) - the last line of detection alerts you to changes to system and application binaries
SYN Flood protection
Ping of death protection
Port Scan tracking and blocking
Permanent and Temporary (with TTL) IP blocking
Exploit checks
Account modification tracking - sends alerts if an account entry is modified, e.g. if the password is changed or the login shell
Shared syslog aware
Messenger Service - Allows you to redirect connection requests from blocked IP addresses to preconfigured text and html pages to inform the visitor that they have been blocked in the firewall. This can be particularly useful for those with a large user base and help process support requests more efficiently
Country Code blocking - Allows you to deny or allow access by ISO Country Code
Port Flooding Detection - Per IP, per Port connection flooding detection and mitigation to help block DOS attacks
