How to defend against a DDoS attack

DDoS and other outages are P1, so lunch break and other activities has to be cut short and action the list ASAP.

• Notify the client
• Raise Cloud Flare status if applicable
• Check logs for possible attack points, get rid of them with quick changes in nginx config, see https://www.nginx.com/blog/mitigating-ddos-attacks-with-nginx-and-nginx-plus/ for possible tricks, for example user agent elimination or deny all on attack points.
• Check load, number of requests per IP, number of running processes, database load, mysql `show processlist;`
• Inform the client about the taken actions
• When the attack is over lower the alert status
• Inform the client about lowered alert status

Nginx configuration

You can limit the rate at which nginx accept incoming requests to a value typical for real users. For example, you might decide that a real user accessing a login page can only make a request every 2 seconds. You can configure nginx to allow a single client IP address to attempt to login only every 2 seconds (equivalent to 30 requests per minute):

limit_req_zone $binary_remote_addr zone=one:10m rate=30r/m;

server {
    # ...
    location /login.html {
        limit_req zone=one;
    # ...
    }
}

Certain cases picking out IP addresses also work:

location / {   
   deny 123.123.123.3;
   deny 123.123.123.5;
   deny 123.123.123.7; 
   # ... }

On specific addresses:

location /foo.php {
    deny all;
}

When user agent is the same or similar:

location / {
    if ($http_user_agent ~* foo|bar) {
        return 403;
    }
    # ...
}

Fail2ban

Install, configure and use fail2ban if possible / necessary.

Top connections

To check the top connection hoarders:

$ netstat -tn 2>/dev/null | grep :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head

Install tcpkill if you don't have it

On centos it's yum -y install dsniff --enablerepo=epel

$ tcpkill host 106.120.173.141