DDoS and other outages are P1, so lunch break and other activities has to be cut short and action the list ASAP.
- Notify the client
- Raise Cloud Flare status if applicable
- Check logs for possible attack points, get rid of them with quick changes in nginx config, see https://www.nginx.com/blog/mitigating-ddos-attacks-with-nginx-and-nginx-plus/ for possible tricks, for example user agent elimination or deny all on attack points.
- Check load, number of requests per IP, number of running processes, database load, mysql `show processlist;`
- Inform the client about the taken actions
- When the attack is over lower the alert status
- Inform the client about lowered alert status
Nginx configuration
You can limit the rate at which nginx accept incoming requests to a value typical for real users. For example, you might decide that a real user accessing a login page can only make a request every 2 seconds. You can configure nginx to allow a single client IP address to attempt to login only every 2 seconds (equivalent to 30 requests per minute):
limit_req_zone $binary_remote_addr zone=one:10m rate=30r/m;
server {
# ...
location /login.html {
limit_req zone=one;
# ...
}
}
Certain cases picking out IP addresses also work:
location / {
deny 123.123.123.3;
deny 123.123.123.5;
deny 123.123.123.7;
# ... }
On specific addresses:
location /foo.php {
deny all;
}
When user agent is the same or similar:
location / {
if ($http_user_agent ~* foo|bar) {
return 403;
}
# ...
}
Fail2ban
Install, configure and use fail2ban if possible / necessary.
Top connections
To check the top connection hoarders:
$ netstat -tn 2>/dev/null | grep :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head
Install tcpkill if you don't have it
On centos it's yum -y install dsniff --enablerepo=epel
$ tcpkill host 106.120.173.141
Comments
0 comments
Please sign in to leave a comment.