More details about the NetSweeper DoS attack.After some exchanging a few emails with a developer at NetSweeper, it has become clear that what appeared to be a denial of service attack was indeed, as my NetSweeper corollary suggested, simply the behaviour of a very broken robot. The long series of requests made (3224 of them, over a 40 minute period) was a partial replay of legitimate traffic from a system running portsnap -- with the critical difference that the system running portsnap waited until one connection had closed before opening another.
Aside from the protocol violations I listed before (the rapid-fire requests from a system which is neither an interactive user-agent nor a proxy, the fraudulent User-Agent header, and the inobservance of robots.txt), a tcpdump of packets between my server and NetSweeper uncovered two serious bugs in their "categorization engine":
NetSweeper only ever sends one request per HTTP
connection, and does not send a "Connection: close" header; yet it
describes itself as an HTTP/1.1 client. To quote RFC 2616,
HTTP/1.1 applications that do not support persistent connections MUST include the "close" connection option in every message.To fix this, NetSweeper should either start including that header line in its requests or label itself as an HTTP/1.0 client. Both fixes are trivial and would have the effect of allowing Apache to close the connection (and free resources) immediately after sending its response.
NetSweeper does not shutdown sockets properly. After Apache gave up
waiting for further requests and sent a FIN packet to close the
connection, NetSweeper acknowledged the FIN (placing Apache's end of
the connection into the FIN_WAIT_2 state, and NetSweeper's end of the
connection into the CLOSE_WAIT state)... and then did nothing for 15
minutes. Finally, 917 seconds after Apache closed its end of the
connection, NetSweeper closed its end by sending a FIN packet (to which
it received a RST response, since FreeBSD drops idle half-closed
connections after a minute).
The fix for this is also very simple: There's a missing shutdown(2) call somewhere in NetSweeper's source code.
blog comments powered by Disqus