[AusNOG] DDoS Attacks - Painful and Persistent.

[Nick Brown]

All,

The overwhelming response so far has been much appreciated.

Roland Dobbins wrote:
> Have you implemented S/RTBH at your edges? If so, you can blackhole
> based upon source addresses, not just destinations.
>   
When I say we have had to blackhole destinations - we have been reliant 
on upstream providers to block the targets at their edge, not our own. 
While we can achieve this using the array of tools provided by all of 
our upstream carriers, be it BGP communities or verbally with a NOC, 
blackholing the source or destination within our network is of little 
consolation if the attack is still saturating your transit to a point 
where no legitimate traffic can traverse your network.

Our ability to blackhole source addresses has been hindered by the 
number of source, and the dynamic nature of where the attack traffic is 
originating from.
> Have you implemented NetFlow export into an appropriate analysis  
> toolset, so as to provide detection/classification/traceback  
> visibility (full disclosure; I work for a vendor which produces  
> commercial NetFlow analysis tools, but note that there are several  
> open-source tools available)?
>   
We are working through the data we have got - however for a significant 
portion of the attack time, our focus has been on restoring services or 
mitigating the effect the attack has on downstream customers.
> Do you have communication paths and relationships established with the  
> relevant folks at your peers/upstreams/downstreams/end-customers so  
> that you can reach out to them in order to get them to filter within  
> their networks?
>   
This experience has been a good lesson as to why its important to ask 
certain questions of your peers before bringing them onboard. Alas all 
of our providers to date have been very helpful - even in the event 
where the attack has resulted in load issues for a specific upstream 
carrier.
> Have you scaled and functionally bulkheaded your DNS infrastructure?
>   
We have managed to mitigate the impact the attack has had by migrating 
services to alternate locations, however at the same time we have been 
mindful not to widen our surface area.
> Have you implemented reverse proxy-caches in front of all Web-based  
> properties?
>   
While the attack continues to target the IP of our own website 
specifically, it is not targeted at a specific service, be it HTTP or 
otherwise.
> Have you implemented tcpwrappers, mod_evasive, mod_security?
>
> Have you implemented an intelligent DDoS mitigation system, or IDMS  
> (full disclosure; I work for a vendor which makes such systems).
>   
We have looked at commercial third party options, however to date the 
cost has significantly outweighed simply throwing more capacity at the 
problem.
> Have you joined the relevant opsec mitigation communities which allow  
> providers to collaborate in handling security events such as DDoS  
> attacks?
>   
In the process of this based on a couple recomendations right now. I 
have also been contacted by both ACMA and AusCERT representatives, and 
we are only too happy to share information with those who believe it to 
be relevant in either mitigating the effect on ourselves, or the greater 
Internet community.
> Can you provide details of the attack traffic/methodologies?  This  
> will help folks to provide more situationally-specific advice.
>
>   
There is a 3 Minute capture (15MB) available for viewing at 
http://mirror.as38887.net/Misc/Attack_2009-08-10_202.45.155.46.txt as 
captured earlier this morning showing some data pertaining to the type 
and volume of traffic. Despite dropping the affected prefixes earlier 
today, bringing the affected prefix back into the global routing table 
immediately shows the return of the malicious traffic.

Anyone who wants specific information on how we have and are mitigating 
this attack so far are welcome to contact me offlist for more info.