Denial of Service
March 29th, 2007UPDATE 2: In Summary – Bad Behavior
But first, I will say that I’m quite sure that this wasn’t done by the U.S. government. Why? Because a few U.S. military and government zombies were involved with the attack. If this was a U.S. government operation, I seriously doubt that they would use zombies from their own networks. I know, never underestimate the stupidity of the U.S. government, but even I don’t think they would be that stupid. Allow their machines to be compromised by a bot herder, probably a 14 year old Russian boy? Now that sounds more like it.
People with multiple propellers in their beanie caps have offered advice; everything from Apache modules to suggestions on re-writing the caching behavior of WordPress (HA!) to onion routed webservers in undisclosed locations. The best tip (in terms of ease of implementation, and chances of working), however, came from a Blue Host support tech. (Blue Host is the company that hosts Cryptogon.) I asked one of their guys to look at what happened and he suggested the Bad Behavior plugin.
I’d heard about BadBehavior, but I thought it was just for comment spam. Oh no, Grasshopper. That thing is a soup to nuts bullshit reduction system.
I like the sound of this:
Bad Behavior runs before your software on each request to your Web site, so if a spam bot does visit, it will receive nothing, and your software never runs. This reduces the amount of server CPU time, database activity and bandwidth spent on processing robots which are just harvesting your site and delivering junk.
And this:
“Online risks are becoming more complex and pervasive by the day,†says the home page of the Cyveillance web site.
Indeed, one of the risks is that your Web site might be targeted by Cyveillance.
The company says that it crawls the Internet looking for phishing scams, identity theft, illegal credit card numbers, trademark and copyright infringement, and more. It’s also been known to work on behalf of the government to spy on whistleblowers who expose waste, fraud and abuse. Some have even alleged that Cyveillance bots attempt to illegally hack into Web servers.
Cyveillance uses robots which crawl your Web site pretending to be a legitimate Web browser and completely ignoring your robots.txt file. Then it tries to figure out whether you’ve downloaded any illegal music, or said something bad about some company. It then sends you threatening letters ordering you to take your site down, even when you haven’t done anything wrong.
Bad Behavior doesn’t just block spammers. It’s meant to target any bot which overloads a Web site, attempts to hack in, delivers spam, acts in an unethical manner, etc.
By relying on analyzing the HTTP requests themselves, rather than simply the IP address, Bad Behavior has blocked Cyveillance almost from the very first release last year, regardless of what IP address range they move to in order to hide their connections. An audit of six months worth of logs shows that Bad Behavior was able to successfully identify and block Cyveillance bots even when they used previously unknown IP address ranges.
In other words, this is exactly what I need. I’ve installed Bad Behavior on Cryptogon and Farmlet…
I don’t know if this will work, but it sure seems like it will. Homeland Stupidity give this software away for free. I’m going to allocate a portion of Cryptogon’s March earnings to assist with further development of this software. Thanks Homeland Stupidity / Bad Behavior! And even more thanks to Cryptogon contributors!
– – – End Update – – –
UPDATE 1: It Stopped
It’s over. It stopped, for now. I don’t know why. Anyway, there you have it.
* Queue up the Twilight Zone theme music *
Maybe that was just the warm-up round. Now that the attacker knows how much my server will tolerate, he/she can more effectively target resources in the future? If you want to strip a bolt, it’s helpful to know how many threads it has before you start turning your wrench. Just a guess. I’m falling back on what I learned about offensive information warfare when I was in school long ago.
– – – End Update – – –
Cryptogon is experiencing a distributed denial of service attack. These aren’t junk Java spam crawlers. They’re IDing themselves as various operating systems and browsers. Each zombie runs a multithreaded get of / until BlueHost’s DOS protection system temporarily suspends my account to protect the server. The offending bots leave. Then others show up.
If you see a BlueHost page with a CPU warning, that is why.
I considered providing a list of IPs to launch strikebacks on, but I thought that fighting back might get ME in trouble. It’s also pointless. The attacker could have thousands of zombies under his/her command. The poor idiots who own the offending machines don’t even know that their systems are pWnEd by a bot herder.
I’m going to try to see what I can do about this.
Why not a plugin for WordPress? Load any single page five times in under a minute, for example, and that session is done. Is there some Apache level thing I could do with htaccess?
Anyway, sorry for the inconvenience.
You could probably do something programmatically, but the best way to stop this type of crap is at the network level. Your host should implement access rules that block access after given a certain threshold of stupidity, I mean accesses per second. I’ve been looking into doing this at my job cause I get a lot of bots hitting the ssh port on some of my servers, but it’s a bit beyond my powers at the moment.
Well, this really frosts my shorts. I know what I’m gonna do about it. Kevin can call it a Denial of Service Donation. All y’all Cryptogoners should follow suit. Teach those sons of bitches.
Ideally if you were going to write a DOS protection code you wouldnt actually want to prevent the user from downloading .. better would be to make each subsequent attempt take a bit longer, passing back keep-alive messages to the client .. these shouldnt take up too much resources, but will tie up each client of the botnet into spinning on itself doing nothing (since it THINKS its still downloading the page)
Kevin,
See my email. I sent you:
1) Some info on Apache modules to stop/slow DoS attacks.
2) a .htaccess to stop the majority of bad bots from hitting your site.
3) a .htaccess to allow you to delegate parts of your site off to other servers (with bluehost, or elsewhere)
4) PHP code for some extreme caching, that I use on my site. If your problem is DB/CPU/Memory, then this will save you.
You may have to tackle the problem further upstream, and block the IP’s on the router/firewall. Apache accessing .htaccess files, or PHP caching, or any other server based method will still consume resources. Saying that, its possible to overload a router as well, but the limits are beyond what you can do with shared hosting, and any tweaks ‘on the boxen’.
You may have to talk to your server admin at the hhosting company, and provide them with a list IP addresses. They may help, may not, depends on how well you ask 😉
midnight,
I’ve worked for a hosting provider who took lots and lots (and lots and lots and lots) of these attacks, usually gracefully. If not gracefully, we improved so the next time it didn’t hurt.
I have to assume they have good pipes. Usually attacks like this are taking advantage of the fact that the target pages take resources to load.
Without knowing anything about their provider, but knowing how larger hosting setups are done, I’d have to assume it goes something like this:
Large pipes coming in. Probably GigE fiber from their provider. The GigE fiber would then be distributed by at least something like Cisco 3500’s, where each web server would have their own 100Mb/s links. It’s quite likely that they are using GigE fiber between their switches. It’s cheap enough for the last few years, and easy to implement.
To overwhelm a machine with traffic, they’d hit it with upwards of 80Mb/s of traffic. Since he reported seeing the IP and UserAgent in the logs, it wasn’t a flood on bandwidth, it was a flood directed towards the web server.
Most sites will become crippled when they take too much traffic. They’ll reach the limits for the database, cpu, or memory fairly quickly. Pages, like these, are very time consuming, and will usually run a server out of resources long before they hit the maximum number of allowable connections. The max connections are set to (theoretically) keep the server from getting overwhelmed, but those limits are usually set for normal traffic, where there are a lot of small requests that don’t take up a lot of time.
…. and ….
If it had been a bandwidth based attack, the provider would be freaking out, because it wouldn’t be effecting just Cryptogon, it would effect every site hosted on that part of the network. It’s one thing if one webmaster is screaming because something is broken, but in a shared environment such as this, there are probably several thousand webmasters hosted on just this segment.
A friend got an account at another provider (sorry Kevin, they were much cheaper), to host just the images and A/V content of his site. We didn’t need any features, other than being able to host images. 🙂 They do provide shells, and due to normal Unix permissions, I was able to see the password file. They had about 150 users on just that server, and they were still allowing new signups to it.
You may wish to consider creating a “Tor hidden service” hosted from an undisclosed server location.
“Tor allows clients and servers to offer hidden services. That is, you can offer a web server, SSH server, etc., without revealing your IP to its users. In fact, because you don’t use any public address, you can run a hidden service from behind your firewall.”
Please note the following:
http://www.au.af.mil/info-ops/iosphere/iosphere_fall05_fraser.pdf
@ JW Smythe
mod_evasive would be just the thing! This is shared hosting, though, so I don’t have permission to install Apache modules.
This wasn’t an issue of overwhelming the host’s network at all. Not even close. Fully a CPU issue on the box.
I discussed DDOS protection with BlueHost, but I don’t want to talk about the details here. They know the score.
The helpful BlueHost tech told me to look at the Bad Behavior plugin for WordPress as that thing has DOS protection that might work.
Hmmm. I like the sound of this:
http://www.bad-behavior.ioerror.us/documentation/benefits/
And this:
http://www.bad-behavior.ioerror.us/2006/12/15/cyveillance-bad-behavior/
There. I’ve installed in on Cryptogon and Farmlet…
This looks very good. Let’s see how we make out now.
[…] post an update on the original story about this, and if you want to leave comments, go for it there. Posted in Announcements, Technology | […]
True to your word, I did receive your contribution. Thanks!
For the type of denial of service attack you are seeing, where the intent isn’t to deliver spam, but to cripple you specifically and intentionally, Bad Behavior might not (yet) be the best thing. The sort of attacks I’ve seen and dealt with in Bad Behavior involve spammers delivering blog comment and trackback spam. On occasion they try to deliver far too much at once.
So it might work when someone’s targeted you, and it might not. It does give me something to put on the target list for future development, though.
Hi Michael,
Thanks for checking in. We’ll see how it goes. The tool is stopping lots of comment spam already, and I’ve noticed a total dropoff in the garbage crawlers, insurance, mortgage stuff that was showing up in the logs. (I’m also running Akismet 2 for comments.) Even if this might not stop a specific attack, clearly, I needed to be running Bad Behavior.
User JW Smythe above let me know about mod_evasive which would have cut the attack short at the Apache level:
http://www.zdziarski.com/projects/mod_evasive/
I’m not root on the box so I can’t add/remove Apache modules.
I don’t know how trivial, or not, it would be to implement this type of thing at the plugin level… Maybe a separate DOS-defense tool might be interesting, with user selectable parameters. I’m sure you don’t have enough on your plate already. 😉
Best,
Kevin
An Apache module is the right place for that. It could be done from the PHP level, but keeping data from one request to the next, while remaining fast, is non-trivial at that level, while an Apache module could easily do it. This is, not coincidentally, one of Bad Behavior’s biggest challenges; if I could keep data from one request to the next easily, I could block a lot more bad stuff.
[…] Cryptogon activity is now well within the non-trivial range, and Blue Host has been excellent. (Recent denial of service attack survival story.) Farmlet traffic is increasing as well. Each Blue Host account is capable of hosting six completely […]
I too am having the same problem and have tried everything. Caching, bad behavior, and stopping bad referrers – I am going to look for a plugin that might further help.
If you’ve got any advice for me I’d certainly appreciate it.