Detecting and Managing Abusive Referers

Popular news and blogging sites such as Slashdot and Digg have huge readerships. They are community driven and allow their members to post articles on various topics ranging from hazelnut chocolate bars to global warming. These sites, due to their massive readership, have the power to generate huge spikes in the web traffic to those (un)fortunate enough to get mentioned in their articles. Fortunately ZXTM and TrafficScript™ can help.

If the referenced site happens to be yours, you are faced with dealing with this sudden and unpredictable spike in bandwidth and request rate, causing:

• a large proportion or all of your available bandwidth to be consumed by visitors referred to you by this popular site; and in extreme cases,

• a cascade failure across your web servers as each one becomes overloaded, fails and, in doing so, adds further load onto the remaining web servers.

Bandwidth Management and Rate Shaping

ZXTM has the ability to shape traffic in two important ways. Firstly, you can restrict the amount of bandwidth any client or group of clients are allowed to consume. This is commonly known as "Bandwidth Management" and in ZXTM it's configured by using a bandwidth class. Bandwidth classes are used to specify the maximum bits per second to make available. The alternative method is to limit the number of requests that those clients or group of clients can make per second and/or per minute. This is commonly known as "Rate Shaping" and is configured within a ZXTM rate class.

Both Rate Shaping and Bandwidth Management classes are configured and stored within the catalog section of your ZXTM. Once you have created a class it is ready for use and can be applied to one or more of your Virtual Servers. However the true power of these Traffic Shaping features really becomes apparent when you make use of them with TrafficScript.

What is an Abusive Referer

I would class an Abusive Referer as any site on the internet that refers enough traffic to your server to overwhelm it and effectively deny service to other users. This abuse is usually unintentional, the problem lies in the sheer number of people wanting to visit your site at that one time. This "Slashdot Effect" can be detected and dealt with by a TrafficScript rule and either a Bandwidth or a Rate Class.

Detecting and Managing Abusive Referers

Example One

Take a look at the TrafficScript below for an example of how you could stop a site (in this instance slashdot) from from using a large proportion or all of your available bandwidth.

$referrer = http.getHeader( "Referer" );
if( string.contains( $referrer, "slashdot" ) ) {
   http.addResponseHeader( "Set-Cookie", "slashdot=1" );
   response.setBandwidthClass( "slashdot" );
}

if( http.getCookie( "slashdot" ) ) {
   response.setBandwidthClass( "slashdot" );
}

In this example we are specifically targeting slashdot users and preventing them from using more bandwidth than we have allotted them in our "slashdot" bandwidth class. This rule requires you to know the name of the site you want protection from, but this rule or similar could be modified to defend against other high traffic sites.

Example Two

The next example is a little more complicated, but will automatically limit all requests from any referer. I've chosen to use two rate classes here, BusyReferer for those sites I allow to send a large amount of traffic and StandardReferers for those I don't. At the top I specify a $whitelist, which contains sites I never want to rate shape, and $highTraffic which is a list of sites I'm going to shape with my BusyReferer class. By default, all traffic not in the white list is sent through one of my rate classes, but only on entry to the site. That's because subsequent requests will have myself as the referer and will be whitelisted. In times of high load, when a referer is sending more traffic than the rate class allows, a back log will build up, at that point we will also start issuing cookies to put the offending referers into a bandwidth class.

# Referer whitelist. These referers are never rate limited.
$whitelist = "localhost 172.16.121.100";

# Referers that are allowed to pass a higher number of clients.
$highTraffic = "google mypartner.com";

# How many queued requests are allowed before we track users.
$shapeQueue = 2;

# Retrieve the referer and strip out the domain name part.
$referer = http.getheader("Referer");
$referer = String.regexsub($referer, ".*?://(.*?)/.*", "$1", "i" );

# Check to see if this user has already been given an abuse cookie.
# If they have we'll force them into a bandwidth class
if ( $cookie = http.getCookie("AbusiveReferer") )
{
        response.setBandwidthClass("AbusiveReferer");
}

# If the referer is whitelisted then exit.
if ( String.contains( $whitelist, $referer ) )
{
    break;
}

# Put the incoming users through the busy or standard rate classes
# and check the queue length for their referer.
if ( String.contains( $highTraffic, $referer ) )
{
     $backlog = rate.getbacklog("BusyReferer", $referer);
     rate.use("BusyReferer", $referer);
} else {
     $backlog = rate.getbacklog("StandardReferer", $referer);
     rate.use("StandardReferer", $referer);
}

# If we have exceeded our backlog limit, then give them a cookie
# this will enforce bandwidth shaping for subsequent requests.
if ( $backlog > $shapeQueue )
{
     http.setResponseCookie("AbusiveReferer", $referer);
     response.setBandwidthClass("AbusiveReferer");
}

In order for the TrafficScript to function optimally, you must enter your servers own domainname(s) into the white list. If you do not, then the script will perform rate shaping on everyone surfing your website!

You also need to set appropriate values for the BusyReferer and StandardReferer shaping classes. Remember we're only counting the clients entry to the site, so Perhaps you want to set 10/minute as a maximum standard rate and then 20/minute for your BusyReferer rate.

In this script we also use a bandwidth class for when things get busy. You will need to create this class, called "AbusiveReferer" and assign it an appropriate amount of bandwidth. Users are only put into this class when their referer is exceeding the rate of referrals set by the relevant rate class.

Shaping with Context

Rate Shaping classes can be given a context so you can apply the class to a subset of users, based on a piece of key data. The second script uses context to create an instance of the Rate Shaping class for each referer. If you do not use context, then all referers will share the same instance of the rate class.

Conclusion

ZXTM can use bandwidth and rate shaping classes to control the number of requests that can be made by any group of clients. In this article, we have covered choosing the class based on the referer, which has allowed us to restrict the rate at which any one site can refer visitors to us. These examples could be modified to base the restrictions on other data, such as cookies, or even extended to work with other protocols. A good example would be FTP, where you could extract the username from the FTP logon data and apply a bandwidth class based on the username.

Mark Boddington [Zeus Systems Engineering] 12 July 2007