Stop bandwidth theft!

Bandwidth can be expensive. So it can be annoying if other websites steal your bandwidth from you. A common problem is when people use 'hot-linking' or 'deep-linking' to place images from your site on to their own pages. Every time someone views their website, you will pick up the bandwidth tab.

So how can this be stopped?

When a web browser requests a page or an image from your site, the request includes a 'Referer' header (The misspelling is required in the specs!). This referrer gives the URL of the page that linked to the file. So, if you go to http://www.zeus.com/, your browser will load the HTML for the page, and then load all the images. Each time it asks the web server for an image, it will report that the referrer was http://www.zeus.com/

We can use this referrer header to check that the image is being loaded for your own site, and not for someone else's. If another website embedded a link to one of these images, the Referer: header would contain the URL of their site instead. This site has a more in-depth discussion of bandwidth-stealing.

The most common way to check this is to reconfigure your web server. If you are using the Zeus Web Server, then this is easy - just enable the 'Referrer Checking' option. For other web servers, fixing the problem is not easy. Here is an explanation of how to do so with Apache. Note that this involves re-arranging how the content is stored on your web server. This is not a quick fix! Luckily, the problem can also be solved with ZXTM, and with no changes to your existing servers.

Solving the problem with RuleBuilder™

We can use TrafficScript™ to do the hard work for us. Even better, we won't have to write a single line of code, since we can use ZXTM's RuleBuilder. Here's a walkthrough of creating a rule to stop hotlinking.

In the ZXTM UI, create a new rule (using RuleBuilder). We are interested in examining the HTTP header 'Referer', so click on 'HTTP Header' in the 'Conditions' tab on the right hand side. We want to check that this always contains zeus.com (It is best not to check the exact URL, e.g. people might visit http://www.zeus.com/ or http://zeus.com/). However, remember that we are writing a rule to block requests that are not from a Zeus site, so ensure that the rule checks that the header does not contain this text.

So, if a request from another site is received, what do we do with it? There are several possible actions. We could just drop the request. Or, we could redirect the request to another page (hopefully a smaller one), with some text telling the user that the request was banned. For this example, we'll just drop the connection.

However, as it stands, this isn't good enough. Some browsers don't send a Referer: header, and we don't want to block those people. So we need to add another condition (Make sure that 'All of the conditions must be met' is selected:

We're not quite there yet. We only want to put this hot-linking restriction in place for images. People should be allowed to follow links to our web pages. If we blocked all requests that weren't originating from our web site, no-one would be able to reach us in the first place. So let's just block JPEG images:

All done. Don't forget to add this rule to the virtual server(s) that run your web site, so that it gets used. Remember you must adjust the zeus.com to the name of your site!

TrafficScript improvements

There are several improvements that can be made. The rule could block multiple image types by matching against a regular expression rather than just checking for '.jpg'. We could also extend this rule using TrafficScript, so that it would automatically compare the Referer: header to the actual address of the site. This would mean that the rule would work for any URL, without any reconfiguration. Here's an example of this:

$referrer = string.lowerCase( http.getHeader( "Referer" ));
$host     = http.getHostHeader();
$path     = http.getPath();
$redirect = "/blocked.png";

# Never block requests with no Referer:
if( $referrer != "" ) {
   # Referer header should contain the hostname of the site
   if( !string.contains( $referrer, $host )) {
      # Only block images
      $path = http.getPath();
      if( string.regexMatch( $path, "\\.(jpe?g|png|gif)" )) {
         # OK, block this image (but allow our special blocked image)
         if( $path != $redirect ) {
            # Redirect them to our 'Go away' image
            http.redirect( $redirect);
         }
      }
   }
}

Ben [Zeus Dev Team] 09 July 2007