Reading data from external files and tables

A frequent requirement in TrafficScript is to manage data stored in some sort of table. For example, a user may have a list of redirects they wish to apply, and storing those in a table is generally preferable to hard-coding them into a TrafficScript rule.

An illustration of the problem: the following TrafficScript code is cumbersome and hard to update with additional redirects. It also becomes inefficient as the number of redirects becomes very large.

$url = http.getPath();
 
if( $url == "/campaign1" )
   http.redirect( "/landing/cloud.html?googleparameters" );
if( $url == "/campaign2" )
   http.redirect( "/landing/virtualization.html?googleparameters" );
if( $url == "/zxtm" )
   http.redirect( "/products/zxtm/?googleparameters" );
if( $url == "/glb" )
   http.redirect( "/landing/zxtmglb?googleparameters" );
if( $url == "/afm" )
   http.redirect( "/landing/zxtmafm?googleparameters" );

ZXTM allows you to upload 'extra files' through the user interface (Catalog > Extra Files > Miscellaneous), via the SOAP Control API (Conf.Extra: addFile), or by manually copying them into ZEUSHOME/zxtm/conf/extra. These files can be read from a TrafficScript rule. Our rewriting policies would be much easier if we could quickly look up a mapping from a file like this:

/campaign1      /landing/cloud.html?googleparameters
/campaign2      /landing/virtualization.html?googleparameters
/zxtm           /products/zxtm/?googleparameters
/glb            /products/zxtmglb/?googleparameters
/afm            /products/zxtmafm/?googleparameters

redirects.txt

Each line contains a key ('/campaign1') and a value ('/landing/cloud.html?googleparameters'), separated by a TAB character.

How to do it

Paste the following code to the start of your TrafficScript rule:

sub update( $filename ) {
   $pid = sys.getPid();
 
   $md5 = resource.getMD5( $filename );
   if( $md5 == data.get( $pid.$filename.":md5" ) ) return;
 
   data.reset( $pid.$filename.":" );
   data.set( $pid.$filename.":md5", $md5 );
 
   $contents = resource.get( $filename );
   if( !string.endsWith( $contents, "\n" ) )
      log.warn( $filename." does not end with a newline" );
 
   $i = 0;
   while( ( $j = string.find( $contents, "\n", $i ) ) != -1 ) {
      $line = string.substring( $contents, $i, $j-1 );
      $i = $j+1;
 
      $j = string.find( $line, "\t" );
      $key = string.trim( string.left( $line, $j ) );
      $value = string.trim( string.skip( $line, $j+1 ) );
 
      data.set( $pid.$filename."::".$key, $value );
   }
}
 
sub lookup( $filename, $key ) {
   update( $filename );
 
   $pid = sys.getPid();
   return data.get( $pid.$filename."::".$key );
}

The point of this rather involved looking code is the new subroutine lookup( filename, key ) which efficiently parses the file and retrieves the value associated with the named key.

So, you can now write code like:

$url = http.getPath();
 
$redirect = lookup( "redirects.txt", $url );
if( $redirect ) http.redirect( $redirect );

Make sure you paste the code for the lookup and update functions at the start of each rule that uses them.

How does it work?

This rule uses ZXTM's global associative array. It stores data in a private 'prefix' so that it can be used alongside other uses of this shared global datastructure.

Each time a lookup() is made, the rule checks to see if the file has changed by comparing the current MD5 with a cached one. This is much more efficient than it might appear; ZXTM reads and calculates the MD5s of files in the conf/extra directory asynchronously, so this test results in a simple comparison.

If the file has changed, the update() subroutine reloads it. ZXTM runs several single-threaded processes (one per CPU core). To avoid race conditions, the rule keeps a separate copy of the data for each process (identified by the $pid) so that an ongoing update in one process will not corrupt a lookup in another process.

All data is stored with the prefix "<pid><filename>:". Because each process is single-threaded, there's no danger of a lookup occuring while another rule performs an update.

What else can I do with this?

ZXTM's Control API makes it easy to upload a file to a ZXTM cluster:

#!/usr/bin/perl -w
 
use SOAP::Lite 0.60;
 
my $filename = $ARGV[0] or die "No filename specified!";
 
my $contents = do {
        open IN, "<$filename" or die "Cannot read $filename: $!";
        local $/;
        <IN>;
};
# strip any leading path: i.e., from /path/to/file.txt to file.txt
$filename =~ s/^.*\///;
 
# This is the url of the ZXTM admin server
my $admin_server = 'https://user:password@host:9090';
 
my $conn = SOAP::Lite
    -> ns('http://soap.zeus.com/zxtm/1.0/Conf/Extra/')
    -> proxy("$admin_server/soap")
    -> on_fault( sub {
        my( $conn, $res ) = @_;
        die ref $res?$res->faultstring:$conn->transport->status; } );
 
$conn->writeFile( [ $filename ], [ $contents ] );

uploadFile.pl

So, if you have any auto-generated table of data that you need to act upon... a list of IP addresses, usernames, redirects, HTTP status codes or whatever... this gives you a potential means to upload and act on them in ZXTM.

Our marketing team access our main ZXTM directly to manage rewrite rules for the campaigns they run. They are very happy writing TrafficScript and get support from local product experts.

However, if we were to start again, this would be a much better way to do this. Manage a simple table of redirects in Notepad, a CLI script (or batch file) to upload the file to the ZXTM and you've quickly built a very easy solution for someone who needs to manage this data.

Owen Garrett [Zeus Dev Team] 02 July 2009 5 comments