Being Lazy with Java ExtensionsWith a Java Extension, you can log traffic events in real time to an external database. The example in this article describes how to log the ‘referring’ source that each visitor comes in from when they enter a website, so that you can determine which sites are sending you the most traffic. Logging is done to a MySQL database. The article then presents a modification that illustrates how to lazily perform operations such as database writes in the background (i.e. asynchronously) so that the performance the end user observes is not impaired. OverviewIt’s often very revealing to find out which web sites are referring the most traffic to the sites that you are hosting. Tools like Google Analytics and web log analysis applications are one way of doing this, but in this example we’ll show an alternative method where we log the frequency of referring sites in a local database for easy access. When a web browser submits an HTTP request for a resource, it commonly includes a header called "Referer" (description) which identifies the page that linked to that resource. We’re not interested in internal referrers – where one page in the site links to another. We’re only interested in external referrers. We're going to log these 'external referrers' to a MySQL database, counting the frequency of each so that we can easily determine which occur most commonly.
Create the databaseCreate a suitable MySQL database, with limited write access for a remote user: % mysql –h dbhost –u root –p Enter password: ******** mysql> CREATE TABLE website.referers ( referer VARCHAR(256) PRIMARY KEY, count INTEGER ); mysql> GRANT SELECT,INSERT ON website.referers TO ‘web’@’%’ IDENTIFIED BY ‘W38_U5er’; mysql> GRANT SELECT,INSERT ON website.referers TO ‘web’@’localhost’ IDENTIFIED BY ‘W38_U5er’; mysql> QUIT; Verify that the table was correctly created and the ‘web’ user can access it: % mysql –h dbhost –u web –p Enter password: W38_U5er mysql> DESCRIBE website.referers; +---------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------+--------------+------+-----+---------+-------+ | referer | varchar(256) | NO | PRI | NULL | | | count | int(11) | YES | | NULL | | +---------+--------------+------+-----+---------+-------+ 2 rows in set (0.00 sec) mysql> SELECT * FROM website.referers; Empty set (0.00 sec) The database looks good... Create the Java ExtensionCompile up the following 'RefererTrack' Java Extension. You can refer to the instructions in the KnowledgeHub Article "Watermarking Images with Java Extensions" for instructions on compiling extensions:
import java.io.IOException;
import java.io.PrintWriter;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public class RefererTrack extends HttpServlet {
private static final long serialVersionUID = 1L;
private Connection conn = null;
private String userName = null;
private String password = null;
private String database = null;
private String dbserver = null;
public void init( ServletConfig config) throws ServletException {
super.init( config );
userName = config.getInitParameter( "username" );
password = config.getInitParameter( "password" );
database = config.getInitParameter( "database" );
dbserver = config.getInitParameter( "dbserver" );
if( userName == null || password == null || database == null || dbserver == null )
throw new ServletException( "Missing username, password, database or dbserver config value" );
try {
Class.forName("com.mysql.jdbc.Driver").newInstance();
} catch( Exception e ) {
throw new ServletException( "Could not initialize mysql: "+e.toString() );
}
}
public void doGet( HttpServletRequest req, HttpServletResponse res )
throws ServletException, IOException
{
try {
if( conn == null ) {
conn = DriverManager.getConnection(
"jdbc:mysql://"+dbserver+"/"+database, userName, password);
}
String referer = req.getHeader( "Referer" );
if( referer != null ) {
PreparedStatement s = conn.prepareStatement(
"INSERT INTO referers( referer, count ) VALUES( ?, 1 )" +
"ON DUPLICATE KEY UPDATE count=count+1" );
s.setString(1, referer);
s.executeUpdate();
}
} catch( Exception e ) {
conn = null;
log( "Could not log referer to database: " + e.toString() );
}
}
public void doPost( HttpServletRequest req, HttpServletResponse res )
throws ServletException, IOException
{
doGet( req, res );
}
}
Upload the resulting RefererTrack.class file to ZXTM. You’ll need to supply the following configuration so that it can communicate with the database server:
This article describes how to configure Initialization Parameters in more detail. Add the TrafficScript ruleWe only want to log referrers from remote sites, so use the following TrafficScript rule to call the Extension only when it is required:
# This site
$host = http.getHeader( "Host" );
# The referring site
$referer = http.getHeader( "Referer" );
# Only log the Referer if it is an absolute URI and it comes from a different site
if( string.contains( $referer, "://" ) && !string.contains( $referer, "://".$host."/" ) ) {
# log.info( "Logging referer header ".$referer );
java.run( "RefererTrack" );
}
Add this rule as a request rule to a virtual server that processes HTTP traffic. As users access the site, the referer header will be pushed into the database. A quick database query will tell you what's there: % mysql –h dbhost –u web –p Enter password: W38_U5er mysql> SELECT * FROM website.referers ORDER BY count DESC LIMIT 4; +--------------------------------------------------+-------+ | referer | count | +--------------------------------------------------+-------+ | http://www.google.co.uk/search?q=zxtm | 92 | | http://www.zeus.com/products/zxtm | 45 | | http://www.vmware.com/appliances | 26 | | http://www.zeus.com/ | 5 | +--------------------------------------------------+-------+ 4 rows in set (0.00 sec) Lazy writes to the databaseThis is a useful application of Java Extensions, but it has one big drawback. Every time a visitor arrives from a remote site, his first transaction is stalled while the Java Extension writes to the database. This breaks one of the key rules of website performance architecture – do everything you can asynchronously (i.e. in the background) so that your users are not impeded (see "Lazy Websites run Faster"). Instead, a better solution would be to maintain a separate, background thread that wrote the data in bulk to the database, while the foreground threads in the Java Extension simply appended the Referer data to a table:
The following Java Extension (
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.util.LinkedList;
import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public class RefererTrackAsync extends HttpServlet {
private static final long serialVersionUID = 1L;
protected boolean running = false;
protected static LinkedList<String> referers = new LinkedList<String>();
protected class Writer extends Thread {
private Connection conn = null;
public void init( String username, String password, String url )
throws Exception {
Class.forName("com.mysql.jdbc.Driver").newInstance();
conn = DriverManager.getConnection( url, username, password);
running = true;
start();
}
public void run() {
while( running ) {
try { sleep( 20*1000 ); } catch( InterruptedException e ) {};
try {
PreparedStatement s = conn.prepareStatement(
"INSERT INTO referers( referer, count ) VALUES( ?, 1 )" +
"ON DUPLICATE KEY UPDATE count=count+1" );
conn.setAutoCommit( false );
synchronized( referers ) {
while( !referers.isEmpty() ) {
String referer = referers.removeFirst();
s.setString(1, referer);
s.addBatch();
}
}
s.executeBatch();
} catch ( SQLException e ) {
log( e.toString() );
running = false;
}
}
}
}
public void init( ServletConfig config ) throws ServletException {
super.init( config );
String userName = config.getInitParameter( "username" );
String password = config.getInitParameter( "password" );
String database = config.getInitParameter( "database" );
String dbserver = config.getInitParameter( "dbserver" );
if( userName == null || password == null || database == null || dbserver == null )
throw new ServletException( "Missing username, password, database or dbserver config value" );
try {
Writer writer = new Writer();
writer.init( userName, password, "jdbc:mysql://"+dbserver+"/"+database );
} catch( Exception e ) {
running = false;
throw new ServletException( e.toString() );
}
}
public void doGet( HttpServletRequest req, HttpServletResponse res )
throws ServletException, IOException
{
String referer = req.getHeader( "Referer" );
if( running && referer != null ) {
synchronized( referers ) {
referers.add( referer );
}
}
}
public void doPost( HttpServletRequest req, HttpServletResponse res )
throws ServletException, IOException
{
doGet( req, res );
}
public void destroy() {
running = false;
super.destroy();
}
}
When the Extension is invoked by ZXTM, it simply stores the value of the Referer header in a local list and returns immediately. This minimizes any latency that the end user may observe. The Extension creates a separate thread (embodied by the Compile the extension up and upload the resulting two class files ( You’ll observe that database updates may be delayed by up to 20 seconds (you can tune that delay in the code), but the level of service that end users experience will no longer be affected by the speed of the database. Other Uses of Java ExtensionsZXTM's Java Extension capability lets you expand the power of ZXTM in many ways. "Authenticating users with Active Directory" gives another example of database integration; "Watermarking Images with Java Extensions" shows how to manipulate response data. Search KnowledgeHub for more examples.
Owen Garrett
[Zeus Dev Team] 05 June 2008
|
Recent Articles
Other Resources
|
