Synchronizing SMTP

If you wish to use ZXTM to inspect or modify the headers or body of SMTP messages, you need to write some TrafficScript™ to synchronize the two halves of the conversation, to ensure that the connection doesn't stall. This article explains how to do this, and presents an example TrafficScript rule that can be used as the basis of more advanced SMTP rules.

Background - salient details of SMTP

SMTP is a server first TCP protocol, usually run on port 25. Lines are terminated by carriage return and line feed (CRLF). A typical SMTP conversation proceeds as follows:

Client                                  Server
1. Opens connection to server
                                        2. Server sends banner
3. sends EHLO ...
                                        4. Server acknowledges/rejects
 
# Header processing
5. sends MAIL FROM: someone@example.com
                                        6. Server acknowledges/rejects
7. sends RCPT TO: other@example.com
                                        8. Server acknowledges/rejects
9. sends DATA
                                        10. Server acknowledges
10. Client sends body terminated by '.' on new line
                                        11. Server acknowledges
 
(Client optionally sends another email, by returning to step 5.,
 otherwise sends "QUIT" and server closes connection)

It's important to note the following:

• The protocol is server first, so a proxy must send the server's banner before waiting for a request from the client.
• Each request or response is terminated by CRLF. However, once the server has acknowledged the client's "DATA" request, the client continues to send CRLF terminated lines, without waiting for any response from the server, until it sends the final '.'
• Real world clients may not wait for a response from the server before sending the next header line. It's quite usual to see a client send all the headers in the packet it sends at step 5. The proxy must handle this situation as well as the conventional alternating request/response sequence of events.

Synchronizing SMTP with TrafficScript

To handle SMTP, we need a TrafficScript rule that creates a state-machine with two states:

State 1. Reading Headers: Read the request a line at a time, sending each line on to the server. If the request is "DATA", then transition to state 2.

State 2. Reading Body: Read the request a line at a time, but don't send it to the server straight away. Instead buffer the request until the entire body has been received. Once the whole body has been received, send the body and transition to state 1, ready to receive another email or another command, such as QUIT.

Since our TrafficScript rule is going to be executed every time a new line of data is received from the client, not just once per email sent, we need to keep track of the state between invocations of the rule. TrafficScript provides connection.data.set() and connection.data.get() to set flags that will let us keep track of which state we are in. As their names suggest, these commands allow you to set and read variables that persist as long as the TCP connection they are associated with.

Multiple lines might arrive in the same packet from the client, even though we only wish to pass them to the server one at a time. We therefore need to indicate to ZXTM how to split the data into individual lines. We can do this using request.endsWith(), which takes a single argument, a regular expression that defines the end of the current request. Any data that comes after this will be kept for the next time the request rule is run.

Putting this together, the following is a request rule that will synchronise SMTP. There are three places in it from which you could hook in code to inspect/modify the headers and body on the way through.

# SMTP Synchronizer - ZXTM Request Rule
 
if (connection.data.get("BODY_COMPLETE")) {
    # We've finished one email, so reset the state flags, in case
    # another email is on its way.
    connection.data.set ("DATA_RECEIVED", 0);
    connection.data.set ("BODY_COMPLETE", 0);
}
 
if (! connection.data.get("DATA_RECEIVED")) {
    # State 1: Reading Headers
    # We haven't seen "DATA" yet, so process headers, line by line
 
    # Read the next line of the request into $request
    $request = request.endsWith("\r\n");
    if (string.regexmatch($request, "^DATA\r\n")) {
        # DATA seen - set the DATA_RECEIVED flag to switch into
        # the Reading Body state
        connection.data.set("DATA_RECEIVED", 1);
    }
 
    #--------------------------------------------------
    # INSPECTION/MODIFICATION HOOK
    # USE $request TO MODIFY/INSPECT HEADERS HERE
    #--------------------------------------------------
 
} else {
    # State 2: Reading Body
    # Keep reading the body until we see the end of it
    $line = "";
    $next = 0;
 
    # The regular expression in the next line matches the
    # "end of body" indicator (a full stop on a line on its own)
    while ( ! string.regexmatch($line, "^\\.\r\n")) {
        $line = request.getline("\r\n", $next);
        $next = $1;
 
        #----------------------------------------------------
        # INSPECTION/MODIFICATION HOOK
        # USE $line TO INSPECT/MODIFY BODY LINE BY LINE HERE
        #----------------------------------------------------
 
        $request = $request . $line;
    }
 
    #--------------------------------------------------
    # INSPECTION/MODIFICATION HOOK
    # USE $request to INSPECT/MODIFY WHOLE BODY HERE
    #--------------------------------------------------
 
    # We've read the whole body, so set the BODY_COMPLETE flag
    connection.data.set("BODY_COMPLETE", 1);
}
 
# Send the request on to the back-end server.
# At this point $request will contain either a single header line, or
# the entire body, depending on which state we are in.
 
request.set($request);

You should note the difference in behavior of request.endsWith() and request.getline(). Whilst they both return some data from the request, request.endsWith() tells ZXTM that this is the end of this request, and no more data should be read from the client until after the request has been sent to the server and a response received. request.getline(), on the other hand, merely reads some data from the client without ending the request, and so can be called multiple times within the same invocation of a request rule. If no data is available when request.getline() is called, processing will pause until the client sends some. We can thus use it to read and buffer the body. If we tried to use request.endsWith() instead, then the connection would stall. ZXTM would read the first line of the body, send it to the server, then pause waiting for a response from the server, which would never come.

The condensed version

If you don't want to perform line by line inspection or modification of the body, you can condense the above rule into the more elegant one below. This uses request.endswith() to read both the headers and the body, with only a single connection.data variable that both tracks the state and supplies the appropriate regular expression to request.endswith().

# SMTP Synchronizer Condensed- ZXTM Request Rule
 
# State is tracked in the connection.data variable "STATE", which is
# set to one of the following two constants.  The constants themselves
# are the regular expressions supplied as arguments to
# request.endsWith() to define the end of a request in each state.
 
$STATE_HEADERS = "\r\n";
$STATE_BODY    = "\r\n\\.\r\n";
 
# Initialise STATE if this is the first request on a new connection
if (! connection.data.get("STATE")) {
    connection.data.set("STATE", $STATE_HEADERS);
}
 
$request = request.endsWith(connection.data.get("STATE"));
 
#### STATE SWITCHING
# We read the whole body in one go, so if we've got to here, and the
# state is STATE_BODY, then switch back to reading headers.
if (connection.data.get("STATE") == $STATE_BODY) {
    connection.data.set("STATE", $STATE_HEADERS);
} else if (string.regexmatch($request, "^DATA\r\n")) {
    # Whereas if we were reading headers, and see DATA, we should
    # prepare ourselves for the body
    connection.data.set("STATE", $STATE_BODY);
}
#### END STATE SWITCHING
 
#----------------------------------------------------------------------
# INSPECTION/MODIFICATION HOOK
# USE $request to INSPECT/MODIFY WHOLE BODY OR INDIVIDUAL HEADERS HERE
#----------------------------------------------------------------------
 
request.set($request);

Julian [Zeus Dev Team] 09 August 2007