iZeus part 2: The "Home" screen

iZeus on iPhone It has been quite some time since my first iZXTM^WiZeus article. I was beginning to wonder if I'd ever get around to part 2. However, the massive Zeus 6.0 release has now been launched. This means that those of us in the Zeus Dev Team have a little more breathing room in our schedules – although we're already hard at work designing and implementing features for the next release! That said, I've managed to squeeze in some time to work on this article in which I present an extension to the previous proof-of-concept: we'll parse the data out of the Zeus Traffic Manager MainIndex page and repackage it for the iPhone web browser. (The Zeus Dev Team has reached 55% iPhone penetration, Android is at 9%. That said, a few of us are certainly tempted by some of the upcoming Android handsets.)

In my previous article on the subject I opened with the following hypothetical scenario:

  1. We had an existing complex UI.
  2. A requirement had arisen to get an iPhone UI out the door in a week.
  3. Zeus software comes to the rescue, perfectly positioned to use the old UI as an API to build a new UI.

Within 4 hours of never having had considered creating UIs for the iPhone Safari browser I'd implemented a working iPhone-sleek login screen for my traffic manager. That's not going to cut the mustard however, after you've logged in you're greeted with the usual front page.

Zeus TM 6.0 in the iPhone web browser

Not sleek! No matter which way you look at it.

Zeus TM 6.0 in the iPhone web browser - sideways

If you're using a normal desktop or laptop LCD note that the iPhone screen is actually about 2/3 the size of these images. Sure, you can zoom in on the page and scoot around, but the user experience is far from ideal.

In this follow-up article I'm going to replace the traffic manager's home page with something more appropriate for the iPhone. To do this I'm going to bring in a technology I've written about previously: Jython! This is because the parsing and HTML manipulation is going to get to a point where I'd like to employ some library code not available in TrafficScript. I could use Java, but I prefer to avoid Java really. Also, I know and have used BeautifulSoup for parsing dodgy HTML in the past – when in a hurry stick with what you know.

So to begin, a quick run-down on setting up the Jython and iUI environments:

  1. Follow the instructions in my Jython article – but only these parts:

    1. Installing Jython to your chosen $JYTHONHOME
    2. Uploading ZeusPyServlet.jar to the Java Extensions Catalog on your Traffic Manager
    3. Setting a python_home attribute for ZeusPyServlet assigned to a value of your chosen $JYTHONHOME

    At the time of writing this article I'm using Jython 2.5.1.

  2. We need to make some additions to the Python libraries available to Jython:

    • BeautifulSoup is a very forgiving HTML parser, designed for dealing with the usual sort of terrible HTML you find on the web. Download BeautifulSoup.py and put it in your $JYTHONPATH/Lib directory.
    • ElementTree is a simple XML tree representation with various useful traversal and search methods, including a subset of XPath. Jython does actually ship with a version of ElementTree in xml.etree, but it is an older version with less useful XPath support so I have downloaded and installed version 1.3. Put it in your $JYTHONPATH/Lib/elementtree directory:

      cd $JYTHONHOME/Lib
      mkdir -p elementtree
      cd elementtree
      for py in ElementInclude.py ElementPath.py ElementTree.py \
            HTMLTreeBuilder.py SimpleXMLWriter.py TidyTools.py __init__.py
         do wget http://svn.effbot.org/public/elementtree-1.3/elementtree/$py
      done
    • ElementSoup glues the above items together, using BeautifulSoup to parse the document and build an ElementTree representation of it. Download ElementSoup.py and put it in your $JYTHONPATH/Lib directory. Unfortunately the available version of ElementSoup isn't up to date with the version of BeautifulSoup I'm using, nor the updated install of ElementTree (it will pick the older version by default.) A small patch must be applied to the ElementSoup.py file:

      --- ElementSoup.py.orig	2009-10-30 09:04:44.219938960 +0000
      +++ ElementSoup.py	2009-10-28 13:25:06.699940760 +0000
      @@ -7,14 +7,9 @@
       # soup classes that are left out of the tree
       ignorable_soup = BS.Comment, BS.Declaration, BS.ProcessingInstruction
      
      -# slightly silly
      -try:
      -    import xml.etree.cElementTree as ET
      -except ImportError:
      -    try:
      -        import cElementTree as ET
      -    except ImportError:
      -        import elementtree.ElementTree as ET
      +# import the updated (1.3) ElementTree
      +import elementtree.ElementTree as ET
      +# parsing is done by BeautifulSoup, so we don't need cElementTree
       
       import htmlentitydefs, re
       
      @@ -73,7 +68,7 @@
               except UnicodeError:
                   encoding = "iso-8859-1"
           soup = BS.BeautifulSoup(
      -        text, convertEntities="html", fromEncoding=encoding
      +        text, fromEncoding=encoding
               )
           # build the tree
           if not bob:

      This patch is trivial enough to apply by hand, but you can also download the diff as iZeus_ElementSoup.py.diff.txt and if placed in the same directory as the original ElementSoup.py file the patch can be applied using the patch command:

      patch -p0 < iZeus_ElementSoup.py.diff.txt
  3. You should also install the iUI files in Catalogs > Extra Files > Miscellaneous Files. I've used iui-0.31.tar.gz, from the iUI download page. Unpack the tarball and move all the files from the iui directory into Miscellaneous Files. This can be done most easily on the traffic manager host machine with something along the lines of

    tar -xv --strip-components 1 -C $ZEUSHOME/zxtm/conf/extra -f iui-0.31.tar.gz iui

    You will then need to set up a Virtual Server and Request Rule similar to that outlined in my previous iPhone article. I've created an HTTPS Virtual Server listening on port 9099 and tied it to a Pool pointed at the local Zeus admin server on port 9090. The VS and Pool are named iZeus. I have also created an associated TrafficScript request rule called iZeus-request:

    # iZeus file server
    $path = http.getPath();
    $ciuipath = "/iui/";
    if (string.startswith($path, $ciuipath)) {
       $file = string.skip($path, string.length($ciuipath));
       if (resource.exists($file)) {
          $mime = "text/plain";
          if (string.endswith($file, ".png")) {
             $mime = "image/png";
          } else if (string.endswith($file, ".html")) {
             $mime = "text/html";
          }
          http.sendResponse( 200, $mime, resource.get($file), "" );
       } else {
          http.sendResponse( 404, "text/html", "<h1>File not found</h1>", "" );
       }
    }

    You can download this rule as iZeus-request and copy it directly to $ZEUSHOME/zxtm/conf/rules.

OK, we're all set up and ready for the real work!

I tackled this problem in two parts. First I wrote a Jython servlet called iZeus.py and put it in my Java Libraries & Data Catalog. The code in the initial servlet parsed and extracted data from the Zeus Traffic Manager MainIndex page and just dumped the data to the web browser as plain text with only basic formatting. The servlet is executed from a TrafficScript response rule named iZeus-response:

$path = http.getPath();
$qs = http.getQueryString();
if (http.getResponseCode() == 200 &&
      (($path == "/apps/zxtm/" && $qs == "") ||
         string.contains($qs, "MainIndex"))) {
   java.run( "ZeusPyServlet", "iZeus.py" );
}

As you can see, the iZeus.py servlet is executed only if the Zeus index page is requested. The rule also only runs the servlet for 200 (OK) responses, this means that login page redirects will get through without invoking the servlet. The following image is a snapshot of my Services > Config Summary table after configuring the virtual server (on port 9100 in this case) and pool correctly and adding the rules.

The servlet itself first uses ElementSoup to parse the document, then makes liberal use of simple XPath search syntax to dig through the resultant HTML data-structure. Here's a small sample from the code:

import ElementSoup as ES

...

class iZeus(HttpServlet):

   ...

   def doGet(self, req, res):
      brf = BufferedReaderFile(res.getReader())
      html = ES.parse(brf)
      # hack to generate mapping from all children to their parents
      for p in html.getiterator():
         for c in p:
            c.parent = p

      # extract just the bits of HTML we're interested in
      services = None
      tms = None
      for ele in html.findall('.//span[@class="home_page_title"]'):
         if ele.text == "Traffic Managers":
            tms = extract_tm_data(ele.parent.parent.parent)
         elif ele.text == "Services":
            services = extract_svc_data(ele.parent.parent.parent)
         elif services is not None and tms is not None:
            break

   ...

With this much done I moved on to creating a static mock iZeus UI using iUI. This was a single HTML file with examples of functional and broken traffic managers and services. Then I split this HTML into the component parts that the code will need to generate from the data extracted from the MainIndex page:

  • iZeus_main_tpl.html
  • iZeus_tm_line_tpl.html
  • iZeus_tm_page_tpl.html
  • iZeus_service_line_tpl.html
  • iZeus_service_page_tpl.html
  • iZeus_pool_tpl.html

Here's how each template corresponds to part of the iZeus UI:

iZeus templates

Here's an example of the HTML, this is the content of the iZeus_service_page_tpl.html file:

   <ul id="svc_iZeus" title="Service">
      <li class="group">Virtual Server</li>
      <li class="vs_full"><div>
         <a target="_self" href="http://link-to-vs-edit-page">
            <strong>iZeus</strong> <img src="http://ssl-icon-image" /><br />
            <span class="note Running">&ndash; Running &ndash;</span><br />
            <span class="details">HTTP on port 9099</span>
         </a><br />
         <img src="http://link-to-start-icon" /><img src="http://link-to-stop-icon" />
      </div></li>
      <li class="group">Pool(s)</li>
      __POOL_LIST_GOES_HERE__
   </ul>

Now I needed to turn these files into useful templates. In the previous article I used __FOO__ style tags and TrafficScript string substitution to implement a trivial templating system. But in this case we have nested and listed template content, and conditionally displayed data (images) as well! It is time for something more complete. We also have the power of Python's built in string templating available to us. In Python you can do the following: print "Hello %(name)s!" % my_dict – this will call the __getitem__ method on my_dict with the key "name". This means we can just put all our simple text substitutions into a dictionary and use the %(key)s syntax, which gives us the same level of functionality we had in the previous article.

We now also want to conditionally display images such as the SSL icon, and insert listed sub-items using additional templates. This is simple enough in Python that we can get away with implementing our own trivial templater:

class TemplateDict(object):

   def __init__(self, dict, templates):
      self.dict = dict
      self.templates = templates

   def __getitem__(self, item):
      value = ""
      if 0 < item.find('|'):
         value = self.do_template(item)
      elif 0 < item.find(':'):
         value = self.do_tag(item)
      else:
         if not self.dict.has_key(item):
            value = "GET:NOVALUE:%s" % item
         else:
            value = self.dict[item]
      #print "TPL: %s ==> %s" % (item, value)
      return value

   def do_template(self, item):
      parts = item.split("|", 1)
      key = parts.pop(0)
      template = parts.pop(0)
      value = ""
      if not self.dict.has_key(key):
         value = "TEMPLATE:NOVALUE:%s" % item
      elif not self.templates.has_key(template):
         value = "TEMPLATE:UNKNOWN:%s" % item
      else:
         template = self.templates[template]
         for dict in self.dict[key]:
            value += template % TemplateDict(dict, self.templates)
      return value

   def do_tag(self, item):
      parts = item.split(":", 1)
      tag = parts.pop(0)
      key = parts.pop(0)
      value = ""
      if self.dict.has_key(key):
         value = "<%s" % tag
         for attr, val in self.dict[key].iteritems():
            if re.match("onmouse", attr): continue
            value += ' %s="%s"' % (attr, val)
         value += "/>"
      return value

Re-writing the HTML above to match our template syntax gives:

   <ul id="svc_%(name)s" title="Service">
      <li class="group">Virtual Server</li>
      <li class="vs_full"><div>
         <a target="_self" href="%(link)s">
            <strong>%(name)s</strong> %(img:ssl_icon)s<br />
            <span class="note %(status)s">&ndash; %(status)s &ndash;</span><br />
            <span class="details">%(protocol)s on port %(port)s</span>
         </a><br />
         <img src="%(start)s" /><img src="%(stop)s" />
      </div></li>
      <li class="group">Pool(s)</li>
      %(pools|pool_line)s
   </ul>

I marked up all my templates using this syntax and uploaded them to the Miscellaneous Files catalog. I also created one extra image which needs to be uploaded to the catalog: iZeusFailListArrow.png

The next step is to load the template files from the iZeus.py servlet. Remember that once the servlet object is instantiated it is persistent in memory, we can optimize loading of the templates so that we do not need to read them into memory on every page view:

...
class iZeus(HttpServlet):

   templates = {}
   tpl_timestamps = {}
   tpl_files = {
      "main" : "iZeus_main_tpl.html",
      "tm_line" : "iZeus_tm_line_tpl.html",
      "tm_page" : "iZeus_tm_page_tpl.html",
      "service_line" : "iZeus_service_line_tpl.html",
      "service_page" : "iZeus_service_page_tpl.html",
      "pool_line" : "iZeus_pool_tpl.html",
   }
   tpl_path = "zxtm/conf/extra"

   def __init__(self):
      print "iZeus: init"
      HttpServlet.__init__(self)
      # load the template files
      self.loadTemplates()

   def loadTemplates(self):
      """
      Load template files into memory, if they've been updated.
      """
      zh = os.getenv('ZEUSHOME')
      for (name, filename) in self.tpl_files.iteritems():
         fullname = os.path.join(zh, self.tpl_path, filename)
         try:
            stat = os.stat(fullname)
         except os.error, e:
            print "iZeus: ERROR: Cannot stat template %s: %s" % (fullname, e)
            continue
         # load the template into memory if it is not up to date
         if not self.tpl_timestamps.has_key(fullname) or \
               stat.st_mtime > self.tpl_timestamps[fullname]:
            self.tpl_timestamps[fullname] = stat.st_mtime
            try:
               tpl = file(fullname)
               self.templates[name] = tpl.read()
               print "iZeus: Loaded template %s:%s" % (name, filename)
            except os.error, e:
               print "iZeus: ERROR: Cannot read template %s: %s" % (fullname, e)
               continue

   def doGet(self, req, res):
      # load the template files
      self.loadTemplates()
      ...

Bringing the templates and data-structure together is one final, trivial, step away:

   def doGet(self, req, res):

      ...

      # output the harvested data
      output = res.getOutputStream()
      data = { "services" : services, "tms" : tms, "tm_name" : tm_name }
      output.write(self.templates["main"] % TemplateDict(data, self.templates))
      output.flush()

Now we have iZeus!

iZeus side-by-side

The following list contains everything required to get this project running, aside from 3rd party files (Jython, iUI, and additional Python code.) You can download them all at once via either a zip or a tar.gz file (remember, you'll still have to download all the 3rd party files using the links above and in the Jython article.)

If you set this all up at once and visit the Virtual Server the page will take a long while to load after you first log in. This is because Jython must first compile all uncompiled Python code into cached bytecode files. Subsequent page loads will be faster.

My observations on this "iZeus" project so far are:

  • I'm worried about the deadline in our scenario right now, I had a week and getting the implementation this far has taken 2 long days of work. This was pretty much iZeus 2: iZeus Harder! The login page implementation in the previous article took 4 hours. This leaves me with only a couple of days, realistically speaking.
  • Was Jython the right choice? Hard to say. TrafficScript doesn't have the required functionality. Java certainly does, but I expect it would have taken me longer to implement in Java – mainly as my Java skills are far more rusty than my Python skills.
  • BeautifulSoup is slooooow! The iZeus page load time is 4 to 5 seconds. I put timers in the Jython code and discovered that the BeautifulSoup parse of the MainIndex page accounts for almost all of this!
  • You'd be crazy to do things this way ... unless there was no other choice. The Zeus software has a SOAP API, wrapping an iPhone UI around this would be a much better approach. Unfortunately most websites out there don't have APIs and barely have any frontend/backend separation, so there may not be a lot of choice for some.

What will I do next? Wait and see in iZeus 3: iZeus with a Vengeance!

Yvan Seth [Zeus Dev Team] 31 October 2009 Bookmark with del.icio.us Post this article to Digg Post this article to reddit Post this article to Facebook Tweet this article  
Leave a comment ...
Your email address will not be displayed.
Your URL will be displayed.
This public messageboard is not a forum for technical support. To report technical support problems, please contact our dedicated Support team using the instructions at the bottom of this page.
Options:
 
(Line breaks become <br />)
(Set cookies for name, email & url)

Recently...

Other Resources