Framework for Automated Testing (fat)

Usage of the fat.py testing framework

Implementing your test as part of the framework

1. Create a file with your test name
   vi flexinstall.py
2. Import in BaseTestModule from fat.py
   from fat import BaseTestModule
3. Create a class inside your file called TestModule
   class TestModule(BaseTestModule):
4. Overload the get_name() method and return a short description of the test
         def get_name(self):
              return "Flex21 firmware installer"
5. Overload the run() method, this is where you do your tests
         def run(self):
   

In the TestModule the following methods are available to you when executing other programs:

• catch_return(cmd)
  ret = self.catch_return(command)
  This will execute 'command' and return the exit code
• catch_output(cmd)
  out = self.catch_output(command)
  This will execute 'command' and return its output
• catch_return_output(cmd)
  (ret,out) = self.catch_return_output(command)
  This will execute 'command' and return a tuple of (exit status, output)

When the TestModule is instantiated by the framework it is given three arguments which are available to you

1. A logging mechanism object is given to you as self.log with the following methods are available to the log:
   

   self.log.debug(msg)	will print DEBUG: messages if a -D argument was handed to the framework
   self.log.teststart(msg)	will begin gathering statistics on the test 'msg' and print a STARTED message
   self.log.info(msg)	will print an INFO message
   self.log.warn(msg)	will print a WARNING message
   self.log.error(msg)	will print an ERROR message and log an error (the definition of error in this case is external to the test itself, i.e., port was busy, or ip not available)
   self.log.passed(msg)	if test ran within expected parameters, this will log a PASSED message and be captured as a statistic
   self.log.fail(msg)	if test failed to perform within expected parameters this will print a FAILED message and be captured as a statistic

   
   



2. A Interface object will be available to you as self.ifc. In the case of Flex, this is a FlexAccess object as can be found in the flexcontact.py module.
   
   This uses the FlexAccess object which acts as a cache of the various access objects so that multiple test scripts can be run within our test framework one after the other and each one won't have to re-connect to serial CLI, etc. On the other hand, we create each of the access objects on demand so that we won't spend time connecting to things someone doesn't need.
   
   For telnet sessions, reglib sessions, etc, it is possible to have multiple simultaneous sessions, so an instance # is used to retrieve independent objects.
   
   
   • self.ifc.get(request, inst=0)
     Accepts the following requests:
     
     • "leftconsole" - returns left serial cons.
     • "rightconsole" - returns right serial cons.
     • "lefttelnet" - returns telnet to left mgr CLI
     • "righttelnet" - returns telnet to right mgr CLI
     • "telnet" - returns telnet to floating IP
     • "leftshell" - returns telnet to left mgr port 5000
     • "rightshell" - returns telnet to right mgr port 5000
     
     ** Note that the subclass of pexpect object also provides utility methods "setAttrValue" and "getAttrValue" so that they can be used in place of a reglib object for simple interactions.
     
     The following requests return a reglib object
     
     • "leftreglib" - returns reglib for left mgr
     • "rightreglib" - returns reglib for left mgr
     • "reglib" - returns reglib to floating IP
     
     The object returned is a pexpect object that has all pexpect.Spawn options (see pexpect documentation for further details).
     In order to use this object you will need to import pexpect:
           import pexpect
     
     Create desired object (in this case a connection to flex3's right flex manager)
           self.spid = self.ifc.get("rightconsole")
     
     The following are the most commonly used methods of the object:
     
     • self.spid.send("sh\r")
       This sends the sh command to the terminal
       ***It is important not to forget the '\r'!***
       
     
     
     • i = spid.expect([pexpect.TIMEOUT,"#"])
       if i == 0:
             raise Error("Connecting to fmgr shell timed out")
       elif i == 1:
             # got our shell prompt, continue on
             pass
       else:
             from testconfig import Error
             raise Error("Got unknown argument while waiting for shell prompt")
       
       This will watch for the '#' shell prompt and throw a fat Error if we time out or something unexpected happens.
       
       
   • self.ifc.getTarget()
     This will return the target your interface is using. Its return will be a text string i.e., flex3 This will match the argument given by the frameworks -t argument.
   • self.ifc.getFrl(target)
     This will return a string with information about the target flex manager in the form of ip:linuxbox:tty. i.e., getFrl(flex1a) will return 172.17.77.11:clarinet:5
   • self.ifc.frl_to_dict
     This takes a frl string (such as returned by getFrl()) and returns a dictionary of {'ip':xx,'linuxbox':xx,'port':xx}



3. A list of arguments as supplied to the framework by the caller which can be retrieved as self.args. It will have the form of ["arg","opt","arg",opt"] and is easily searchable by your class

Exception handling

All errors should be tried for within your program, any that are not caught willbe captured by the framework and the overall test results considered a failure.

The exception to this rule is the framework provided exception class FatalError(). If the module throws a FatalError to the framework, the framework will take this as a flag that the module has failed in such a manner that no further testing will be possible. In this case the framework will simply exit and report statistics without continuing.

The FatalError class can be imported from fat.py:
      from testconfig import FatalError
It can than be used as any exception class, i.e.,:
      raise FatalError("Something boffed")
      except FataError,err:
      print str(err)
The Error class is also available for your use as:
      from testconfig import Error
fat.py uses only these errors internally. If an other exception type occurs, the framework will print a stack trace and than continue testing.

Integrating your test into the Framework

To have your test part of those automatically run, add the filename (minus the .py) to the testconfig.py public function get_tests().
For example:
To add the flexinstall.py module to the automatically run tests, change the return of get_tests() so...
return ["test1",]
becomes
return ["test1","flexinstall"]

Using the test Framework

The test framework is housed in the file fat.py. Its usage:

Some examples:

I have a module named flexinstall.py that I want to run manually. The flexinstall module requires some additional parameters to run, it needs a --flex argument telling it which of the module pair to run and a --file of which file to load and install.

example:

           python fat.py -m flexinstall -- --flex b --file /home/jclark/all.fm

* This will run the flex install module in a non-interactive mode (defaulted -i script) and will send its output to the screen (defaulted -o screen). It will give the argument list ["--flex","b","--file","/home/jclark/all.fm"] to my program as self.args.

Application framework class diagram (from the users perspective)

End of Document