fat Architecture

What is fat

'fat' is the Framework for Automated Testing and was created to fill the need for automated reliable testing of software and firmware products on a regular basis.

The goal of this product is to have a fully built and defined set of tools that can be 'dropped' into a software project environment and immediately enhance the reliablity of said product by giving a defined and simplistic interface for that project's engineers to use in creating a full bodied test suite.

• fat framework internals

How it works

The framework consists of two framework files and as many module files as desired.

• fat.py - Houses the framework itself
• testconfig.py - Allows for the framework to target a specific product

• The Input Interface
• The Output Interface
• The Test Wrapper
• User created Test Modules
• Target Interface

Users view:

Data flow

• The user sends input in via the Input Interface Component, that information is recieved by the Test Wrapper object and forwarded to the user created Test Module.
• The Test Module gains access to the Target hardware/software by way of a predefined Target Interface that is defined in the testconfig module.
• The Test Module is given access to the output mechanism by the Test Wrapper which in turn sends information back to the user

Main()

The framework when executed directly (considered normal usage) has its main routine called.
The Main routine will allow for several arguments:

-h|-?	help, prints the usage/help menu
-i <script|gui|cli>	user's input interface to use (script is a non-interactive implementation and is default) *** default is script***
-o <file='filename'|stdout|echo='filename'>	output log type will define which type of log to write to 'file' will write information to the file filename, 'screen' will send it to the stdout, and 'echo' will write to 'filename' and echo to the screen *** default is screen ***
-t <target>	the name of the target as it will go to the TargetInterface object, i.e. -t flex4, or -t hl *** -t is a required argument ***
-m <module>	the name of a specific module to run, overrides the frameworks normal directive to run all tests as returned by get_tests() function in testconfig.py
-D	debug, allows printing/logging of debug information for the test
-- args	will tell the TestWrapper to take the quoted arguments following the '--' and give them to both the TestModule being run as well as the TestInterface in testconfig.py

A typical usage of fat.py being called directly would be:
     python fat.py -t flex3 -- --flex b --file /tmp/all-current.fm
...or if we wanted to install the all-current.fm file and write to a log file only, we would change our output and specify a module with the -m argument:
     python fat.py -o file=install-log.txt -t flex3 -m flexinstall -- --flex b --file /tmp/all-current.fm

Arguments following the -- argument are removed and put into a different list. This list is than provided to the TestWrapper which in turn supplies it to the Input Interface, and the Target Interface components for use if desired.

This list is useful for specifying certain arguments or flags unique to the particular test implementation without cluttering the framework with argument tests that only apply to a specific product.

Object Initialization

The Input Interface

The input interface is given to the TestWrapper as the 'input' argument of its __init__() method. It will normally be provided to the framework by the -i argument given to fat.py when executed directly.

All input interfaces are derived from the base class UserInterface and have the methods __init__(), run() , and fini().

• __init__() - this is the default constructor and is called, by the Main routine during initialization.
• run() - method is where test modules are imported and their subsequent run methods called. it is called by the TestWrapper, with the following arguments:
  
  1. log - the output mechanism object
  2. ifc - an interface object
  3. args - a list of arguments from cli to give to each test
  4. mods - the modules available to import and run
• fini() - method allows for any cleanup (fileIO, etc...) to be done when the tests are completed. It will be called by the TestWrapper upon program completion

From the UserInterface base class we have pre-defined several sub-classes that will be most commonly used:

• UserScript (-i script) - a class that works in a non-interactive manner to run a set of predefined tasks, i.e., as a cron job for nightly testing.
  * UserScript is the default input interface used when no -i <input> option is given to the TestWrapper
• UserCLI (-i cli) - a user interactive class that will be used for those tests that absolutely require human intervention, i.e., visually inspecting LED patterns, etc...
• UserGUI (-i gui) - a graphically rich user interface to make testing more intuitive and allow for alternate means of bug identification.

Class diagram:

Input Interface Interaction

The input interface object is given to the TestWrapper as an argument, but once the TestWrapper has the other frame work components in place, the input interface will be given control of the program by the TestWrapper via the 'run()' method. At that time all data is handed to the input interface. At the completion of its run() method the input interface returns (without status code) control to the TestWrapper which calls everyone's fini() method and cleans up.

The Input Interface makes use of some of the output objects hidden methods to allow for more data collection to occur:

• start() - to begin data collection for a module
• stop() - to stop data collection for a module
• imported() - to record the import status of a module
• dump() - to get the list of all modules ran and their outcomes

Input Interface program flow

1. We save our arguments:
   
   • logging mechanism as self.log
   • the target interface object as self.ifc
   • the command line arguments (from fats main, arguments after '--') as self.args
   • a list of modules to run as self.testmodules
2. For each module in the self.testmodules list we do the following:
   
   1. log a module starting with module name
   2. attempt to import module, report our success or failure, and log the import status
   3. create an instance of module.TestModule class giving its __init__() method our logger, our interface, and our arguments
   4. call the TestModule objects run() method
   5. we don't watch for the TestModules return status and leave it up to the module to define its internal tests using the logging mechanisms teststart() method and passed(), failed(), and fatal() to determine the beginning and outcome of a particular test.
      We do, however, protect against exceptions raised, in particular if the module raises a FatalError type we will take that as a flag to cease all testing and not continue with any further modules.
3. Once all modules are complete, we call our output object (self.log)'s dump() method to get the entire tests final results.
4. We exit the run() method returning control to the TestWrapper.

Input Interface Interaction Diagram:

The Output Interface

The output interface is a collection of objects controlled through the logging mechanism's Logger object.
The Logger receives all calls from the input interface and the user created test modules and writes them to the actual output device, i.e., stdout or to file. The Logger also sends information to the Statistics collection object where appropriate.

The componenets of the output/logging mechanism are obscured from the user and are not meant to be directly accessed. They are:

• LogTarget - an object used by the logger to write data
• Logger - access to all output operations is done through this object
• StatCollect - a statistics gathering object
• Timer - a utillity object used by StatCollect to gather time related statistics

The LogTarget is a base class used in creating a defined interface for objects that are to act as recipients of the Logger objects calls. It is required to have the following methods:

• open_target() - should create a writable context and return its descriptor, i.e., open a file, return FD
• get_fd() - will return the FD created by open_target()
• fini() - will be called on the programs completion allowing for any file closing/cleanup to occur

• write()
• flush()
• close()

The log target is created by the user and given as an argument to the TestWrapper. In the case of normal usage, the -o argument given to fat will specify one of three predefined output classes:

• FileLog - flat file I/O
• ScreenLog - writes to stdout
• EchoLog - writes to file and stdout

The Logger is the wrapper object and is given its log target as an argument by the TestWrapper when it is initialized. It then creates an instance of the StatCollect object to gather information about the types of calls it has recieved. When any of its methods are invoked the Logger will write to the log target and then forward information to the StatCollect object when appropriate. The logger sends all of its writing tasks to a single internal method add_entry() which does all the writing to the Log Target (using write()). The Logger has the following hidden methods available that are only used by the Input Interface:

• start
• stop
• logimport
• dump

• teststart
• info
• warn
• error
• debug
• passed
• fail
• fatal

The StatCollect class stores a list of dictionarys relating to a given module and tracks import outcome, errors, number of tests, warnings, passes, fatalities, failures, and time (as tracked by its Timer object). The data is stored in the following format:
     {"Test-Module":,"Import":, "RunTime":,"Errors":,"Passes":,"Failures":, "FatalErrors":,"Warnings":,"TotalTests":,"Final":}
The entire list of dictionarys is retrieved by the Logger via the dump() method and written to the log target.

The Timer class is used as a utillity by the StatCollect class to gather elapsed time for the module to run. It captures time when its start() and stop() methods are called and then returns the difference in a various formats as required.

Output Class diagrams

The Target Interface and testconfig.py

The target interface is contained within the testconfig.py module. Its implementation is target specific making it impossible to fully describe. The following items are required to be in testconfig.py:

• get_test(args) - will return a list of modules to run
• and the three classes:
  
  • class Error(Exception):
  • class FatalError(Exception):
  • class TestInterface:

The TestInterface class will need to provide a method for the individual TestModule objects to access the target (firmware/hardware/software).
It will be instantiated by the TestWrapper and the object given to the TestModules as self.ifc (interface) so the writers of those modules will need to know how to use that object

The Test Wrapper

The TestWrapper class brings all the components together and handles final cleanup.
The TestWrapper object is created with the following arguments:

• input - a UserInterface object
• output - a LogTarget object
• target - textstring target for testing
• args - a list of arguments to pass to the modules and target interface

When the TestWrapper's run() method is called it in turn calls the run() method of the UserInterface and giving it the Logger object, the list of args, a list of modules to run, and the TargetInterface object.

Once the UserInterface returns control to the TestWrapper we call our own fini() method. Our fini() method in turn calls the fini() for each of the following in order:

1. TargetInterface object
2. Logger object
3. Output object - the actual Log target
4. Input object - the UserInterface

TestWrapper relationships

The User Created Test Modules

1. this module is imported by the UserInterface (input) object
2. our TestModule class is instantiated and the following arguments are given to our constructor:
   
   • log - the output mechanism object, available as self.log
   • ifc - an interface object, available as self.ifc
   • args - a list of arguments from cli available as self.args
3. the TestModule's run() method is called by the UserInterface
4. For each test done in our run method we start it by calling our logging mechanism's teststart() method.
5. As desired we call the logging mechanism's error, warn, debug, and info methods to log information about the test.
6. When test is completed we call one of our logging mechanism's completion calls - passed, fail, or fatal
7. Once all internal testing is complete, we return control to the UserInterface that called us.
8. If I've put the system into a condition that precludes any other test from running once I'm done, a FatalError (as imported from testconfig.py) is thrown to flag the framework to stop testing.

How the Module interacts from its view:

End of Document