What is a Test Automation Framework?

A test automation framework is a set of tools used to perform automated software testing. Test automation frameworks provide significant advantages over manual testing and are commonly used in rapid software development. These frameworks allow software developers to create tests, execute tests and analyze test results. A test automation framework might be driven by code, data, a graphical user interface (GUI) or a combination of these.

Test automation frameworks are required to meet the needs of rapid software development. In the classic waterfall software development life cycle, software testing was a discrete phase performed after implementation, or coding. As such, human-executed tests were typical.

Many development teams have replaced the waterfall model with various agile software development methodologies, which typically require quick, iterative development. Rapid development has created a need for rapid testing. A test automation framework allows a software team to perform frequent, automated testing on a code base. For example, rather than requiring a separate testing phase after development, a team that uses a test automation framework might choose to run a full suite of tests overnight, as well as smaller chunks of automated tests throughout the day as changes are made to the source code.

The most commonly used type of framework is the code-driven framework, which is used to perform white-box testing. Code-driven frameworks require developers to write small snippets of code, in the form of unit tests, to test software functionality or performance. Developers using code-driven frameworks must take care to write their software in a way that is testable — that is, the code must be modularized such that a unit test can call a snippet of code and verify the result. These frameworks typically include an implementation of xUnit, where "x" varies based on the programming language being tested.

GUI-driven test frameworks are used to perform black-box testing, using a software tool as a user would, through mouse and keyboard inputs. They might be comprised of scripts written by developers to exercise a series of functionality, or they might be comprised of tool-generated scripts, such as record-and-play-back scripts. One potential drawback to these frameworks is that the script will break whenever the GUI changes. Some GUI-automation frameworks might generate a script that can be edited by a developer in case of GUI changes, and others might require the user to create a fresh record-and-play-back session.

Data-driven test frameworks perform software testing using data, and these tests typically work with data tables that specify both inputs and outputs. The data can be kept in various locations, such as databases, spreadsheets or text files. Scripts can be written to execute software methods taking the inputs as parameters and to validate the software method's return value against the output. Anything else that can be varied, such as the environment under which the test executes, also can be stored with the data and separated from the test script. An advantage provided by these frameworks is the separation of data and logic, which often leads to lower long-term maintenance.