Package org.testng
Class TestNG
java.lang.Object
org.testng.TestNG
This class is the main entry point for running tests in the TestNG framework. Users can create
their own TestNG object and invoke it in many different ways:
- On an existing testng.xml
- On a synthetic testng.xml, created entirely from Java
- By directly setting the test classes
The command line parameters are:
- -d
outputdir
: specify the output directory - -testclass
class_name
: specifies one or several class names - -testjar
jar_name
: specifies the jar containing the tests - -sourcedir
src1;src2
: ; separated list of source directories (used only when javadoc annotations are used) - -target
- -groups
- -testrunfactory
- -listener
Please consult documentation for more details.
FIXME: should support more than simple paths for suite xmls
- See Also:
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final String
The default name for a suite launched from the command linestatic final String
The default name for a test launched from the command linestatic final String
The default name of the result's output directory (keep public, used by Eclipse).private static final String
static final Integer
private ExitCode
private final ExitCodeListener
private boolean
private static final Logger
This class' log4testng Logger.private final Map<Class<? extends IAlterSuiteListener>,
IAlterSuiteListener> private boolean
private IAnnotationTransformer
private final Map<Class<? extends IClassListener>,
IClassListener> private Class<?>[]
private XmlSuite.FailurePolicy
private IConfigurable
private IConfiguration
private final Map<Class<? extends IDataProviderInterceptor>,
IDataProviderInterceptor> private final Map<Class<? extends IDataProviderListener>,
IDataProviderListener> private Integer
private final IAnnotationTransformer
private String
private String
protected long
private String[]
private final Map<Class<? extends IExecutionVisualiser>,
IExecutionVisualiser> private IExecutorFactory
private boolean
private Boolean
private IHookable
private String[]
private static TestNG
private final Map<Class<? extends IInvokedMethodListener>,
IInvokedMethodListener> private boolean
private Boolean
private Boolean
private String
private static com.beust.jcommander.JCommander
private final List<IMethodInterceptor>
private ITestObjectFactory
private String
private XmlSuite.ParallelMode
private Boolean
private boolean
private final Set<XmlMethodSelector>
private URLClassLoader
private Boolean
protected long
private final Map<Class<? extends ISuiteListener>,
ISuiteListener> private Integer
private final Map<Class<? extends ITestListener>,
ITestListener> The list of test names to run from the given suiteprivate ITestRunnerFactory
private int
protected boolean
private Integer
If m_verbose gets set, it will override the verbose setting in testng.xmlprivate String
The path of the testng.xml file inside the jar fileprivate final Map<Class<? extends ITestNGListener>,
ITestNGListener> -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionvoid
addClassLoader
(ClassLoader loader) Add a class loader to the searchable loaders.void
addListener
(Object listener) Deprecated.Use addListener(ITestNGListener) insteadvoid
addListener
(ITestNGListener listener) private void
void
addMethodSelector
(String className, int priority) void
addMethodSelector
(XmlMethodSelector selector) private void
addReporter
(Class<? extends IReporter> r) private void
addReporter
(ReporterConfig reporterConfig) private void
private void
Using reflection to remain Java 5 compliant.void
alwaysRunListeners
(boolean alwaysRun) void
Deprecated.use new configure(CommandLineArgs) methodprotected void
configure
(CommandLineArgs cla) Configure the TestNG instance based on the command line parameters.createCommandLineSuitesForClasses
(Class[] classes) createCommandLineSuitesForMethods
(List<String> commandLineMethods) private IExecutorFactory
createExecutorFactoryInstanceUsing
(String clazzName) private SuiteRunner
createSuiteRunner
(XmlSuite xmlSuite) Creates a suite runner and configures its initial stateprivate void
createSuiteRunners
(SuiteRunnerMap suiteRunnerMap, XmlSuite xmlSuite) Creates theSuiteRunner
s and populates the suite runner map with this informationprivate static void
(package private) static void
exitWithError
(String msg) private void
generateReports
(List<ISuite> suiteRunners) Returns the configuration failure policy.protected IConfiguration
static TestNG
Deprecated.since 5.1protected long
getEnd()
private OverrideProcessor
protected long
getStart()
int
private int
getVerbose
(XmlSuite xmlSuite) boolean
boolean
boolean
hasSkip()
private void
init
(boolean useDefaultListeners) private void
private void
private static void
initializeCommandLineSuitesGroups
(XmlSuite s, boolean hasIncludedGroups, String[] m_includedGroups, boolean hasExcludedGroups, String[] m_excludedGroups) private void
private void
private void
void
Invoked by the remote runner.void
static void
The TestNG entry point for command line execution.private static <E> void
maybeAddListener
(Map<Class<? extends E>, E> map, E value) private static <E> void
maybeAddListener
(Map<Class<? extends E>, E> map, Class<? extends E> type, E value, boolean quiet) private static int
private void
parseSuite
(String suitePath) private void
private void
populateSuiteGraph
(IDynamicGraph<ISuite> suiteGraph, SuiteRunnerMap suiteRunnerMap, XmlSuite xmlSuite) Populates the dynamic graph with the reverse hierarchy of suites.static TestNG
privateMain
(String[] argv, ITestListener listener) Note: this method is not part of the public API and is meant for internal usage only.void
run()
Run TestNG.private void
runExecutionListeners
(boolean start) private void
Run the test suites.This needs to be public for maven2, for now..At least until an alternative mechanism is found.private void
runSuitesSequentially
(XmlSuite xmlSuite, SuiteRunnerMap suiteRunnerMap, int verbose, String defaultSuiteName) Recursively runs suites.private void
Before suites are executed, do a sanity check to ensure all required conditions are met.private void
void
setCommandLineSuite
(XmlSuite suite) void
setConfigFailurePolicy
(XmlSuite.FailurePolicy failurePolicy) Sets the policy for whether or not to ever invoke a configuration method again after it has failed once.private void
void
setDataProviderThreadCount
(int count) void
setDefaultSuiteName
(String defaultSuiteName) void
setDefaultTestName
(String defaultTestName) void
setExcludedGroups
(String groups) Define which groups will be excluded from this run.void
setExecutorFactory
(IExecutorFactory factory) void
setExecutorFactoryClass
(String clazzName) void
setGroupByInstances
(boolean b) void
Define which groups will be included from this run.private void
void
setInjectorFactory
(IInjectorFactory factory) void
Specify if this run should be made in JUnit modevoid
setListenerClasses
(List<Class<? extends ITestNGListener>> classes) Define which listeners to user for this run.void
setListenersToSkipFromBeingWiredInViaServiceLoaders
(String... listeners) void
setMethodInterceptor
(IMethodInterceptor methodInterceptor) void
void
void
setObjectFactory
(ITestObjectFactory factory) void
setOutputDirectory
(String outputdir) Sets the output directory where the reports will be created.void
setParallel
(String parallel) Deprecated.Use #setParallel(XmlSuite.ParallelMode) insteadvoid
setParallel
(XmlSuite.ParallelMode parallel) void
setPreserveOrder
(boolean b) void
setRandomizeSuites
(boolean randomizeSuites) void
void
void
setSourcePath
(String path) Deprecated.void
setSuiteThreadPoolSize
(Integer suiteThreadPoolSize) void
setTestClasses
(Class[] classes) Set the test classes to be run by this TestNG object.void
setTestJar
(String jarPath) Sets a jar containing a testng.xml file.void
setTestNames
(List<String> testNames) protected void
private void
setTestRunnerFactoryClass
(Class testRunnerFactoryClass) void
setTestSuites
(List<String> suites) Set the suites file names to be run by this TestNG object.void
setThreadCount
(int threadCount) void
setUseDefaultListeners
(boolean useDefaultListeners) void
setVerbose
(int verbose) Sets the level of verbosity.void
setXmlPathInJar
(String xmlPathInJar) void
setXmlSuites
(List<XmlSuite> suites) Specifies the XmlSuite objects to run.private String[]
Given a string com.example.Foo.f1, return an array where [0] is the class and [1] is the method.void
toggleFailureIfAllTestsWereSkipped
(boolean failIfAllTestsSkipped) private static void
usage()
protected static void
Double check that the command line parameters are valid.
-
Field Details
-
LOGGER
This class' log4testng Logger. -
DEFAULT_COMMAND_LINE_SUITE_NAME
The default name for a suite launched from the command line- See Also:
-
DEFAULT_COMMAND_LINE_TEST_NAME
The default name for a test launched from the command line- See Also:
-
DEFAULT_THREADPOOL_FACTORY
- See Also:
-
DEFAULT_OUTPUTDIR
The default name of the result's output directory (keep public, used by Eclipse).- See Also:
-
m_instance
-
m_jCommander
private static com.beust.jcommander.JCommander m_jCommander -
m_commandLineMethods
-
m_suites
-
m_cmdlineSuites
-
m_outputDir
-
m_includedGroups
-
m_excludedGroups
-
m_isJUnit
-
m_isMixed
-
m_useDefaultListeners
protected boolean m_useDefaultListeners -
m_failIfAllTestsSkipped
private boolean m_failIfAllTestsSkipped -
m_listenersToSkipFromBeingWiredIn
-
m_testRunnerFactory
-
m_classListeners
-
m_testListeners
-
m_suiteListeners
-
m_reporters
-
m_dataProviderListeners
private final Map<Class<? extends IDataProviderListener>,IDataProviderListener> m_dataProviderListeners -
m_dataProviderInterceptors
private final Map<Class<? extends IDataProviderInterceptor>,IDataProviderInterceptor> m_dataProviderInterceptors -
m_executorFactory
-
DEFAULT_VERBOSE
-
m_threadCount
private int m_threadCount -
m_parallelMode
-
m_configFailurePolicy
-
m_commandLineTestClasses
-
m_defaultSuiteName
-
m_defaultTestName
-
m_methodDescriptors
-
m_selectors
-
m_objectFactory
-
m_invokedMethodListeners
private final Map<Class<? extends IInvokedMethodListener>,IInvokedMethodListener> m_invokedMethodListeners -
m_dataProviderThreadCount
-
m_jarPath
-
m_xmlPathInJar
The path of the testng.xml file inside the jar file -
m_stringSuites
-
m_hookable
-
m_configurable
-
m_end
protected long m_end -
m_start
protected long m_start -
m_alterSuiteListeners
-
m_isInitialized
private boolean m_isInitialized -
isSuiteInitialized
private boolean isSuiteInitialized -
exitCodeListener
-
exitCode
-
m_executionVisualisers
private final Map<Class<? extends IExecutionVisualiser>,IExecutionVisualiser> m_executionVisualisers -
m_verbose
If m_verbose gets set, it will override the verbose setting in testng.xml -
m_defaultAnnoProcessor
-
m_annotationTransformer
-
m_skipFailedInvocationCounts
-
m_methodInterceptors
-
m_testNames
The list of test names to run from the given suite -
m_suiteThreadPoolSize
-
m_randomizeSuites
private boolean m_randomizeSuites -
m_alwaysRun
private boolean m_alwaysRun -
m_preserveOrder
-
m_groupByInstances
-
m_configuration
-
m_serviceLoaderClassLoader
-
serviceLoaderListeners
-
-
Constructor Details
-
TestNG
public TestNG()Default constructor. Setting also usage of default listeners/reporters. -
TestNG
public TestNG(boolean useDefaultListeners) Used by maven2 to have 0 output of any kind come out of testng.- Parameters:
useDefaultListeners
- Whether or not any default reports should be added to tests.
-
-
Method Details
-
init
private void init(boolean useDefaultListeners) -
toggleFailureIfAllTestsWereSkipped
public void toggleFailureIfAllTestsWereSkipped(boolean failIfAllTestsSkipped) - Parameters:
failIfAllTestsSkipped
- - Whether TestNG should enable/disable failing when all the tests were skipped and nothing was run (Mostly when a test is powered by a data provider and when the data provider itself fails causing all tests to skip).
-
setListenersToSkipFromBeingWiredInViaServiceLoaders
- Parameters:
listeners
- - An array of fully qualified class names that should be skipped from being wired in via service loaders.
-
getStatus
public int getStatus() -
setOutputDirectory
Sets the output directory where the reports will be created.- Parameters:
outputdir
- The directory.
-
setUseDefaultListeners
public void setUseDefaultListeners(boolean useDefaultListeners) - Parameters:
useDefaultListeners
- If true before run(), the default listeners will not be used.- org.testng.reporters.TestHTMLReporter
- org.testng.reporters.JUnitXMLReporter
- org.testng.reporters.XMLReporter
- See Also:
-
setTestJar
Sets a jar containing a testng.xml file.- Parameters:
jarPath
- - Path of the jar
-
setXmlPathInJar
- Parameters:
xmlPathInJar
- Sets the path to the XML file in the test jar file.
-
parseSuiteFiles
private void parseSuiteFiles() -
getProcessor
-
parseSuite
-
initializeSuitesAndJarFile
public void initializeSuitesAndJarFile() -
setThreadCount
public void setThreadCount(int threadCount) - Parameters:
threadCount
- Define the number of threads in the thread pool.
-
setParallel
Deprecated.Use #setParallel(XmlSuite.ParallelMode) instead- Parameters:
parallel
- Define whether this run will be run in parallel mode.
-
setParallel
-
setCommandLineSuite
-
setTestClasses
Set the test classes to be run by this TestNG object. This method will create a dummy suite that will wrap these classes called "Command Line Test".If used together with threadCount, parallel, groups, excludedGroups than this one must be set first.
- Parameters:
classes
- An array of classes that contain TestNG annotations.
-
splitMethod
Given a string com.example.Foo.f1, return an array where [0] is the class and [1] is the method. -
createCommandLineSuitesForMethods
- Parameters:
commandLineMethods
- a string with the form "com.example.Foo.f1,com.example.Bar.f2"- Returns:
- a list of XmlSuite objects that represent the list of classes and methods passed in parameter.
-
createCommandLineSuitesForClasses
-
addMethodSelector
-
addMethodSelector
-
setTestSuites
Set the suites file names to be run by this TestNG object. This method tries to load and parse the specified TestNG suite xml files. If a file is missing, it is ignored.- Parameters:
suites
- A list of paths to one more XML files defining the tests. For example:TestNG tng = new TestNG(); List<String> suites = Lists.newArrayList(); suites.add("c:/tests/testng1.xml"); suites.add("c:/tests/testng2.xml"); tng.setTestSuites(suites); tng.run();
-
setXmlSuites
Specifies the XmlSuite objects to run. -
setExcludedGroups
Define which groups will be excluded from this run.- Parameters:
groups
- A list of group names separated by a comma.
-
setGroups
Define which groups will be included from this run.- Parameters:
groups
- A list of group names separated by a comma.
-
setTestRunnerFactoryClass
-
setTestRunnerFactory
-
setObjectFactory
-
setObjectFactory
-
setListenerClasses
Define which listeners to user for this run.- Parameters:
classes
- A list of classes, which must be either ISuiteListener, ITestListener or IReporter
-
addListener
Deprecated.Use addListener(ITestNGListener) instead- Parameters:
listener
- The listener to add
-
maybeAddListener
-
maybeAddListener
-
addListener
-
getReporters
-
getTestListeners
-
getSuiteListeners
-
setVerbose
public void setVerbose(int verbose) Sets the level of verbosity. This value will override the value specified in the test suites.- Parameters:
verbose
- the verbosity level (0 to 10 where 10 is most detailed) Actually, this is a lie: you can specify -1 and this will put TestNG in debug mode (no longer slicing off stack traces and all).
-
setExecutorFactoryClass
-
createExecutorFactoryInstanceUsing
-
setExecutorFactory
-
getExecutorFactory
-
initializeCommandLineSuites
private void initializeCommandLineSuites() -
initializeCommandLineSuitesParams
private void initializeCommandLineSuitesParams() -
initializeCommandLineSuitesGroups
private void initializeCommandLineSuitesGroups() -
initializeCommandLineSuitesGroups
-
addReporter
-
initializeDefaultListeners
private void initializeDefaultListeners() -
initializeConfiguration
private void initializeConfiguration() -
addListeners
-
addServiceLoaderListeners
private void addServiceLoaderListeners()Using reflection to remain Java 5 compliant. -
sanityCheck
private void sanityCheck()Before suites are executed, do a sanity check to ensure all required conditions are met. If not, throw an exception to stop test execution- Throws:
TestNGException
- if the sanity check fails
-
initializeEverything
public void initializeEverything()Invoked by the remote runner. -
run
public void run()Run TestNG. -
runSuites
Run the test suites.This method can be overridden by subclass.
For example, DistributedTestNG to run in master/slave mode according to commandline args.- Returns:
- - List of suites that were run as
ISuite
objects. - Since:
- 6.9.11 when moving distributed/remote classes out into separate project
-
runSuiteAlterationListeners
private void runSuiteAlterationListeners() -
runExecutionListeners
private void runExecutionListeners(boolean start) -
usage
private static void usage() -
generateReports
-
runSuitesLocally
This needs to be public for maven2, for now..At least until an alternative mechanism is found.- Returns:
- The locally run suites
-
error
-
getVerbose
- Returns:
- the verbose level, checking in order: the verbose level on the suite, the verbose level on the TestNG object, or 1.
-
runSuitesSequentially
private void runSuitesSequentially(XmlSuite xmlSuite, SuiteRunnerMap suiteRunnerMap, int verbose, String defaultSuiteName) Recursively runs suites. Runs the children suites before running the parent suite. This is done so that the results for parent suite can reflect the combined results of the children suites.- Parameters:
xmlSuite
- XML Suite to be executedsuiteRunnerMap
- MapsXmlSuite
s to respectiveISuite
verbose
- verbose leveldefaultSuiteName
- default suite name
-
populateSuiteGraph
private void populateSuiteGraph(IDynamicGraph<ISuite> suiteGraph, SuiteRunnerMap suiteRunnerMap, XmlSuite xmlSuite) Populates the dynamic graph with the reverse hierarchy of suites. Edges are added pointing from child suite runners to parent suite runners, hence making parent suite runners dependent on all the child suite runners- Parameters:
suiteGraph
- dynamic graph representing the reverse hierarchy of SuiteRunnerssuiteRunnerMap
- Map with XMLSuite as key and its respective SuiteRunner as valuexmlSuite
- XML Suite
-
createSuiteRunners
Creates theSuiteRunner
s and populates the suite runner map with this information- Parameters:
suiteRunnerMap
- Map with XMLSuite as key and it's respective SuiteRunner as value. This is updated as part of this method callxmlSuite
- Xml Suite (and its children) for whichSuiteRunner
s are created
-
createSuiteRunner
Creates a suite runner and configures its initial state -
getConfiguration
-
main
The TestNG entry point for command line execution.- Parameters:
argv
- the TestNG command line parameters.
-
privateMain
Note: this method is not part of the public API and is meant for internal usage only.- Parameters:
argv
- The param argumentslistener
- The listener- Returns:
- The TestNG instance
-
configure
Configure the TestNG instance based on the command line parameters.- Parameters:
cla
- The command line parameters
-
setSuiteThreadPoolSize
-
getSuiteThreadPoolSize
-
setRandomizeSuites
public void setRandomizeSuites(boolean randomizeSuites) -
alwaysRunListeners
public void alwaysRunListeners(boolean alwaysRun) -
setSourcePath
Deprecated.This method is invoked by Maven's Surefire, only remove it once Surefire has been modified to no longer call it.- Parameters:
path
- The path
-
parseInt
-
configure
Deprecated.use new configure(CommandLineArgs) methodThis method is invoked by Maven's Surefire to configure the runner, do not remove unless you know for sure that Surefire has been updated to use the new configure(CommandLineArgs) method.- Parameters:
cmdLineArgs
- The command line
-
setTestNames
- Parameters:
testNames
- Only run the specified tests from the suite.
-
setSkipFailedInvocationCounts
-
addReporter
-
setJUnit
Specify if this run should be made in JUnit mode- Parameters:
isJUnit
- - Specify if this run should be made in JUnit mode
-
setMixed
- Parameters:
isMixed
- Specify if this run should be made in mixed mode
-
validateCommandLineParameters
Double check that the command line parameters are valid.- Parameters:
args
- The command line to check
-
hasFailure
public boolean hasFailure()- Returns:
- true if at least one test failed.
-
hasFailureWithinSuccessPercentage
public boolean hasFailureWithinSuccessPercentage()- Returns:
- true if at least one test failed within success percentage.
-
hasSkip
public boolean hasSkip()- Returns:
- true if at least one test was skipped.
-
exitWithError
-
getOutputDirectory
-
getAnnotationTransformer
-
setAnnotationTransformer
-
getDefaultSuiteName
- Returns:
- the defaultSuiteName
-
setDefaultSuiteName
- Parameters:
defaultSuiteName
- the defaultSuiteName to set
-
getDefaultTestName
- Returns:
- the defaultTestName
-
setDefaultTestName
- Parameters:
defaultTestName
- the defaultTestName to set
-
setConfigFailurePolicy
Sets the policy for whether or not to ever invoke a configuration method again after it has failed once. Possible values are defined inXmlSuite
. The default value isXmlSuite.FailurePolicy.SKIP
- Parameters:
failurePolicy
- the configuration failure policy
-
getConfigFailurePolicy
Returns the configuration failure policy.- Returns:
- config failure policy
-
getDefault
Deprecated.since 5.1- Returns:
- The default instance
-
setConfigurable
-
setHookable
-
setMethodInterceptor
-
setDataProviderThreadCount
public void setDataProviderThreadCount(int count) -
addClassLoader
Add a class loader to the searchable loaders.- Parameters:
loader
- The class loader to add
-
setPreserveOrder
public void setPreserveOrder(boolean b) -
getStart
protected long getStart() -
getEnd
protected long getEnd() -
setGroupByInstances
public void setGroupByInstances(boolean b) -
setServiceLoaderClassLoader
-
addServiceLoaderListener
-
getServiceLoaderListeners
-
setInjectorFactory
-