Refactor run_suite_class

[jrotkiewicz]

def run_suite_class(argv=None):
  """Executes tests in the test suite.

  Args:
    argv: A list that is then parsed as CLI args. If None, defaults to sys.argv.
  """
  cli_args = _parse_cli_args(argv)
  suite_class = _find_suite_class()
  if cli_args.list_tests:
    _print_test_names([suite_class])
    sys.exit(0)
  test_configs = config_parser.load_test_config_file(
      cli_args.config, cli_args.test_bed
  )
  config_count = len(test_configs)
  if config_count != 1:
    logging.error('Expect exactly one test config, found %d', config_count)
  config = test_configs[0]
  runner = test_runner.TestRunner(
      log_dir=config.log_path, testbed_name=config.testbed_name
  )
  suite = suite_class(runner, config)
  console_level = logging.DEBUG if cli_args.verbose else logging.INFO
  ok = False
  with runner.mobly_logger(console_level=console_level):
    try:
      suite.setup_suite(config.copy())
      try:
        runner.run()
        ok = runner.results.is_all_pass
        print(ok)
      except signals.TestAbortAll:
        pass
    finally:
      suite.teardown_suite()
  if not ok:
    sys.exit(1)

There are two issues with this function:

1. it doesn't support --list_tests - it's currently broken

2. it doesn't work as proposed here:

   Create a subclass of base_suite.BaseSuite and add the individual test

   classes. Using the BaseSuite class allows users to define their own setup
   and teardown steps on the suite level as well as custom config for each test
   class.

   .. code-block:: python

   from mobly import base_suite
   from mobly import suite_runner

   from my.path import MyFooTest
   from my.path import MyBarTest

   class MySuite(base_suite.BaseSuite):

    def setup_suite(self, config):
      # Add a class with default config.
      self.add_test_class(MyFooTest)
      # Add a class with test selection.
      self.add_test_class(MyBarTest,
                          tests=['test_a', 'test_b'])
      # Add the same class again with a custom config and suffix.
      my_config = some_config_logic(config)
      self.add_test_class(MyBarTest,
                          config=my_config,
                          name_suffix='WithCustomConfig')
   

   if name == 'main':
   suite_runner.run_suite_class()

Which would be great if it worked as currently I see no other option to setup test suite with skipping some tests. I would like to use it such that I would be able to skip some predefined tests as in : self.add_test_class(MyBarTest, tests=['test_a', 'test_b'])

Let me know if there is other option for doing that, thank you !

[mhaoli]

Hi Jakub,

1. Yes, --list_tests is broken, I'll create a fix for it.
2. Test case selection is supported, could you share your code and console logs?

Here are my console logs and code for your reference, you can see that MyBarTest.test_c is skipped:

(test_mobly) $ python test_suite.py -c config.yaml
[SampleTestBed] 01-09 15:23:34.852 INFO ==========> MyFooTest <==========
[SampleTestBed] 01-09 15:23:34.853 INFO [Test] test_hello
[SampleTestBed] 01-09 15:23:34.853 INFO [Test] test_hello PASS
[SampleTestBed] 01-09 15:23:34.854 INFO Summary for test class MyFooTest: Error 0, Executed 1, Failed 0, Passed 1, Requested 1, Skipped 0
[SampleTestBed] 01-09 15:23:34.855 INFO ==========> MyBarTest <==========
[SampleTestBed] 01-09 15:23:34.855 INFO [Test] test_a
[SampleTestBed] 01-09 15:23:34.855 INFO [Test] test_a PASS
[SampleTestBed] 01-09 15:23:34.856 INFO [Test] test_b
[SampleTestBed] 01-09 15:23:34.856 INFO [Test] test_b PASS
[SampleTestBed] 01-09 15:23:34.858 INFO Summary for test class MyBarTest: Error 0, Executed 2, Failed 0, Passed 2, Requested 2, Skipped 0
[SampleTestBed] 01-09 15:23:34.859 INFO Summary for test run SampleTestBed@01-09-2025_15-23-34-851:
Total time elapsed 0.006948533002287149s
Artifacts are saved in "/tmp/logs/mobly/SampleTestBed/01-09-2025_15-23-34-851"
Test summary saved in "/tmp/logs/mobly/SampleTestBed/01-09-2025_15-23-34-851/test_summary.yaml"
Test results: Error 0, Executed 3, Failed 0, Passed 3, Requested 3, Skipped 0
True
(test_mobly) $ cat test_suite.py

from mobly import base_suite
from mobly import suite_runner

from my_foo_test import MyFooTest
from my_bar_test import MyBarTest


class MySuite(base_suite.BaseSuite):

  def setup_suite(self, config):
    self.add_test_class(MyFooTest)
    self.add_test_class(MyBarTest, tests=['test_a', 'test_b'])


if __name__ == '__main__':
        suite_runner.run_suite_class()
(test_mobly) $ cat my_bar_test.py

from mobly import base_test
from mobly import test_runner


class MyBarTest(base_test.BaseTestClass):

  def setup_class(self):
    pass

  def test_a(self):
    pass

  def test_b(self):
    pass

  def test_c(self):
    pass


if __name__ == '__main__':
  test_runner.main()

[jrotkiewicz]

Thank you! I will wait for the fix for --list_tests and recheck, I have it tied together to the tests