Python 3.12 is not able to iterate every test case from test suite file

Hello Everyone,

Issue Statement:
Upgrading Python from 3.4 to 3.12.1 broke my Python test framework (PATH). When running a specific test suite via batch commands, it now fails to iterate through all test cases. However, running the same suite directly from my IDE (PyCharm) works as expected, successfully executing all tests.

Is there any known issue in Python 3.12.1 ?
Kindly help in fixing this issue. Also, let me know if any further details are required as I am unable to attach batch file code screenshots or PATH framework related details.

Thanks,
GV

set PYTHONIOENCODING=utf-8
IF "%UCCE_AUTO%"=="" SET UCCE_AUTO=c:\ucce_auto\
@echo UCCE_AUTO=%UCCE_AUTO%
set FUNCTIONAL_HOME=%UCCE_AUTO%TestAutomation\FunctionalFramework
call %FUNCTIONAL_HOME%\path\commands\PathSetVirtualEnv_parallel.bat
@echo on
set TEST_BED=%1
set TEST_SUITE=%2
set CONFIG_SET_NAME=%3
set TEST_RESULT_PATH=%4
set REMOTE_ARG= 
set TESTSUITEPATH=%UCCE_AUTO%TestAutomation\TestSuites\PathTestSuites\%TEST_BED%\%TEST_SUITE%
IF "%TEST_SUITE%"=="" SET TESTSUITEPATH=%UCCE_AUTO%TestAutomation\TestSuites\PathTestSuites\%TEST_BED%
if defined REMOTE_REGISTRY set REMOTE_ARG=--useremotehostforregistry=%REMOTE_REGISTRY%
@echo TEST_BED=%TEST_BED%
@echo TEST_SUITE=%TEST_SUITE%
@echo TESTSUITEPATH=%TESTSUITEPATH%
@echo PYTHONPATH=%PYTHONPATH%
@echo TEST_RESULT_PATH=%TEST_RESULT_PATH%
if exist %TEST_RESULT_PATH%\ (
  echo  Test result path exists %TEST_RESULT_PATH%
  ) else (
    echo No Test result path exists. Creating %TEST_RESULT_PATH%
	MD %TEST_RESULT_PATH%
	)
cd /D %UCCE_AUTO%TestAutomation\TestSuites\PathTestSuites
@echo call python %FUNCTIONAL_HOME%\path\commands\ptf_runner.py -d %TESTSUITEPATH% -o %TEST_RESULT_PATH%\PyUnit_Auto_%TEST_SUITE%.xml %REMOTE_ARG%
call python -v %FUNCTIONAL_HOME%\path\commands\ptf_runner.py -d %TESTSUITEPATH% -o %TEST_RESULT_PATH%\PyUnit_Auto_%TEST_SUITE%.xml %REMOTE_ARG% 

Above is the batch file code for reference.

import getopt
import unittest
import sys
import os
import commands.xmlrunner
from commands.xmlrunner import TIMSXMLReport
from commands.xmlrunner import JUnitXMLReport
from commands.xmlrunner import XMLTestRunner

__author__ = 'boston'


class PTFRunner:
    test_directory = '.'
    xml_output_filename = "defaultResults.xml"
    tims_report_name = ""
    junit_report_name = ""
    checksum = True
    dataBalance = True

    def __init__(self, *args):
        self.checksum = True
        self.dataBalance = True
        self.useRemoteHostForRegistry = False
        self._parse_cmd_line(self, *args)

        # Required for 2k to get the build number from specified remote host's registry
        if self.useRemoteHostForRegistry:
            # This doesn't reallly have logging, so plain print - so that this line appears in Jenkins Log
            print("INFO: Would read remote registry from {0}".format(self.remoteHostName))
            commands.xmlrunner.use_different_host_for_build_number(self.remoteHostName)

        # Save the output XML to a specified file.
        self.tims_report_name = self.xml_output_filename.split('.')[0] + ".tims.xml"
        self.junit_report_name = self.xml_output_filename.split('.')[0] + ".junit.xml"
        # Test that we currently want to run
        self.test_suite = unittest.TestLoader().discover(self.test_directory, pattern="test_*.py")
        # add Standard Object Check sum test to the end
        if self.checksum:
            chksum_test_suite = unittest.TestLoader().discover(self.test_directory, pattern="CheckSumValidation.py")
            self.test_suite.addTests(chksum_test_suite)
        # add Standard Object Data Balance test to the end
        if self.dataBalance:
            dataBalance_test_suite = unittest.TestLoader().discover(self.test_directory, pattern="DataBalanceValidation.py")
            self.test_suite.addTests(dataBalance_test_suite)

    def _parse_cmd_line(self, *args):
        argv = sys.argv
        USAGE = """\
        putf_runner  -d <test_suite_dir> [-h -t <tims_report_name> -o <results_output>]

        Options:
          -h, --help                    Print this message
          -t, --timsreport              Specify the name for the TIMS report that is produced
          -d, --testdir                 Specify the directory where the tests reside
          -o, --output                  Output specification, where the first part is the output format, then ":", and the parameter for that specification
          --checksum                    Checksum validation enabled
          --nochecksum                  Checksum validation disabled
          --databalance                 Databalance validation enabled
          --nodatabalace                Databalance validation disabled
          --useremotehostforregistry    Use the provided hostname for reading Build Number from the Registry

        Examples:
          # Run the tests  and writes an XML output to OutputXML.ctext.xml, running the tests in the s:\mytests
          putf_runner -o xml:OutputXML.ctest.xml -d s:\mytests
        """
        alen = len(args)
        if len(args) < 2 and len(args[1]) == 0:
            sys.stderr.write(USAGE)
            exit(1)

        long_options = ['help',
                        'output=',
                        'timsreport=',
                        'testdir',
                        'nochecksum',
                        'nodatabalance',
                        'checksum',
                        'databalance',
                        'useremotehostforregistry=']

        try:
            options, args = getopt.gnu_getopt(argv[1:], "hd:o:r:", long_options)

            for opt, value in options:
                if opt in ('-h', '--help'):
                    print(USAGE)
                    exit(0)
                if opt in ('-d', '--testdir'):
                    self.test_directory = value
                    continue
                if opt in ('-o', '--output'):
                    self.xml_output_filename = value
                if opt in ('-t', '--timsreport'):
                    self.tims_report_name = value
                    continue
                if opt in ('--checksum'):
                    self.checksum = True
                    continue
                if opt in ('--nochecksum'):
                    self.checksum = False
                    continue
                if opt in ('--databalance'):
                    self.dataBalance = True
                    continue
                if opt in ('--nodatabalance'):
                    self.dataBalance = False
                    continue
                if opt in ('--useremotehostforregistry'):
                    self.useRemoteHostForRegistry = True
                    self.remoteHostName = value

        except getopt.error:
            sys.stderr.write(USAGE)
            exit(2)

    def run_tests(self):
        try:
            test_output = open(self.xml_output_filename, 'w+')
            tims_configuration = "PythonTestFramework"
            tims_reporter = TIMSXMLReport(self.tims_report_name)
            tims_reporter.set_tims_configuration_name(tims_configuration)
            junit_reporter = JUnitXMLReport(self.junit_report_name)
            runner = XMLTestRunner(test_output)
            runner.add_reporter(junit_reporter)
            runner.add_reporter(tims_reporter)
            runner.verbosity = 2
            runner.run(self.test_suite)
            # Kill all threads and exit
            os._exit(0)
        except Exception:
            test_output.close()


def main(*args):
    runner = PTFRunner(*args)
    runner.run_tests()

if __name__ == "__main__":
    main(sys.argv[1:])

Above is the ptf_runner.py which is being called from batch file.

test_dap.py (test suite code is below):

import unittest
import logging


class DropAnyParty(unittest.TestCase):
    """Drop Any Party test cases"""
    loader = None
    tstHelper = None

    @classmethod
    def setUpClass(cls):
        # Configure test folder for the results
        from packages.Components.configurepathenvironment import ConfigurePathEnvironment
        environment = ConfigurePathEnvironment(logging.INFO)
        environment.setPythonPath()

        # Go to ConfigurePathEnvironment->set_default_tracing to find and change the trace level
        environment.set_default_tracing()
        environment.set_logging_level('InteractiveProcess', logging.DEBUG)

        logging.info('BaseCallTest setupclass')

        # load database, rest interface and initialize DataModel for the test
        from PathTestSuites.Functional2K._environment.Functional2K import Functional2K
        cls.loader = Functional2K(do_not_init_periph=True)
        # TODO: OO interface uses CTI Test to accept the call until MessageSerialization starts to support ECC variable
        cls.loader.initPeripheralInterfaces("pg1", "1234",
                                            cls.loader.hosts.HOST_UCCE_PGA(cls.loader.hosts.HOST_NAME), "66666",
                                            cls.loader.hosts.HOST_UCCE_PGB(cls.loader.hosts.HOST_NAME), "77777",
                                            enableMRSim=False, enableCTITest=False)
        cls.loader.initPeripheralInterfaces("pg2", "1235",
                                            cls.loader.hosts.HOST_UCCE_PGA(cls.loader.hosts.HOST_NAME), None,
                                            cls.loader.hosts.HOST_UCCE_PGB(cls.loader.hosts.HOST_NAME), None,
                                            enableMRSim=False)
        cls.loader.initPeripheralInterfaces("pg3", "1236",
                                            cls.loader.hosts.HOST_UCCE_PGA(cls.loader.hosts.HOST_NAME), "0987",
                                            cls.loader.hosts.HOST_UCCE_PGB(cls.loader.hosts.HOST_NAME), None,
                                            enableMRSim=False)
        # Enable DropAnyParty on CTI server through procmon
        cls.enable_disable_drop_any_party("CG1A", cls.loader.hosts.HOST_UCCE_PGA, enable='1')
        cls.enable_disable_drop_any_party("CG1B", cls.loader.hosts.HOST_UCCE_PGB, enable='1')


    @classmethod
    def tearDownClass(cls):
        logging.info('DAP teardownclass')
        # Revert DropAnyParty on CTI server through procmon
        cls.enable_disable_drop_any_party("CG1A", cls.loader.hosts.HOST_UCCE_PGA, enable='0')
        cls.enable_disable_drop_any_party("CG1B", cls.loader.hosts.HOST_UCCE_PGB, enable='0')
        cls.loader.close_interfaces()

    def setUp(self):
        from PathTestSuites.Functional2K.Functional2KFull.OOHACMgrFailover.restartservices import RestartServices
        from packages.Components.testconfig import TestConfig
        TestConfig.getInstance().set_test_case_start_datetime()
        restart_loggers = RestartServices(self.loader, DropAnyParty.setUp.__qualname__)
        restart_loggers.restart_services()
        self.loader.set_debug_router_tracing()
        logging.info('DAP setup')

    def tearDown(self):
        """
            Logout both the agents.
            Close and Exit the CTISession for the peripheral

        Returns:

        """
        logging.info('TearDown started.')
        self.loader.logout_all_agents()
        self.tstHelper.traceEndTest(DropAnyParty.__qualname__)
        logging.info('DAP teardown ended')

    @classmethod
    def enable_disable_drop_any_party(cls, node, host, enable='0'):
        from packages.RunnerHelp.runnerhelper import RunnerHelper
        runnerHelper = RunnerHelper(cls.loader)
        runnerHelper.start_procmon(node, "ctisvr", host(cls.loader.hosts.HOST_NAME))
        runnerHelper.send_procmon_command("reg_cd ..\\..\\..\\..\\..\\CG\\CurrentVersion\\CTIServer\\Dynamic")
        runnerHelper.send_procmon_command("reg_set DropAnyPartyEnabled " + enable)
        validation_string = "DropAnyPartyEnabled.*0x" + enable
        runnerHelper.validateExtProcessResp(runnerHelper.procmon_process, validation_string, 10)
        runnerHelper.stop_procmon()

    def testTransferANI(self):
        from PathTestSuites.Functional2K.Functional2KFull.DropAnyParty.TransferANI import TransferANI
        self.tstHelper = TransferANI(self.loader, DropAnyParty.testTransferANI.__qualname__)
        self.tstHelper.runtestcase()

    def testTransferANIWithQ(self):
        from PathTestSuites.Functional2K.Functional2KFull.DropAnyParty.TransferANIWithQ import TransferANIWithQ
        self.tstHelper = TransferANIWithQ(self.loader, DropAnyParty.testTransferANIWithQ.__qualname__)
        self.tstHelper.runtestcase()

Cmd window is getting frozen forever after one test case is completed as attached in the below screenshot,

Given you are catching up with 8+ years of python changes best you debug your code. No way anyone can guess what change is responsible for your code breaking.

FYI you can copy the text from a CMD window and paste as text here.
No need for a screen shot.

1 Like

Thank you for the response !

Wondering how does the same code working through IDE(Pycharm).
We are new to this code. What are some key areas to focus on as we start our deep dive?

You need to find where the code is hanging.
Add debug logging to have the code revive where it is in the test frame work.

Python 3.4 init.py (C:\Python34\Lib\unittest) code as below,

__all__ = ['TestResult', 'TestCase', 'TestSuite',
           'TextTestRunner', 'TestLoader', 'FunctionTestCase', 'main',
           'defaultTestLoader', 'SkipTest', 'skip', 'skipIf', 'skipUnless',
           'expectedFailure', 'TextTestResult', 'installHandler',
           'registerResult', 'removeResult', 'removeHandler']


# Expose obsolete functions for backwards compatibility

__all__.extend(['getTestCaseNames', 'makeSuite', 'findTestCases'])

__unittest = True

from .result import TestResult
from .case import (TestCase, FunctionTestCase, SkipTest, skip, skipIf,
                   skipUnless, expectedFailure)

from .suite import BaseTestSuite, TestSuite
from .loader import (TestLoader, defaultTestLoader, makeSuite, getTestCaseNames,
                     findTestCases)
from .main import TestProgram, main
from .runner import TextTestRunner, TextTestResult
from .signals import installHandler, registerResult, removeResult, removeHandler

# deprecated
_TextTestResult = TextTestResult

Python 3.12 init.py (C:\Python312\Lib\unittest) code as below,

__all__ = ['TestResult', 'TestCase', 'IsolatedAsyncioTestCase', 'TestSuite',
           'TextTestRunner', 'TestLoader', 'FunctionTestCase', 'main',
           'defaultTestLoader', 'SkipTest', 'skip', 'skipIf', 'skipUnless',
           'expectedFailure', 'TextTestResult', 'installHandler',
           'registerResult', 'removeResult', 'removeHandler',
           'addModuleCleanup', 'doModuleCleanups', 'enterModuleContext']

# Expose obsolete functions for backwards compatibility
# bpo-5846: Deprecated in Python 3.11, scheduled for removal in Python 3.13.
__all__.extend(['getTestCaseNames', 'makeSuite', 'findTestCases'])

__unittest = True

from .result import TestResult
from .case import (addModuleCleanup, TestCase, FunctionTestCase, SkipTest, skip,
                   skipIf, skipUnless, expectedFailure, doModuleCleanups,
                   enterModuleContext)
from .suite import BaseTestSuite, TestSuite
from .loader import TestLoader, defaultTestLoader

from .main import TestProgram, main
from .runner import TextTestRunner, TextTestResult
from .signals import installHandler, registerResult, removeResult, removeHandler
# IsolatedAsyncioTestCase will be imported lazily.
from .loader import makeSuite, getTestCaseNames, findTestCases

# Lazy import of IsolatedAsyncioTestCase from .async_case
# It imports asyncio, which is relatively heavy, but most tests
# do not need it.

def __dir__():
    return globals().keys() | {'IsolatedAsyncioTestCase'}

def __getattr__(name):
    if name == 'IsolatedAsyncioTestCase':
        global IsolatedAsyncioTestCase
        from .async_case import IsolatedAsyncioTestCase
        return IsolatedAsyncioTestCase
    raise AttributeError("module {__name__!r} has no attribute {name!r}")

While finding the difference between above version code,

Lazy Imports in Python 3.12 vs. Direct Imports in Python 3.4

In the unittest module’s __init__.py file, Python 3.4 directly imports makeSuite , getTestCaseNames , and findTestCases from the loader submodule. Python 3.12, however, implements lazy imports for these functions, meaning they’re only imported when explicitly requested.

Here’s a breakdown of the difference:

Python 3.4:

  • When the __init__.py file is imported, all functions, including the ones from loader, are loaded upfront .
  • Even if these functions are not immediately used, they still occupy memory and contribute to loading time.

Python 3.12:

  • Instead of eagerly importing everything, __init__.py only imports the necessary parts (TestResult , TestCase , etc.) at the beginning.
  • If needed, makeSuite , getTestCaseNames , and findTestCases are imported on demand using the from .loader import syntax within the relevant parts of the unittest module.
  • This avoids unnecessary imports for users who might not use these specific functions, leading to improved efficiency and reduced memory usage .

Is this causing issue?

I would not spend time looking for differences but instead look for why the test code breaks.

Once you know why the code break you can work on a fix.

1 Like