Governace of Automated Test Framework

 

Automated Test Framework - Best Practices & Governance

Some Considerations

When writing test cases with ATF, I always recommend checking out the Quick Start Tests to see if they can benefit your work.

For creating customized test cases, be sure to follow ATF best practices.

If you have a CICD pipeline (AEMC pipeline) configured, make sure to write customized test cases for the scoped app. Remember to include them in the Application Deployment Test Suite so they can run automatically when deploying the scoped app to another instance.

For more info: 

• https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0862335 
• https://www.youtube.com/watch?v=ZX47LgZpwmU&ab_channel=ServiceNowCommunity
• https://docs.servicenow.com/bundle/xanadu-application-development/page/build/pipelines-and-deployments/concept/pipelines-deployments-workflow-vs2.html 

 Best Practices

ATF is a ServiceNow application purpose-built for automated testing within a ServiceNow instance to confirm that the instance still works as designed after being upgraded or modified. ATF is active by default in the platform and is available at no extra license cost. ATF reduces testing time and costs by creating reusable and automated tests that are repeatable & UI independent – reducing overall project costs and schedule.

What best practices should I consider when creating ATF tests?

Scoped application - Always create test cases in the related scoped application

Naming format for test case - ATF <Module:><Short description of test case>

Example: ATF CHANGE: Copy change on Normal Change Request

Usage of dummy records - No use of existing records, instead use dummy records. Move all the dummy records and test cases to the PRD once completed. Also while filling the dates using test steps, you need to use the javascript and generate the date dynamically instead keeping it static.

Align ATF to key functional test scenarios - ATF is intended for functional testing of your specific business processes. Testing should focus on key user flows or process flows within your ServiceNow applications.

Impersonate first - Typically, the first test step of every test impersonates the appropriate user to perform the work. To impersonate a user, use the Create a User test step to create a user with specified roles and groups for the test. The user record gets rolled back after the test completes.

Group tests - Building and running automated test suites allows you to group tests in a specific order to test an application or a group of related features. This way, you’re able to run tests and see results as one job.

Run tests in parallel - Reduce testing time by running multiple tests and test suites in parallel. Run tests that create their own data to prevent resource conflicts and data dependencies. You can prevent conflicting tests from running in parallel by marking tests as mutually exclusive. This ensures that tests that require the same data don’t run simultaneously.

Take a timeout - User interface (UI) test steps have an intelligent built-in wait mechanism that requires the UI change to complete before the next UI test step proceeds. However, asynchronous updates on the server, such as event processing, workflow updates, and email notifications, might require the test designer to incorporate an additional wait for processing to complete. Setting the timeout on server test steps can also be useful when debugging tests. For example, you could incorporate a 30- to 60-second timeout to manually inspect the state of records before the rollback.

Keep it short - Test a discrete set of items during testing. This makes it easier to build tests and evaluate any failures. It is easy to string multiple tests together as part of a Suite and you can even use test suites hierarchically. This allows you to perform additional testing should an early step in a test fail.

Avoid test step repetition - Don’t repeat testing for the same UI functionality in multiple tests. If other tests require similar functionality, simulate it using server test steps rather than repeating the same UI test steps. Server test steps perform faster.

Test the outcome - Focus on the outcome to verify rather than testing each point along the way. Keep this in mind when verifying business rules or Flow Designer processes. When testing flows, build the conditions to trigger the flow in the test, then check the flow outcome for the expected result.

Consider access control rule (ACL) security - When debugging unexpected test errors, a common mistake relates to security authorization. Based on the roles of the impersonated user, a test step might fail. By default, ACL security is enforced for record inserts, updates, and deletes. Sometimes this setting must be disabled for test steps performing actions to set up a test. Other test steps actually performing the verification might require it to be enabled.

Validate record updates - Include a Record Validation test step after every Record Update test step to ensure the record was actually updated. A Record Update appears to always pass even if the record was not updated for any reason, such as being rejected by a data policy. This can lead to confusing failures in subsequent test steps. It can also mislead you in trying to troubleshoot the wrong test step.

Test functionality instead of data - Rather than test every possible combination of a data-driven feature, create one or two tests focusing on the functionality. Then verify that the values change as intended without spending too many cycles on all the possible combinations of a vast data matrix.

Approach conditional testing differently - When you want to perform ATF test steps based on given criteria or branch a test along different paths, create a separate test for each condition.

Running test cases

Schedule ATF test suits - Once in a non business day (Sunday) and 3 times per week (Monday, Wednesday and Thursday) at 9 PM CET. Check the status of each test suit on next business day

When process is changed - When something is configured on the modules for which ATF cases are created, need to run the whole suite for the module and then only changes has to be moved to PRD if no error seen in ATF.