AI for Testing> AI for Coding
AI for testing and security might be a better use case than coding. Adobe tried it, maybe you can, too.
We had a contest called AI Prodigy, held at Simform last year.
The participants were to develop tools, workflows, frameworks, etc., to show how AI could improve efficiency in various activities.
We got lots of interesting submissions.
There was also one from a few members of the QA team. They called it TESTCASE.AI.
Using it, you could get test cases generated directly in your Test Rail project by only providing the names of features or modules as input.
This tool helped save time AND allowed engineers to focus on what’s important.
While AI coding assistants have become so popular lately, the use of AI to improve testing and security intrigues me a lot.
In this edition, we’ll go through
Solving QA challenges with AI-powered test automation
A test automation strategy breakdown
How Adobe used ML to classify and address security issues better
Solve QA challenges with AI-powered test automation
Many processes and tasks in test automation can benefit from the application of AI, potentially expediting and streamlining the testing process. Here are seven ways AI can help improve test automation:
1. Visual Verification - AI algorithms like machine learning can potentially fill gaps in automation tools, ensuring the UI appears correctly and elements are positioned accurately. This could help verify color, shape, size, and ensure elements don't overlap.
2. Codeless Test Automation - AI can be used to develop codeless test tools that understand business flows, written in plain English, and reduce time spent on test script creation. This may minimize the dependency on automation engineers and speed up the testing process.
3. Robust Element Locators - Training an AI model on a large data set involving various websites could create more efficient weighted locator sets, making it easier to locate dynamic UI elements and reducing script failures.
4. Spidering/Web Crawling - AI-powered tools can potentially automate web crawling and test script generation, saving significant time and effort in web application testing. These tools can automatically explore the application and generate test scripts in a much shorter time frame.
5. Automatic Triaging & Defects Assignment - AI engines can potentially classify and prioritize defects based on severity and feature, automating the assigning process and helping developers predict and fix similar defects. This could save time and effort for development teams.
6. Self-healing Test Scripts - AI can potentially help maintain test scripts by self-healing when faced with locator changes, leading to more stable and reliable test automation. AI frameworks could modify affected scripts and suggest new locators as needed.
7. Test Report Analysis - Machine learning can potentially analyze test reports, generating predictive analysis on build health based on historical and current data. This would enable stakeholders to make better-informed decisions about their application's quality.
You can read more about AI-powered test automation here.
A test automation strategy breakdown
Even though AI could be a huge help, you still need a sound test automation strategy.
Here are the steps you should follow to build one.
You can learn more about test automation strategy here.
How Adobe used ML to classify and address security issues better
Quality assurance is all about risk protection, and so is software security.
Adobe Security found itself managing a large number of security issues, making it challenging to see patterns and identify core areas of weakness. By applying machine learning and data analytics, they aimed to shift focus from day-to-day problem-solving to gaining better insight into their security concerns.
To address this challenge, they began by creating six basic yet comprehensive security issue categories, simplifying the classification process and making it easier for developers to address common attack avenues exploited by adversaries.
Using machine learning models and natural language processing techniques, they prepared and pre-processed the necessary data for classification. They cleaned the text, tokenized it with the n-gram approach, and created numerical vectors by employing the TF-IDF vectorization technique.
An MLP (Multi-layer Perceptron) deep-learning model was built to classify the security issues based on the prepared data. However, the initial model had a high rate of false positives and imbalanced data, leading the team to try downsampling and upweighting techniques.
When these measures didn't significantly improve the model's performance, they switched to an ensemble model approach, training binary classifiers for each security category and aggregating their outputs for the final prediction.
This new approach significantly improved performance with an accuracy rate of about 92%, enabling Adobe Security to identify patterns and address core weaknesses more efficiently. By leveraging machine learning and data analytics, they successfully shifted from day-to-day problem-solving to strategic insights and more effective security issue management.
You can dive deeper into the details here.