Testing in the Age of AI

Artificial Intelligence (AI) is transforming how applications function, and one of the most exciting developments is Retrieval-Augmented Generation (RAG). Unlike traditional AI models that rely solely on pre-trained data, RAG fetches real-world, up-to-date information before generating responses.

For QA professionals, this introduces new challenges. Traditional testing methods are designed for predictable outputs, but AI applications, especially those using RAG, produce dynamic, context-aware responses that change over time. So, how do we test such applications effectively? Let us explore.

Understanding RAG in AI Applications

Artificial Intelligence has significantly improved in answering questions, assisting users, and generating meaningful insights. However, traditional AI models only rely on pre-trained data and cannot fetch or process new information once training is complete.

This means that a regular AI model cannot adapt to new events, trends, or real-time updates—a major limitation in many real-world scenarios.

Retrieval-Augmented Generation (RAG) solves this by allowing AI to fetch real-time information before generating responses, making AI much more dynamic and useful.

Example: AI-Powered Movie Recommendation Assistant

Imagine you have an AI chatbot that recommends movies based on user preferences.

Without RAG (Traditional AI Model)

A user asks:
"What are the top trending movies right now?"

Chatbot Response:
"Some popular movies are Inception, The Dark Knight, and Interstellar."

Problem:

• The AI recommends old movies because it relies only on its pre-trained data.
• If the user wants new, trending movies, the AI fails to provide relevant information.

With RAG (AI + Real-Time Data Retrieval)

A user asks the same question:
"What are the top trending movies right now?"

 Step 1: Retrieval → The AI fetches real-time trending movie data from sources like IMDb or Rotten Tomatoes.
Step 2: Generation → The AI processes the retrieved data and generates a response.

Chatbot Response:
Here are the top trending movies this week:

1️Dune: Part Two - 8.5/10 IMDb
2️Oppenheimer - 8.3/10 IMDb
3️Spider-Man: Across the Spider-Verse - 8.5/10 IMDb

Would you like recommendations based on your favorite genre?"*

Why is this better?

• The AI provides up-to-date movie recommendations.
• Users get relevant suggestions based on current trends.
• The AI is more dynamic and useful compared to a static model.

Challenges in Testing AI and RAG-Based Applications

Challenge 1: Unpredictable Responses

AI outputs are not static. The same input might result in different answers depending on the retrieved data.

Example:

• Query: "What are the latest stock prices for Tesla?"
• Response Today: "$600 per share."
• Response Tomorrow: "$610 per share."

Since responses change, traditional assertion-based testing (expected == actual) fails.

Challenge 2: Data Freshness and Accuracy

If the RAG model retrieves outdated or incorrect data, the response can be misleading.

Example: A financial AI assistant might retrieve last week’s stock prices instead of today’s.

Challenge 3: Bias and Hallucinations

AI models sometimes make up facts (hallucinations) or reflect biases from retrieved data.

Example: A medical chatbot might suggest outdated treatments because it fetched an old research paper instead of the latest one.

Challenge 4: Performance Bottlenecks

Since RAG-based models fetch data dynamically, they might introduce latency issues.

Example: If an AI-driven legal assistant needs to retrieve 100+ policy documents, it could slow down the user experience.

How to Test RAG and AI-Driven Applications?

Since AI responses aren’t deterministic, we need adaptive testing strategies.

Test 1: Consistency and Stability Testing

Instead of checking exact words, validate if the response meaning remains consistent.

Code Example: Using AI-Based Validation

import { test, expect } from '@playwright/test';
import { SentenceTransformer } from 'sentence-transformers';

test('AI response consistency test', async ({ page }) => {
    const model = new SentenceTransformer('all-MiniLM-L6-v2');

    const expected = "Elon Musk is the CEO of Tesla.";
    const aiResponse = "Tesla's current CEO is Elon Musk.";

    const embedding1 = await model.encode(expected);
    const embedding2 = await model.encode(aiResponse);

    const similarity = embedding1.dot(embedding2) / (embedding1.norm() * embedding2.norm());

    expect(similarity).toBeGreaterThan(0.85);
});

Why?

• It checks semantic similarity, not exact text, ensuring flexibility in AI responses.

Test 2: Data Freshness and Relevance Checks

AI models should retrieve recent and accurate data.

Example: If a news AI assistant retrieves a week-old article instead of today’s news, it should be flagged.

Code Example: Verifying Data Timestamp

import { test, expect } from '@playwright/test';

test('Check AI data freshness', async ({ page }) => {
    await page.goto('https://news-ai.com');

    const articleDateText = await page.locator('.article-date').textContent();
    const articleDate = new Date(articleDateText!);
    const currentDate = new Date();

    const timeDifference = Math.abs(currentDate.getTime() - articleDate.getTime());
    const daysDifference = timeDifference / (1000 * 3600 * 24);

    expect(daysDifference).toBeLessThan(2);
});

Test 3: Hallucination and Bias Detection

AI responses should be fact-checked and neutral.

Example: If an AI says, “Smoking is completely harmless,” it should be flagged as misinformation.

How?

• Use AI-generated test cases to simulate biased inputs and verify AI’s response neutrality.

Test 4: Load and Performance Testing

Ensure that retrieving data dynamically doesn’t introduce delays.

Code Example: Measuring API Response Time

import { test, expect } from '@playwright/test';

test('AI response time validation', async ({ request }) => {
    const start = Date.now();
    const response = await request.get('https://weather-ai.com/api?location=NewYork');
    const end = Date.now();

    const duration = end - start;
    expect(duration).toBeLessThan(2000); // Expect response time under 2 seconds
});

AI-Assisted Testing: Using AI to Test AI

Testing AI manually is inefficient. AI itself can help in testing.

Self-Healing Tests

Self-healing tests adapt automatically when UI changes, such as button renaming or locator restructuring. Instead of hardcoding selectors, we can make our test more resilient using:

1️ AI-powered locators (using fuzzy matching).
2️ Multiple locator strategies (e.g., trying ID, text, and role-based locators).
3️ Handling missing elements gracefully instead of failing immediately.

import { test, expect } from '@playwright/test';

test('Self-healing AI search validation', async ({ page }) => {
    await page.goto('https://ai-search.com');

    // Define multiple fallback locators for the search input field
    const searchBoxLocators = [
        page.locator('#search-box'), // ID-based
        page.locator('input[placeholder*="search"]'), // Placeholder-based (fuzzy matching)
        page.locator('role=textbox[name*="search"]') // Accessible role-based
    ];

    // Try different locators until one works
    let searchBoxFound = false;
    for (const locator of searchBoxLocators) {
        if (await locator.count() > 0) {
            await locator.fill('Latest AI trends');
            searchBoxFound = true;
            break;
        }
    }
    expect(searchBoxFound).toBeTruthy(); // Ensure we found a valid search box

    // Define multiple fallback locators for the search button
    const searchButtonLocators = [
        page.locator('#search-button'), // ID-based
        page.locator('button:has-text("Search")'), // Text-based (robust against renaming)
        page.locator('role=button[name*="search"]') // Accessible role-based
    ];

    let searchButtonFound = false;
    for (const locator of searchButtonLocators) {
        if (await locator.count() > 0) {
            await locator.click();
            searchButtonFound = true;
            break;
        }
    }
    expect(searchButtonFound).toBeTruthy(); // Ensure we clicked a valid button

    // Validate search results appear dynamically, using AI-friendly assertions
    const results = await page.locator('.result-item').allTextContents();
    expect(results.some(result => /AI/i.test(result))).toBeTruthy(); // Ensure at least one result mentions AI
});

Why This is Better?

• Self-Healing Mechanism → If a locator fails, it tries other strategies before failing.
• More Robust Against UI Changes → Uses text-based, role-based, and attribute-based locators.
• Fuzzy Matching for Better Adaptability → Works even if UI elements slightly change (e.g., "Search Now" instead of "Search").
• AI-Friendly Assertions → Uses pattern matching instead of exact string comparisons.

This approach ensures that even if minor UI changes occur, the test continues running successfully. 

Synthetic Data Generation

AI can generate edge case test data.

Example: AI chatbot should handle misspellings and synonyms.

Code Example: Generating Test Cases Using OpenAI API

import openai from 'openai';

openai.apiKey = "your-api-key";

async function generateTestCases() {
    const response = await openai.ChatCompletion.create({
        model: "gpt-4",
        messages: [{ role: "user", content: "Generate 5 variations of: 'I want to cancel my order'" }]
    });

    console.log("Generated Test Cases:\n", response.choices[0].message.content);
}
generateTestCases();

Final Thoughts

Testing AI applications is not about finding fixed bugs—it is about validating intelligence.

• Automated AI Validators → Test meaning, not just words.
• Self-Healing Tests → Adapt to UI changes.
• Synthetic Data Generation → Generate diverse test cases.

By leveraging AI-assisted testing, QA professionals can ensure AI systems remain accurate, unbiased, and efficient.