Setting Up a Test Project

This guide will walk you through the process of setting up a test project with the Factifai Agent Suite. By the end, you'll have a fully functional test environment ready for creating and running automated tests.

Prerequisites

Before you begin, make sure you have:

• Node.js 18+ installed on your system
• npm or pnpm for package management
• An OpenAI API key or Azure OpenAI credentials or AWS Bedrock credentials for the AI capabilities

Step 1: Install Factifai Agent

First, install the Factifai Agent CLI tool globally:

# Using npm
npm install -g @presidio-dev/factifai-agent

# Using pnpm
pnpm add -g @presidio-dev/factifai-agent

Step 2: Install Playwright Dependencies

Factifai Agent uses Playwright for browser automation. First install Playwright globally, then install the necessary browser dependencies:

# Install Playwright globally
npm install -g playwright

# Install browser dependencies
npx playwright install --with-deps

This process installs browser binaries for Chromium, Firefox, and WebKit, along with any necessary system dependencies.

Step 3: Configure API Credentials

Configure your API credentials for the LLM provider you want to use:

For OpenAI

factifai-agent config --set OPENAI_API_KEY=your-api-key-here

For Azure OpenAI

factifai-agent config --set AZURE_OPENAI_API_KEY=your-api-key-here
factifai-agent config --set AZURE_OPENAI_API_INSTANCE_NAME=your-instance-name
factifai-agent config --set AZURE_OPENAI_API_DEPLOYMENT_NAME=your-deployment-name
factifai-agent config --set AZURE_OPENAI_API_VERSION=your-api-version

For AWS Bedrock

factifai-agent config --set AWS_ACCESS_KEY_ID=your-access-key-id
factifai-agent config --set AWS_SECRET_ACCESS_KEY=your-secret-access-key
factifai-agent config --set AWS_DEFAULT_REGION=us-west-2

Step 4: Create a Project Directory

Create a directory for your test project:

mkdir my-factifai-tests
cd my-factifai-tests

Step 5: Initialize a Package

Initialize a package.json file:

npm init -y

This creates a basic package.json file that you can customize later.

Step 6: Create a Tests Directory

Create a directory for your test files:

mkdir tests

Step 7: Create Your First Test File

Create a simple test file to verify your setup:

touch tests/first-test.txt

Open the file in your editor and add a simple test:

**Objective:** Verify DuckDuckGo Search

**Test Steps:**

1. **Navigate to duckduckgo.com**
   * **Expected:** DuckDuckGo homepage loads

2. **Search for "automated testing"**
   * **Action:** Type "automated testing" in search box and press Enter
   * **Expected:** Search results for "automated testing" appear

3. **Verify search results**
   * **Expected:** At least 5 search results are displayed
   * **Expected:** The page title contains "automated testing"

Step 8: Run Your First Test

Run your first test using the Factifai Agent CLI:

# Using OpenAI
factifai-agent --model openai run --file tests/first-test.txt

# Using Azure OpenAI
factifai-agent --model azure-openai run --file tests/first-test.txt

# Using AWS Bedrock
factifai-agent --model bedrock run --file tests/first-test.txt

You should see the test execute in real-time, with the browser opening, navigating to DuckDuckGo, performing the search, and verifying the results.

Step 9: Review the Test Results

After the test completes, review the results in the terminal. You should see:

• A summary of the test execution
• Details for each step
• Screenshots taken during the test
• Any errors or warnings

Step 10: Create a Test Script in package.json

Add a test script to your package.json file:

{
  "scripts": {
    "test": "factifai-agent run --file tests/first-test.txt"
  }
}

Now you can run your tests using:

npm test

Coming Soon: Configuration file support will be available in a future release, allowing you to set default options like model, browser, timeout, and retry settings.

Step 12: Create Additional Test Files

Create additional test files for different features or scenarios:

touch tests/login-test.txt
touch tests/checkout-test.txt

Add appropriate test steps to each file.

Step 13: Organize Your Tests

As your test suite grows, consider organizing your tests into directories:

mkdir -p tests/e2e
mkdir -p tests/regression
mkdir -p tests/smoke

# Move or create tests in these directories
touch tests/e2e/full-user-journey.txt
touch tests/regression/login-scenarios.txt
touch tests/smoke/critical-paths.txt

Step 14: Set Up a CI/CD Configuration

If you're using a CI/CD system, create a configuration file for it. For example, for GitHub Actions:

mkdir -p .github/workflows
touch .github/workflows/tests.yml

Add the following content to the file:

name: Factifai Tests

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
      - name: Install dependencies
        run: |
          npm install -g @presidio-dev/factifai-agent
          npx playwright install --with-deps
      - name: Configure Factifai Agent
        run: |
          factifai-agent config --set OPENAI_API_KEY=${{ secrets.OPENAI_API_KEY }}
      - name: Run tests
        run: |
          factifai-agent run --file tests/first-test.txt

Coming Soon: Directory-based test running with the --dir option will be available in a future release, allowing you to run all tests in a directory.

Step 15: Create a README

Create a README.md file to document your test project:

touch README.md

Add information about your test project, how to run the tests, and any other relevant details.

Next Steps

Now that you have set up your test project, you can:

• Integrate with CI/CD pipelines

Coming Soon:

• Creating more complex test suites
• Writing effective test instructions
• Cross-browser testing

Troubleshooting

Browser Dependencies

If you encounter issues with browser dependencies, try reinstalling them:

npx playwright install --with-deps

API Key Issues

If you have issues with your API key, verify it's correctly set:

factifai-agent config --show

Permission Issues

If you encounter permission issues when running tests, make sure you have the necessary permissions for the directories you're using.

Browser Launch Issues

If the browser fails to launch, check if you have any conflicting browser processes running or if your system has the necessary dependencies installed.