> ## Documentation Index
> Fetch the complete documentation index at: https://vvennela-ftl-18-95.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Adversarial Testing

> Parallel test generation that runs alongside the coding agent to catch bugs early

FTL can generate adversarial tests in parallel with the coding agent, creating tests that actively try to break the implementation before the code is even complete.

## How It Works

The adversarial testing system runs a separate model alongside the primary coding agent:

1. **Task-based generation** - Tests are generated from the task description alone, without seeing the implementation
2. **Parallel execution** - The tester runs at the same time as the coder, not after
3. **Adversarial focus** - Tests target edge cases, null inputs, boundary conditions, and unexpected usage
4. **Real API testing** - No mocking - tests use actual API credentials from the environment

```python Test Generation theme={null}
from ftl.planner import generate_tests_from_task

# Generate tests from just the task description
test_code = generate_tests_from_task(
    task="Create a function that validates email addresses",
    model="claude-3-5-sonnet-20241022"
)
```

```python Running Tests theme={null}
from ftl.planner import run_test_code

exit_code, output = run_test_code(
    test_code=test_code,
    sandbox=sandbox,
    console=console
)

if exit_code == 0:
    print("Tests passed")
else:
    print(f"Tests failed: {output}")
```

## Test Generation Strategy

The test generator receives this system prompt:

> [!NOTE]
>
> You are an adversarial test engineer. Given a coding task description, generate a test script that verifies the implementation is correct and tries to break it. Focus on edge cases, null inputs, boundary conditions, and unexpected usage. Your goal is to find bugs.

Key characteristics:

* **Language detection** - Automatically detects Python vs JavaScript from code patterns
* **Auto-installation** - Installs missing test dependencies (pytest, etc.) on first failure
* **Real credentials** - Uses environment variables for actual API testing

## Language Support

The system detects the language from the generated test code:

```python Source: ftl/planner.py:63-68 theme={null}
_js_signals = ("const ", "let ", "var ", "describe(", "it(", "require(")
is_js = (
    test_code.strip().startswith(_js_signals)
    or "from '" in test_code[:300]
    or 'from "' in test_code[:300]
)
```

* **Python** - Uses `pytest` with `-v` flag
* **JavaScript** - Runs with `node` directly

## Manual Verification

You can manually trigger verification on code changes:

```python Source: ftl/planner.py:97-112 theme={null}
def run_verification(diffs, tester, sandbox):
    """Manual test trigger: generate tests from diff and run them."""
    from ftl.diff import diff_to_text

    console = Console()
    console.print(f"[bold]Running verification ({tester})...[/bold]")

    if tester in AGENTS:
        agent = get_agent(tester)
        diff_text = diff_to_text(diffs)
        task = (
            "Review the following code changes and write tests that try to break them. "
            "Focus on edge cases, null inputs, boundary conditions, and unexpected usage. "
            "Run the tests and report results.\n\n"
            f"{diff_text}"
        )
```

> [!TIP]
>
> Verification can use either a specific model or a full FTL agent for more sophisticated test generation.

## Missing Dependencies

The test runner automatically handles missing modules:

```python Source: ftl/planner.py:80-84 theme={null}
# If tests failed due to missing modules, install them and retry once.
missing = _extract_missing_modules(stdout + stderr)
if missing and exit_code != 0:
    sandbox.exec(f"pip install {' '.join(missing)} -q")
    exit_code, stdout, stderr = sandbox.exec(run_cmd)
```

This extracts package names from `ModuleNotFoundError` messages and installs them automatically.

## Configuration

Adversarial testing is activated by passing a model name to the planner:

```bash theme={null}
ftl --tester claude-3-5-sonnet-20241022
```

The tester can be:

* A LiteLLM model name (e.g., `claude-3-5-sonnet-20241022`)
* An FTL agent name for more advanced testing

## Benefits

> [card] Catch Bugs Early
>
> Find issues before the agent even finishes coding

> [card] No Mocking
>
> Tests use real APIs with actual credentials

> [card] Edge Case Focus
>
> Adversarial mindset finds boundary conditions

> [card] Auto-Install
>
> Missing test dependencies installed automatically


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.