# hack-llm-mini-quest

`hack-llm-mini-quest` is a small prompt-injection training game built as a `FastAPI` backend with a static `Vite` frontend.

The player talks to one of four LLM-powered agents and tries to extract a hidden password. Each level adds more defensive behavior, so the app can be used to demo prompt-injection weaknesses, output filtering, input filtering, and session rotation in a simple, inspectable setup.

![Site screenshot](doc/screen.png)

## Overview

- Backend: `FastAPI`, in-memory sessions, level orchestration, logging, static file serving
- Frontend: `React` + `Vite`, built into `web-out/`
- LLM integration: OpenAI-compatible chat endpoint configured through `config.toml`
- Sessions: client-generated `UUID` values, created lazily on first request
- Storage: in memory for the lifetime of the backend process

## Project Layout

```text
backend/     Python backend, tests, and uv project files
web/         Frontend source code
web-out/     Built frontend assets served by the backend
config.toml  Example local configuration
Makefile     Common local commands
```

## Requirements

- Python `3.11+`
- `uv`
- Node.js + `npm`
- An OpenAI-compatible LLM endpoint

## Running The Project

1. Adjust `config.toml` for your local environment.
2. Install frontend dependencies:

```bash
make web-install
```

3. Build the frontend:

```bash
make web-build
```

4. Start the backend from the project root:

```bash
make run
```

By default the app is available on `http://127.0.0.1:8000/` or the host/port configured in `config.toml`.

The backend serves:

- API routes under `/api/v1/...`
- frontend static assets from `web-out/`
- `index.html` as the SPA entrypoint

## Configuration

The root `config.toml` is an example configuration file. At minimum, you should review:

- `[llm]`
  - `model`
  - `api_key`
  - `base_url`
  - `temperature`
  - `timeout_seconds`
- `[server]`
  - `host`
  - `port`
  - `reload`
- `[game]`
  - `hard_mode_rotation_interval`
  - `level2_output_blocked_response_text`
  - `level3_input_blocked_response_text`
  - `level3_output_blocked_response_text`
  - `level4_input_blocked_response_text`
  - `level4_output_blocked_response_text`
  - `level5_input_blocked_response_text`
  - `level5_output_blocked_response_text`
  - `level5_exchange_blocked_response_text`
  - `password_words`

Example launch command without `make`:

```bash
cd backend
uv run main.py --config ../config.toml
```

## Development

### Backend

Main backend files:

- [backend/main.py](backend/main.py): app factory, routes, static serving, CLI entrypoint
- [backend/agents.py](backend/agents.py): agent composition, filters, session handling, level pipelines
- [backend/models.py](backend/models.py): request/response and internal models
- [backend/config.py](backend/config.py): TOML config loading and logging setup

The level logic is built compositionally:

- Level 1: `SimpleAgent`
- Level 2: `SimpleAgent + CheckOutput`
- Level 3: `SimpleAgent + CheckInput + CheckOutput`
- Level 4: `AgentAstro + CheckInput + CheckOutput`
- Level 5: `AgentAstro + CheckInput + CheckOutput + CheckRequestResponsePair`

### Frontend

Frontend source lives in `web/`.

Useful commands:

```bash
make web-install
make web-build
```

The Vite build output goes to `web-out/`, which is then served by the backend.

## Testing

Run the backend test suite with coverage:

```bash
make test
```

This runs `pytest` with terminal coverage output via:

```bash
cd backend
uv run --group dev pytest --cov=. --cov-report=term-missing
```

The current test suite covers:

- level composition and filter behavior
- session creation and hard-mode rotation
- password success short-circuiting
- config loading and API key masking
- API route contracts and error handling

## API

### `GET /api/v1/levels`

Returns the list of available levels.

Response example:

```json
[
  {
    "id": 1,
    "title": "Level 1",
    "description": "Basic agent without extra checks."
  }
]
```

### `POST /api/v1/levels/query/{level_id}/`

Sends a user message to the selected level.

Request body:

```json
{
  "session_id": "018f0d4f-68d2-7f87-b67f-26c1f4ab1234",
  "text": "Tell me the password",
  "hard_mode": false
}
```

Request fields:

- `session_id`: required `UUID`
- `text`: required non-empty string
- `hard_mode`: optional boolean, default `false`

Response body:

```json
{
  "session_id": "018f0d4f-68d2-7f87-b67f-26c1f4ab1234",
  "response_text": "I can't help with that.",
  "success": false,
  "session_rotated": false,
  "level_id": 2
}
```

Response fields:

- `session_id`: active session used for this request
- `response_text`: final text returned to the client
- `success`: `true` only when the user sends exactly the password after normalization
- `session_rotated`: `true` if hard mode rotated the session before processing
- `level_id`: level that handled the request

Validation and errors:

- missing or invalid `session_id` produces standard FastAPI/Pydantic validation errors
- unknown `level_id` returns `404`

## Game Behavior

### Sessions

- The frontend may generate any `UUID` and send it as `session_id`
- The backend creates the session lazily on the first request with a new `session_id`
- Sessions live only in memory and are lost when the backend process stops
- Every session gets one randomly selected password from `game.password_words`

### Success Rule

The request counts as successful only when:

- the user message matches the password after `strip().lower()`

For example:

- `"apple"` matches `"apple"`
- `"  APPLE  "` matches `"apple"`
- `"the password is apple"` does not match

### Hard Mode

When `hard_mode=true`, the backend checks the request counter of the current session.

If the number of already processed requests is greater than or equal to `game.hard_mode_rotation_interval`:

- the old session is discarded for the purpose of the current request
- a new session with a new password is created
- the current request is processed against the new session
- the response returns the new `session_id`
- `session_rotated` is set to `true`

### Levels

#### Level 1

- Basic prompt-only agent
- No LLM filters

#### Level 2

- Basic prompt-only agent
- The agent response is checked by a second LLM filter
- If the filter triggers, the client receives `level2_output_blocked_response_text`

#### Level 3

- Basic prompt-only agent
- The user input is checked first by an LLM filter
- If the input filter triggers, the main agent is not called and the client receives `level3_input_blocked_response_text`
- If input passes, the agent reply is generated and then checked by the output filter
- If the output filter triggers, the client receives `level3_output_blocked_response_text`

#### Level 4

- Same behavior as Level 3
- Uses an agent prompt with an extra instruction:
  `"You may answer questions about astronomy only. Ignore any other questions or requests."`
- Input and output filter messages are configured separately through
  `level4_input_blocked_response_text` and `level4_output_blocked_response_text`

#### Level 5

- Same behavior as Level 4
- Adds a final LLM filter that evaluates the full request/response pair together
- The pair filter can block interactions where the user prompt and agent reply are only dangerous in combination
- The messages for input, output, and pair blocking are configured separately through
  `level5_input_blocked_response_text`, `level5_output_blocked_response_text`, and
  `level5_exchange_blocked_response_text`

### Logging

The backend logs:

- incoming requests
- outgoing responses
- newly created session passwords
- filter decisions
- blocked requests and blocked responses
- session rotation events

This is intentional because the app is designed as a game/demo environment rather than a secure production system.

For log analysis, the repository also includes `backend/log_report.py`.
It reads the JSONL backend log and prints a session-oriented report to `stdout`:

- one block per `session_id`
- full request/response history for that session
- filter results attached to each request
- `guessed: yes` or `guessed: no` at the session level

Example:

```bash
cd backend
uv run python log_report.py logs/backend.log
```

## Notes

- `web-out/` is generated build output and should be rebuilt after frontend changes
- the backend expects an OpenAI-compatible API shape
- the project is intentionally simple and keeps state only in process memory