Testing

Two repositories

AutoMem uses different test frameworks in each repository: pytest for the automem server (Python) and Vitest for the mcp-automem client (TypeScript).

AutoMem Server — Testing (pytest)

Purpose and Scope

AutoMem’s test suite validates core functionality through unit tests with emphasis on the consolidation engine. The test framework uses pytest with mock objects to isolate graph and vector store interactions, enabling fast execution without external dependencies.

Currently, the test suite focuses on consolidation logic. API endpoints and background workers are validated through manual testing and production monitoring.

Test Suite Organization

tests/
├── conftest.py
├── test_consolidation_engine.py
├── test_api_endpoints.py
├── test_app.py
├── test_enrichment.py
├── test_embedding_providers.py
├── test_integration.py
├── test_content_size.py
├── test_vector_size_safety.py
├── test_recall_entity_extraction.py
├── support/
├── contracts/
└── benchmarks/

Pytest Markers

Tests are tagged with markers to control which tier runs:

Marker	Command	Requires	Purpose
unit	pytest -m unit tests/	No external services	Fast, isolated logic tests
integration	pytest -rs -m integration tests/	Docker stack running	Full API + database flow
live	AUTOMEM_ALLOW_LIVE=1 pytest -m live tests/	Railway deployment	Production smoke tests

make test runs unit tests only. make test-integration starts Docker Compose and runs integration tests.

Coverage by Component

Component	Test File	Coverage Status
Memory Consolidation	test_consolidation_engine.py	Comprehensive
Decay Calculations	test_consolidation_engine.py	Covered
Creative Associations	test_consolidation_engine.py	Covered
Clustering Logic	test_consolidation_engine.py	Covered
Controlled Forgetting	test_consolidation_engine.py	Covered
Flask API Endpoints	test_api_endpoints.py, test_app.py	Covered
Enrichment Pipeline	test_enrichment.py	Covered
Embedding Providers	test_embedding_providers.py	Covered
Integration Flow	test_integration.py	Integration marker
Content Size Limits	test_content_size.py	Covered
Vector Size Safety	test_vector_size_safety.py	Covered
Recall Entity Extraction	test_recall_entity_extraction.py	Covered
MCP Bridge	—	Manual testing only

Running Tests

Prerequisites — install development dependencies:

pip install -r requirements-dev.txt

Basic execution:

pytest -m unit tests/                            # Run unit tests (no external services)
pytest -m unit tests/ -v                         # Verbose output
pytest tests/test_consolidation_engine.py        # Specific file
pytest -m unit tests/ -k "test_relevance_score"  # Specific function
pytest -rs -m integration tests/                 # Integration tests (requires Docker stack)

Test Architecture

graph TB
    subgraph "Test Layer"
        TestFunctions["Test Functions\ntest_calculate_relevance_score_*\ntest_discover_creative_*\ntest_cluster_similar_*\ntest_apply_controlled_*"]
        Fixtures["pytest Fixtures\nfreeze_time\nmonkeypatch"]
    end

    subgraph "Mock Objects"
        FakeGraph["FakeGraph\nfake FalkorDB graph"]
        FakeVectorStore["FakeVectorStore\nfake vector store"]
        FakeResult["FakeResult\nmimics FalkorDB result"]
    end

    subgraph "Code Under Test"
        MemoryConsolidator["MemoryConsolidator\nconsolidation.py:121-1117"]
        calculate_relevance["calculate_relevance_score()\nconsolidation.py:227-283"]
        discover_creative["discover_creative_associations()\nconsolidation.py:334-456"]
        cluster_similar["cluster_similar_memories()\nconsolidation.py:457-618"]
        apply_forgetting["apply_controlled_forgetting()\nconsolidation.py:619-819"]
    end

    TestFunctions --> Fixtures
    TestFunctions --> FakeGraph
    TestFunctions --> FakeVectorStore

    FakeGraph --> MemoryConsolidator
    FakeVectorStore --> MemoryConsolidator

    MemoryConsolidator --> calculate_relevance
    MemoryConsolidator --> discover_creative
    MemoryConsolidator --> cluster_similar
    MemoryConsolidator --> apply_forgetting

    FakeGraph --> FakeResult

Mock Object Implementation

The test suite uses in-memory mock objects that implement the same protocols as production dependencies, enabling fast tests without FalkorDB or Qdrant instances.

FakeGraph Class

FakeGraph simulates FalkorDB query behavior (tests/support/fake_graph.py:42-847):

• Query pattern matching: Uses string matching to identify query type (e.g., "COUNT(DISTINCT r)" for relationship counts)
• Deterministic responses: Returns pre-configured data from state attributes
• Side effect tracking: Records deletions, archives, and score updates
• Query history: Records every call in graph.queries as (query, params) tuples for verification

FakeVectorStore Class

FakeVectorStore records each delete() call as a (collection_name, points_selector) tuple in vector_store.deletions (tests/test_consolidation_engine.py:19-24).

FakeResult Class

FakeResult mimics FalkorDB query result structure with a result_set attribute (tests/support/fake_graph.py:9-11).

Test Fixtures

freeze_time Fixture

The freeze_time fixture uses monkeypatch to replace datetime.now() with a fixed timestamp (2024-01-01 00:00:00 UTC), ensuring deterministic decay calculations (tests/test_consolidation_engine.py:27-39):

• Auto-use: Applies to all tests automatically
• Module patching: Patches consolidation_module.datetime, not global datetime
• Timezone support: Handles both naive and aware datetime calls
• Cleanup: Automatically restores original datetime after each test

Helper Functions

iso_days_ago(days: int) generates ISO timestamp strings relative to the frozen time:

def iso_days_ago(days: int) -> str:
    return (FROZEN_TIME - timedelta(days=days)).isoformat()

Writing New Tests

Test Function Structure

Follow this pattern for consolidation engine tests:

1. Setup: Create FakeGraph and configure mock data
2. Execution: Instantiate MemoryConsolidator and call method
3. Verification: Assert return values and check mock state

calculate_relevance_score() takes a single memory dict (plus an optional current_time), not individual keyword arguments — it reads id, timestamp, last_accessed, importance, and confidence off that dict:

def test_calculate_relevance_score_recent_memory():
    # 1. Setup
    graph = FakeGraph()
    graph.relationship_counts = {"mem_001": 3}

    # 2. Execution
    consolidator = MemoryConsolidator(graph, vector_store=FakeVectorStore())
    score = consolidator.calculate_relevance_score(
        {
            "id": "mem_001",
            "timestamp": iso_days_ago(2),
            "last_accessed": iso_days_ago(1),
            "importance": 0.8,
            "confidence": 0.7,
        }
    )

    # 3. Verification
    assert 0 < score <= 1.0  # Recent, high-importance memory

Mock Data Configuration

Relationship counts — control _get_relationship_count() return values:

graph.relationship_counts = {
    "mem_001": 5,  # 5 relationships
    "mem_002": 0   # No relationships
}

All of these row attributes hold plain lists of column values (List[List[Any]]) in the same order the corresponding Cypher query returns them — not dicts.

Sample memories for creative associations — configure sample_rows as [id, content, type, embedding, timestamp], matching the query in discover_creative_associations():

graph.sample_rows = [
    ["mem_001", "Chose PostgreSQL for reliability", "Decision", [1.0, 0.0, 0.0], iso_days_ago(3)],
    ["mem_002", "Prefer typed languages", "Preference", [0.0, 1.0, 0.0], iso_days_ago(4)],
]

Cluster data — configure cluster_rows as [id, content, embedding, type] for clustering tests:

graph.cluster_rows = [
    ["mem_001", "content_a", [0.1, 0.2, 0.3], "Insight"],
    ["mem_002", "content_b", [0.15, 0.22, 0.31], "Insight"],
]

Decay and forgetting data — configure decay_rows as [id, content, last_accessed, importance, timestamp, relevance_score] (or forgetting_rows for the forgetting query):

graph.decay_rows = [
    ["mem_001", "Early note", iso_days_ago(30), 0.8, iso_days_ago(60), 0.75],
    ["mem_002", "Recent insight", iso_days_ago(1), 0.7, iso_days_ago(1), 0.9],
]

Testing Dry Run vs Execution

Many consolidation methods support a dry_run parameter. Test both modes:

def test_apply_forgetting_dry_run():
    graph = FakeGraph()
    # ... configure graph ...
    consolidator = MemoryConsolidator(graph, vector_store=FakeVectorStore())
    result = consolidator.apply_controlled_forgetting(dry_run=True)
    assert graph.deleted == []  # Nothing deleted in dry run

def test_apply_forgetting_execution():
    graph = FakeGraph()
    # ... configure graph with low-relevance memories ...
    vector_store = FakeVectorStore()
    consolidator = MemoryConsolidator(graph, vector_store=vector_store)
    result = consolidator.apply_controlled_forgetting(dry_run=False)
    assert len(graph.deleted) > 0        # Nodes deleted
    assert len(vector_store.deletions) > 0  # Vector deletions issued

Verifying Mock Interactions

Query history — graph.queries holds (query, params) tuples, so match on the first element:

assert any("MATCH (m:Memory)" in query for query, _params in graph.queries)

State changes — graph.deleted is a list of memory IDs, while graph.archived and graph.updated_scores are lists of (memory_id, score) tuples:

assert "mem_001" in graph.deleted
assert "mem_002" in [memory_id for memory_id, _score in graph.archived]
assert dict(graph.updated_scores)["mem_003"] < 0.2

Vector store interactions — vector_store.deletions holds (collection_name, points_selector) tuples:

collection, selector = vector_store.deletions[0]
assert collection == "memories"
assert selector.get("point_ids") == ["mem_001"]

Test Coverage Report

Consolidation Engine Coverage

Module	Function	Test Coverage
MemoryConsolidator	calculate_relevance_score()	Complete
MemoryConsolidator	discover_creative_associations()	Complete
MemoryConsolidator	cluster_similar_memories()	Complete
MemoryConsolidator	apply_controlled_forgetting()	Complete (dry run + execution)
MemoryConsolidator	_apply_decay()	Complete
MemoryConsolidator	consolidate()	Partial (individual steps tested)
ConsolidationScheduler	—	Not tested

Uncovered Components

MCP Bridge — deploy to Railway and test with Claude Desktop.

Performance Testing Procedures

Based on the current batching and logging implementation:

Embedding batching verification — rapidly create 25 memories and check logs:

for i in {1..25}; do
  curl -X POST http://localhost:8001/memory \
    -H "Authorization: Bearer $TOKEN" \
    -d "{\"content\": \"Memory $i\", \"tags\": [\"test\"]}"
done

Expected log: a batch embedding log line from the embedding provider

Consolidation performance — monitor logs during decay runs. After optimization, should see ~80% reduction in relationship query counts.

Health endpoint — verify enrichment stats:

curl http://localhost:8001/health | jq '.enrichment'
# Expected keys: status, queue_depth, pending, inflight, processed, failed

Structured logging — check logs for structured events after memory operations:

• recall_complete events with latency metrics
• memory_stored events with queue status

Future Testing Priorities

Based on current coverage gaps:

1. API endpoint tests — Test Flask routes with mock database
2. Background worker tests — Test enrichment and embedding workers
3. Integration tests — Test full memory storage → enrichment → recall flow
4. ConsolidationScheduler tests — Test scheduling logic and intervals
5. MCP bridge tests — Test tool call translation
6. Load testing — Verify performance under high memory throughput

---

mcp-automem Client — Testing (Vitest)

Overview

The mcp-automem package uses Vitest as its test runner, providing fast ES module support, built-in coverage reporting, watch mode for TDD, and parallel test execution.

Test Organization

graph TB
    TESTS["Test Suite"]

    TESTS --> UNIT["Unit Tests\n(vitest.config.ts)"]
    TESTS --> INTEGRATION["Integration Tests\n(vitest.integration.config.ts)"]

    UNIT --> CLIENT_TEST["AutoMemClient\nHTTP mocking"]
    UNIT --> CLI_TEST["CLI Handlers\nFilesystem mocking"]
    UNIT --> TOOLS_TEST["MCP Tools\nRequest/response validation"]

    INTEGRATION --> ENDTOEND["End-to-End\nReal AutoMem service"]
    INTEGRATION --> PLATFORM["Platform Integration\nInstaller verification"]

• Unit tests: src/**/*.test.ts or tests/unit/ — mock node-fetch for HTTP requests, mock fs for filesystem operations, use in-memory fixtures for AutoMem responses
• Integration tests: tests/integration/ — require a running AutoMem service

Test Commands

Script	Command	Purpose
test	vitest run	Run all unit tests once
test:watch	vitest	Watch mode for TDD
test:coverage	vitest run --coverage	Generate coverage reports
test:integration	vitest run --config vitest.integration.config.ts	Integration test suite
test:all	Both unit and integration	Full test suite

Coverage Requirements

Coverage reporting is integrated into the CI pipeline via @vitest/coverage-v8, generating reports for line, branch, function, and statement coverage. Coverage failures don’t block builds (continue-on-error: true) — coverage is informational rather than a hard gate during active development.

Mock Strategy

HTTP mocking — mock node-fetch to simulate AutoMem backend responses without a running service:

import { vi } from 'vitest';

vi.mock('node-fetch', () => ({
  default: vi.fn().mockResolvedValue({
    ok: true,
    json: () => Promise.resolve({ memory_id: 'test-123', message: 'Stored' })
  })
}));

Filesystem mocking — mock fs for CLI installer tests:

vi.mock('fs', () => ({
  writeFileSync: vi.fn(),
  readFileSync: vi.fn().mockReturnValue('template content'),
  existsSync: vi.fn().mockReturnValue(false)
}));

In-memory AutoMem fixtures — use realistic response shapes:

const mockRecallResponse = {
  results: [
    {
      id: 'mem-001',
      content: 'Chose PostgreSQL for reliability',
      score: 0.92,
      tags: ['decision', 'database'],
      importance: 0.8,
      metadata: { type: 'Decision' }
    }
  ],
  count: 1
};

Integration Test Requirements

Integration tests require a live AutoMem service. Set the following environment variables before running:

AUTOMEM_API_URL=http://localhost:8001
AUTOMEM_API_KEY=your-test-token

Integration tests create real data

Integration tests store and recall actual memories in the AutoMem service. Use a dedicated test instance or clean up test memories after runs using tag-based deletion.

CI/CD Integration

The CI workflow (.github/workflows/ci.yml) runs on every PR and push to main:

1. Link validation on documentation files
2. npm run lint — ESLint (fails build on errors)
3. npm run build — TypeScript compilation (fails build on errors)
4. npm test — Unit tests (fails build on failures)
5. npm run test:coverage — Coverage report (non-blocking)

Unit tests run in CI without an external AutoMem service. Integration tests are run manually or in dedicated integration environments.