Observability and evaluation

Observability is optional and provider-neutral. Without an observer, PyAIStack does not create events, write files, or add logging output.

Attach observers

from pyaistack import RAG
from pyaistack.observability import JsonlObserver, LoggingObserver

with JsonlObserver("rag_events.jsonl") as file_observer:
    rag = RAG(observers=[LoggingObserver(), file_observer])
    rag.add("AWS Lambda runs code without managing servers.")
    print(rag.ask("Which service runs code?").text)

LoggingObserver sends one sanitized JSON record through Python logging. JsonlObserver writes one sanitized event per line to the file path selected by your application. InMemoryObserver retains events only in process memory for tests or local inspection.

Event privacy

Events include a shared run_id, operation name, status, duration, model name, and counts. They exclude prompts, generated answers, document text, credentials, and metadata values by default.

Event format and operations

Every event has this JSON-compatible shape:

{
  "schema_version": 1,
  "timestamp": "2026-09-19T12:30:45.123Z",
  "run_id": "one-id-for-the-rag-operation",
  "operation": "retrieval",
  "status": "success",
  "duration_ms": 12.34,
  "details": {"mode": "vector", "result_count": 3},
  "error_type": null
}

schema_version identifies the event contract and timestamp records the event creation time in UTC. status is success, failure, or skipped. Failed events include the exception class name in error_type, not the exception message. Events emitted while one ask() call is running share the same run_id.

Operation	When it is emitted	Safe details
`embedding`	After document or query embedding.	Input count and embedding model name.
`indexing`	After documents and vectors are stored, or when no documents are supplied.	Document count, safe backend name when provided, and explicitly allowlisted metadata only.
`add`	After `rag.add()` completes or fails.	Document count on success.
`retrieval`	After vector or hybrid search.	Retrieval mode, result count, metadata-filter key names, and safe backend name when provided.
`reranking`	After a configured reranker scores candidates.	Candidate and selected-result counts.
`search`	After a direct `rag.search()` call completes or fails.	Result count and safe backend name when provided.
`context`	After bounded context construction during `rag.ask()`.	Context character and excerpt counts.
`generation`	After the chat provider generates an answer.	Chat model name.
`answer`	After `rag.ask()` completes, fails, or is skipped for insufficient context.	Source/citation counts or the skip reason.
`clear`	After `rag.clear()` removes the configured index or fails.	Removed document count and safe backend name when provided.

To intentionally include selected safe metadata fields during indexing, use an allowlist. Only scalar values or items in metadata lists are recorded, with bounded value counts and lengths.

from pyaistack import RAG, RAGConfig
from pyaistack.observability import LoggingObserver

rag = RAG(
    config=RAGConfig(observability_metadata_keys=("category", "language")),
    observers=[LoggingObserver()],
)

Do not allowlist source paths, user identifiers, secrets, or other sensitive application metadata.

Observer failures are isolated and never change an answer or replace the original provider/vector-store exception.

Retrieval evaluation reports

evaluate_retrieval_report() measures retrieval only. It reports Recall@k, MRR, returned document IDs, first relevant rank, and elapsed time per labeled query; it does not claim answer quality.

from pyaistack.evaluation import RetrievalCase, evaluate_retrieval_report

report = evaluate_retrieval_report(
    rag,
    [RetrievalCase("What does FIN-042 require?", frozenset({"document-id"}))],
    top_k=3,
)
saved_report = report.to_dict()

Runnable example

Run the observability example from the repository root:

python examples/observability/rag_events.py

See ready examples for the complete example sequence.