Loaders and Chunking

PyAIStack currently loads UTF-8 .txt files. Directory loading requires an explicit file_type; this prevents a directory scan from silently attempting unsupported formats.

Load one text file

from pyaistack.loaders import TextLoader

documents = TextLoader("knowledge/guide.txt").load()

Each loaded document includes the text and source path metadata.

Load a directory

from pyaistack.loaders import DirectoryLoader

documents = DirectoryLoader(
    "/path/to/knowledge/",
    file_type="text").load()

The loader recursively finds .txt files in deterministic path order. if no text file then it raise a clear unsupported-file-type error.

Generate metadata for a directory

DirectoryLoader accepts an optional metadata_factory. The generated fields are merged with the loader-provided source path and are copied to each chunk.

Type of metadata_factory:

1. FolderMetadataFactory
2. CSVMetadataFactory
3. LLMMetadataFactory

1. Use Folder Names

FolderMetadataFactory is deterministic and does not call a model. It maps parent folders to metadata fields in order:

knowledge/
└── finance/
    └── budgeting/
        └── monthly-budget.txt

from pathlib import Path

from pyaistack.loaders import DirectoryLoader, FolderMetadataFactory

knowledge_root = Path("/path/to/knowledge/")
loader = DirectoryLoader(
    knowledge_root,
    file_type="text",
    metadata_factory=FolderMetadataFactory(knowledge_root),
)

The file receives

{
  "category": "finance",
  "topic": "budgeting",
  "title": "monthly budget"
}

2. Use a CSV Manifest

CSVMetadataFactory is appropriate when metadata is managed in a spreadsheet or exported by a business system. The CSV must have source and metadata_json columns. source is relative to the supplied root, so the CSV remains portable across machines.

source,metadata_json
finance/budget.txt,"{""category"": [""finance""], ""tags"": [""budget"", ""plan""]}"
hr/leave-policy.txt,"{""category"": [""hr""], ""audience"": [""employees""]}"

from pyaistack.loaders import CSVMetadataFactory, DirectoryLoader

metadata_factory = CSVMetadataFactory(
    "metadata.csv",
    root="/path/to/knowledge/",
)
loader = DirectoryLoader(
    "/path/to/knowledge/",
    file_type="text",
    metadata_factory=metadata_factory,
)

The factory reads the manifest once. It raises MetadataFactoryError for invalid JSON, duplicate paths, or a text file without a matching CSV row.

3. Generate Metadata with an LLM

LLMMetadataFactory uses any injected ChatProvider; Ollama works with the existing OllamaChatProvider. It sends the filename and a bounded sample of the first eight sentences (up to 4,000 characters), then validates the returned JSON against the requested fields. Generated values are stored as normalized lowercase snake-case lists, such as {"category": ["sustainability"], "language": ["en"]}.

from pyaistack.loaders import DirectoryLoader, LLMMetadataFactory
from pyaistack.providers import OllamaChatProvider

chat_provider = OllamaChatProvider(model="gemma3:4b")

metadata_factory = LLMMetadataFactory(
    chat_provider,
    fields=("category", "topic", "document_type", "audience", "language", "tags"),
    max_sentences=8,
)
loader = DirectoryLoader(
    "/path/to/knowledge/",
    file_type="text",
    metadata_factory=metadata_factory,
)

Do not ask the LLM to generate trusted fields such as tenant_id, user_id, or access_level; set those from application-controlled data instead. A scalar filter matches one generated list value: metadata_filter={"category": "sustainability"}.

Ingest one file at a time

Use iter_load() to load, annotate, chunk, embed, and persist each file before moving to the next one:

from pyaistack import RAG
from pyaistack.chunking import TextChunker

rag = RAG(chunker=TextChunker())

for document in loader.iter_load():
    rag.add_documents([document])

This basic flow has no resumable-ingestion or duplicate-detection state. If a metadata factory fails, the error identifies the file and processing stops.

Chunk before indexing

from pyaistack import RAG
from pyaistack.chunking import TextChunker

rag = RAG(chunker=TextChunker(chunk_size=1_000, chunk_overlap=150))
rag.add_documents(documents)

Chunking is opt-in. RAG.add() keeps each supplied string as one document; RAG.add_documents() applies the configured chunker. The default separator order is paragraphs ("\n\n"), lines ("\n"), then sentences (". "). It does not split on spaces; a character cut is used only when no configured separator can split an oversized segment.

Each chunk retains the original metadata and adds one-based chunk_index, chunk_start, and chunk_end.

To add application metadata to one loaded file, create a new LoadedDocument with the loader metadata plus your fields:

from pyaistack.loaders import LoadedDocument, TextLoader

loaded = TextLoader("knowledge/personal-finance.txt").load()[0]
document = LoadedDocument(
    text=loaded.text,
    metadata={**loaded.metadata, "category": "finance"},
)