DirectoryLoader class

A versatile document loader that loads Documents from a directory.

This loader can:

Load files from a specified directory
Apply glob patterns to filter files
Recursively search subdirectories
Exclude specific files or patterns
Use custom loaders for different file types
Sample files randomly or by a specific count
Build custom metadata for loaded documents

Default Supported File Types

By default, the DirectoryLoader supports the following file types:

.txt: Text files (loaded using TextLoader)
- Loads the entire file content as a single document
.json: JSON files (loaded using JsonLoader with root schema)
- Extracts all JSON objects or values at the root level
.csv and .tsv: CSV/TSV files (loaded using CsvLoader)
- Converts each row into a separate document

Example usage:

// Load all text and JSON files from a directory recursively
final loader = DirectoryLoader(
  '/path/to/documents',
  glob: '*.{txt,json}',
  recursive: true,
);
final documents = await loader.load();

// Load a random sample of 10 CSV files, excluding hidden files
final sampleLoader = DirectoryLoader(
  '/path/to/csvs',
  glob: '*.csv',
  loadHidden: false,
  sampleSize: 10,
  randomizeSample: true,
);
final sampleDocuments = await sampleLoader.load();

The loader supports customization through various parameters:

filePath: The directory path to load documents from
glob: Glob pattern to match files (defaults to all files)
recursive: Whether to search recursively in subdirectories
exclude: Patterns to exclude from loading
loaderMap: Map of file extensions to specific loaders
loadHidden: Whether to load hidden files
sampleSize: Maximum number of files to load
randomizeSample: Whether to randomize the file sample
sampleSeed: Seed for reproducible random sampling
metadataBuilder: Custom metadata building function

You can extend the default loader support by providing a custom loaderMap.

Constructors

DirectoryLoader.new(String filePath, {String glob = '*', bool recursive = true, List<String> exclude = const [], Map<String, BaseDocumentLoader Function(String)> loaderMap = const {}, bool loadHidden = false, int sampleSize = 0, bool randomizeSample = false, int? sampleSeed, Map<String, dynamic> metadataBuilder(File file, Map<String, dynamic> defaultMetadata)?}): A versatile document loader that loads Documents from a directory.
const

Properties

exclude → List<String>: Patterns to exclude from loading.
final
filePath → String: The path to the directory to load documents from.
final
glob → String: Glob pattern to match files. Use '*' to match all files.
final
hashCode → int: The hash code for this object.
no setterinherited
loaderMap → Map<String, BaseDocumentLoader Function(String)>: Map of file extensions to specific loaders.
final
loadHidden → bool: Whether to load hidden files (starting with '.').
final
metadataBuilder → Map<String, dynamic> Function(File file, Map<String, dynamic> defaultMetadata)?: Optional function to build custom metadata for each document.
final
randomizeSample → bool: Whether to randomize the sample of files.
final
recursive → bool: Whether to search recursively in subdirectories.
final
runtimeType → Type: A representation of the runtime type of the object.
no setterinherited
sampleSeed → int?: Seed for random sampling to ensure reproducibility.
final
sampleSize → int: Maximum number of files to load. Use 0 to load all files.
final

Methods

lazyLoad() → Stream<Document>: Loads documents lazily.
load() → Future<List<Document>>: Loads a list of documents.
inherited
noSuchMethod(Invocation invocation) → dynamic: Invoked when a nonexistent method or property is accessed.
inherited
toString() → String: A string representation of this object.
inherited

Operators

operator ==(Object other) → bool: The equality operator.
inherited

Static Properties

defaultLoaderMap ↔ Map<String, BaseDocumentLoader Function(String)>: Default loader map with common file type loaders.
getter/setter pair