Function embed_and_store_directory

Embeds and stores an entire directory in SurrealDB.

This is the complete end-to-end function that:

1. Chunks the directory structure
2. Generates embeddings using OpenAI's batch API
3. Stores everything in SurrealDB with graph relations

Example

// Embed and store MDX files only
const result = await embed_and_store_directory('/path/to/docs', {
  fileExtensions: ['.mdx'],
  costLimit: 0.50,
  model: 'text-embedding-3-small',
  dimensions: 1536
});
console.log(`Project: ${result.storageStats.projectId}`);
console.log(`Stored: ${result.storageStats.chunksCreated} chunks`);
console.log(`Embeddings: ${result.storageStats.embeddingsCreated} new, ${result.storageStats.embeddingsReused} reused`);
Copy

Example

// Embed and store TypeScript files
const result = await embed_and_store_directory('/path/to/src', {
  fileExtensions: ['.ts', '.tsx'],
  costLimit: 2.0,
  delimiter: /\n{3,}/g,
  deleteExisting: true
});
Copy