Function embedDirectoryStructure

Generates embeddings for all chunks in a chunked directory structure

Uses OpenAI's batch embedding API with intelligent batching to respect:

• Per-chunk token limit (8192 tokens)
• Per-request token limit (300,000 tokens)
• Per-request item limit (2048 chunks)

Returns a new immutable structure with embeddings added to all chunks.

Example

const chunked = recursively_chunk_directory('/path/to/project');
const { embeddedDirectory, stats } = await embedDirectoryStructure(chunked);
console.log(`Generated ${stats.successfulEmbeddings} embeddings in ${stats.totalBatches} batches`);
Copy

Example

// With custom options
const result = await embedDirectoryStructure(chunked, {
  model: 'text-embedding-3-large',
  dimensions: 256,
  maxTokensPerBatch: 200000
});
Copy