Skill Introduction

Overview

Chunking, embeddings, and RAG pipeline integration

Core value

Turns reusable Development know-how into an installable skill, helping users complete github, community, mcp work faster.

Target users

Developers, testers, and maintainers who handle Development tasks in Focus Code.
Teams that already trust workflows or content from kreuzberg-dev.
Users who want standardized prompts, steps, or conventions instead of repeating setup work.

Best practices

Read the skill content first to confirm required inputs, expected outputs, and dependencies.
Try it on a small task before relying on it for critical work.
Add project-specific constraints such as coding style, target platform, test expectations, and delivery format.
For external sources, verify the source link, version, and recent maintenance activity.

Best use cases

Tasks related to github, community, mcp that need a reusable execution flow.
Converting a community repo, team convention, or personal workflow into day-to-day assistance.
Starting from a proven skill instead of writing prompts or procedures from scratch.

Limits and boundaries

Results depend on the quality of the original skill content and may need human correction.
It does not replace code review, tests, security review, or professional judgment.
External tools, APIs, account permissions, and local dependencies still need separate setup.

Differentiation

Structured around Development, making it easier to discover and reuse than loose prompt snippets.
Marked as GitHub, which helps users judge trust and maintenance expectations.
Keeps the original source link available for repository, documentation, or discussion follow-up.
Tagged with github, community, mcp, so it can be filtered by concrete task intent.

Install and use

Install

Copy Install Command

focus install chunking-embeddings-261276

View source

Detail Preview

SKILL.md

Primary filemarkdown6 KB

description: "Chunking, embeddings, and RAG pipeline integration" name: chunking-embeddings priority: critical

Chunking & Embeddings

Text splitting strategies, embedding generation with FastEmbed, RAG pipeline integration

Chunking Architecture Overview

Location: crates/kreuzberg/src/chunking/, crates/kreuzberg/src/embeddings.rs

Extracted Text
    |
[1. Normalization] -> Clean whitespace, remove control chars
    |
[2. Chunk Strategy Selection] -> Fixed-size, semantic, syntax-aware, recursive
    |
[3. Overlap Management] -> Control context window overlap
    |
[4. Optional Embedding] -> Generate vectors with FastEmbed
    |
Output: Vec<Chunk> with text, vectors, metadata

Chunking Strategies

Location: crates/kreuzberg/src/chunking/mod.rs

Strategy	Pattern	Best For
Fixed-Size	Sliding window with configurable overlap	Uniform chunks for embedding models with fixed token limits
Semantic	Split by sentences, merge/split by similarity threshold	Smart context preservation for LLM consumption and semantic search
Syntax-Aware	Split by paragraph/section/heading/code-block structure	Preserving document structure (sections, code blocks) in RAG
Recursive (LangChain pattern)	Try separators in order: `\n\n`, `\n`, `,`	Best general-purpose chunking; auto-finds optimal split points

Key config fields per strategy (see struct definitions in chunking/mod.rs):

Fixed-Size: chunk_size, overlap, trim_whitespace
Semantic: target_chunk_size, min/max_chunk_size, semantic_threshold, use_sentence_boundaries
Syntax-Aware: chunk_by (Paragraph/Section/Heading/Sentence/CodeBlock), max_chunk_size, respect_code_blocks
Recursive: separators[], chunk_size, overlap

Chunking Configuration Presets

Location: crates/kreuzberg/src/chunking/mod.rs

Preset	Chunk Size	Overlap	Strategy	Use Case
Balanced	512 tokens	50	Semantic	RAG sweet spot
Compact	256 tokens	32	Fixed-Size	Dense vectors
Extended	1024 tokens	100	Recursive	Full context
Minimal	128 tokens	16	(default)	Lightweight embeddings

Usage: set config.chunking.preset = Some("balanced") in ExtractionConfig.

Embedding Generation with FastEmbed

Location: crates/kreuzberg/src/embeddings.rs

Model Selection

Model	Dimensions	Notes
`BAAI/bge-small-en-v1.5` (default)	384	Fast, excellent for RAG
`BAAI/bge-small-zh-v1.5`	384	Chinese optimized
`BAAI/bge-base-en-v1.5`	768	Better quality, slower
`jinaai/jina-embeddings-v2-base-en`	768	Long context (up to 8192 tokens)
`Custom(path)`	varies	Custom ONNX model path

Embedding Pattern

TextEmbeddingManager provides singleton-cached models per config. Pattern:

get_or_init_model() -- lazy-loads ONNX model (downloads if needed), caches in Arc<RwLock<HashMap>>
embed_chunks() -- collects chunk texts, calls model.embed(texts, batch_size), zips results back to ChunkWithEmbedding

Default config: batch_size=256, device=CPU, parallel_requests=4.

ONNX Runtime Requirement

Embeddings require ONNX Runtime. Feature-gated via:

[features]
embeddings = ["dep:fastembed", "dep:ort"]

Install: brew install onnxruntime (macOS) / apt install libonnxruntime libonnxruntime-dev (Linux). Verify: echo $ORT_DYLIB_PATH.

RAG Integration Pattern

The full extraction-to-RAG pipeline:

Extract: extract_file(path, config) -> ExtractionResult
Chunk: Apply preset strategy to result.content -> Vec<Chunk>
Embed: If embedding config present, TextEmbeddingManager::embed_chunks() -> Vec<ChunkWithEmbedding>
Output: RagDocument { file_path, metadata, chunks } ready for vector DB ingestion

See ChunkWithEmbedding struct in types.rs: contains text, embedding: Vec<f32>, dimensions, norm, metadata.

Critical Rules

Chunking is preprocessing - Always apply before embedding to ensure consistent vector sizes
Overlap prevents information loss - Set overlap to 15-20% of chunk size
Embedding models are stateful - Lazy load and cache to avoid repeated initialization
ONNX Runtime is required - Gracefully degrade if not available (skip embeddings)
Batch embedding for performance - Never embed single chunks; batch 50-1000 chunks
Normalize embeddings for search - Use L2 norm for cosine similarity
Cache embedding results - Don't re-embed identical text chunks
Model selection impacts quality - bge-small (384) for speed, bge-base (768) for quality

Related Skills

extraction-pipeline-patterns - Text extraction preceding chunking
api-server-mcp - Endpoint for chunking + embedding operations
ocr-backend-management - OCR text quality affects chunking success

chunking embeddings