# text-processing

> Browse 8 items tagged with text-processing.

## Items

- [RecursiveCharacterTextSplitter](https://vectordb.ever.works/hi/items/recursivecharactertextsplitter) — LangChain's hierarchical text chunking strategy achieving 85-90% accuracy by recursively splitting using progressively finer separators to preserve semantic boundaries.
- [NLTK](https://vectordb.ever.works/hi/items/nltk) — The Natural Language Toolkit (NLTK) is a leading Python platform for building programs to work with human language data. It provides easy-to-use interfaces to lexical resources like WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.
- [tiktoken](https://vectordb.ever.works/hi/items/tiktoken) — OpenAI's tokenizer library for encoding and decoding text into tokens, primarily used for calculating token counts with OpenAI's models and estimating chunk sizes for vector database document processing.
- [Agentic Chunking](https://vectordb.ever.works/hi/items/agentic-chunking-strategy) — An advanced RAG chunking strategy that uses LLMs to dynamically determine optimal document splitting based on semantic meaning and content structure. Agentic chunking analyzes document characteristics and adapts the chunking approach per document for superior retrieval accuracy.
- [Text Chunking Strategies for RAG](https://vectordb.ever.works/hi/items/text-chunking-strategies-for-rag) — Essential techniques for splitting documents into optimal-sized chunks for Retrieval-Augmented Generation, including fixed-size, recursive, semantic, and document-based chunking with overlap strategies to preserve context.
- [Semantic Chunking](https://vectordb.ever.works/hi/items/semantic-chunking) — Advanced text splitting technique using embeddings to divide documents based on semantic content instead of arbitrary positions, preserving cohesive ideas within chunks for improved RAG performance.
- [Recursive Character Text Splitter](https://vectordb.ever.works/hi/items/recursive-character-text-splitter) — Document chunking strategy that splits text at hierarchical boundaries like paragraphs, sentences, or headings. Industry-standard approach recommended as starting point with 400-512 tokens and 10-20% overlap for optimal RAG performance.
- [Chunk Overlap Strategy](https://vectordb.ever.works/hi/items/chunk-overlap-strategy) — Text chunking technique using 10-20% overlap between consecutive chunks to preserve context continuity and prevent information loss at chunk boundaries for improved retrieval.

---

_Canonical page: https://vectordb.ever.works/hi/tags/text-processing_