Session Compression

AI Session Compression Techniques

Summary

Compress long AI conversations to fit context windows while preserving critical information.

Session compression enables production AI applications to manage multi-turn conversations efficiently by reducing token usage by 70-95% through summarization, embedding-based retrieval, and intelligent context management. Achieve 3-20x compression ratios with minimal performance degradation.

Key Benefits:

• Cost Reduction: 80-90% token cost savings through hierarchical memory
• Performance: 2x faster responses with compressed context
• Scalability: Handle conversations exceeding 1M tokens
• Quality: Preserve critical information with <2% accuracy loss

When to Use

Use session compression when:

• Multi-turn conversations approach context window limits (>50% capacity)
• Long-running chat sessions (customer support, tutoring, code assistants)
• Token costs become significant (high-volume applications)
• Response latency increases due to large context
• Managing conversation history across multiple sessions

Don't use when:

• Short conversations (<10 turns) fitting easily in context
• Every detail must be preserved verbatim (legal, compliance)
• Single-turn or stateless interactions
• Context window usage is <30%

Ideal scenarios:

• Chatbots with 50+ turn conversations
• AI code assistants tracking long development sessions
• Customer support with multi-session ticket history
• Educational tutors with student progress tracking
• Multi-day collaborative AI workflows

Quick Start

Basic Setup with LangChain

from langchain.memory import ConversationSummaryBufferMemory
from langchain_anthropic import ChatAnthropic
from anthropic import Anthropic

# Initialize Claude client
llm = ChatAnthropic(
    model="claude-3-5-sonnet-20241022",
    api_key="your-api-key"
)

# Setup memory with automatic summarization
memory = ConversationSummaryBufferMemory(
    llm=llm,
    max_token_limit=2000,  # Summarize when exceeding this
    return_messages=True
)

# Add conversation turns
memory.save_context(
    {"input": "What's session compression?"},
    {"output": "Session compression reduces conversation token usage..."}
)

# Retrieve compressed context
context = memory.load_memory_variables({})

Progressive Compression Pattern

from anthropic import Anthropic

client = Anthropic(api_key="your-api-key")

class ProgressiveCompressor:
    def __init__(self, thresholds=[0.70, 0.85, 0.95]):
        self.thresholds = thresholds
        self.messages = []
        self.max_tokens = 200000  # Claude context window

    def add_message(self, role: str, content: str):
        self.messages.append({"role": role, "content": content})

        # Check if compression needed
        current_usage = self._estimate_tokens()
        usage_ratio = current_usage / self.max_tokens

        if usage_ratio >= self.thresholds[0]:
            self._compress(level=self._get_compression_level(usage_ratio))

    def _estimate_tokens(self):
        return sum(len(m["content"]) // 4 for m in self.messages)

    def _get_compression_level(self, ratio):
        for i, threshold in enumerate(self.thresholds):
            if ratio < threshold:
                return i
        return len(self.thresholds)

    def _compress(self, level: int):
        """Apply compression based on severity level."""
        if level == 1:  # 70% threshold: Light compression
            self._remove_redundant_messages()
        elif level == 2:  # 85% threshold: Medium compression
            self._summarize_old_messages(keep_recent=10)
        else:  # 95% threshold: Aggressive compression
            self._summarize_old_messages(keep_recent=5)

    def _remove_redundant_messages(self):
        """Remove duplicate or low-value messages."""
        # Implementation: Use semantic deduplication
        pass

    def _summarize_old_messages(self, keep_recent: int):
        """Summarize older messages, keep recent ones verbatim."""
        if len(self.messages) <= keep_recent:
            return

        # Messages to summarize
        to_summarize = self.messages[:-keep_recent]
        recent = self.messages[-keep_recent:]

        # Generate summary
        conversation_text = "\n\n".join([
            f"{m['role'].upper()}: {m['content']}"
            for m in to_summarize
        ])

        response = client.messages.create(
            model="claude-3-5-haiku-20241022",
            max_tokens=500,
            messages=[{
                "role": "user",
                "content": f"Summarize this conversation:\n\n{conversation_text}"
            }]
        )

        # Replace old messages with summary
        summary = {
            "role": "system",
            "content": f"[Summary]\n{response.content[0].text}"
        }
        self.messages = [summary] + recent

# Usage
compressor = ProgressiveCompressor()

for i in range(100):
    compressor.add_message("user", f"Message {i}")
    compressor.add_message("assistant", f"Response {i}")

Using Anthropic Prompt Caching (90% Cost Reduction)

from anthropic import Anthropic

client = Anthropic(api_key="your-api-key")

# Build context with cache control
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Long conversation context here...",
                "cache_control": {"type": "ephemeral"}  # Cache this
            }
        ]
    },
    {
        "role": "assistant",
        "content": "Previous response..."
    },
    {
        "role": "user",
        "content": "New question"  # Not cached, changes frequently
    }
]

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=messages
)

# Cache hit reduces costs by 90% for cached content

Core Concepts

Context Windows and Token Limits

Context window: Maximum tokens an LLM can process in a single request (input + output).

Current limits (2025):

• Claude 3.5 Sonnet: 200K tokens (~150K words, ~600 pages)
• GPT-4 Turbo: 128K tokens (~96K words, ~384 pages)
• Gemini 1.5 Pro: 2M tokens (~1.5M words, ~6000 pages)

Token estimation:

• English: ~4 characters per token
• Code: ~3 characters per token
• Rule of thumb: 1 token ≈ 0.75 words

Why compression matters:

• Cost: Claude Sonnet costs $3/$15 per 1M input/output tokens
• Latency: Larger contexts increase processing time
• Quality: Excessive context can dilute attention on relevant information

Compression Ratios

Compression ratio = Original tokens / Compressed tokens

Industry benchmarks:

• Extractive summarization: 2-3x
• Abstractive summarization: 5-10x
• Hierarchical summarization: 20x+
• LLMLingua (prompt compression): 20x with 1.5% accuracy loss
• KVzip (KV cache compression): 3-4x with 2x speed improvement

Target ratios by use case:

• Customer support: 5-7x (preserve details)
• General chat: 8-12x (balance quality/efficiency)
• Code assistants: 3-5x (preserve technical accuracy)
• Long documents: 15-20x (extract key insights)

Progressive Compression Thresholds

Industry standard pattern:

Context Usage    Action                     Technique
─────────────────────────────────────────────────────────
0-70%           No compression             Store verbatim
70-85%          Light compression          Remove redundancy
85-95%          Medium compression         Summarize old messages
95-100%         Aggressive compression     Hierarchical + RAG

Implementation guidelines:

• 70% threshold: Remove duplicate/redundant messages, semantic deduplication
• 85% threshold: Summarize messages older than 20 turns, keep recent 10-15
• 95% threshold: Multi-level hierarchical summarization + vector store archival
• Emergency (100%): Drop least important messages, aggressive summarization

Compression Techniques