Copilot History Ingest

Copilot History Ingest — Conversation Mining

You are extracting knowledge from the user's past GitHub Copilot CLI conversations and distilling it into the Obsidian wiki. Conversations are rich but messy — your job is to find the signal and compile it.

This skill can be invoked directly or via the wiki-history-ingest router (/wiki-history-ingest copilot).

Before You Start

Writing profile: Before drafting or rewriting natural-language Markdown, read and apply the Writing Profile Resolution section in llm-wiki/SKILL.md. Framework schema, provenance, safety, and operation-specific requirements take precedence. WRITING.md preferences apply only to newly drafted or rewritten natural-language Markdown; preserve source content and structured records.

1. Resolve config — follow the Config Resolution Protocol in llm-wiki/SKILL.md (inline @name override → walk up CWD for .env → global config → prompt setup). This gives OBSIDIAN_VAULT_PATH, COPILOT_HISTORY_PATH (defaults to ~/.copilot/session-state), and COPILOT_VSCODE_STORAGE_PATH (VS Code workspaceStorage; platform-specific — ask the user if absent)
2. Read .manifest.json at the vault root to check what's already been ingested
3. Read index.md at the vault root to know what the wiki already contains

Ingest Modes

Append Mode (default)

Check .manifest.json for each source file (events JSONL, transcript JSONL, checkpoint, session-store DB). Only process:

• Sessions not in the manifest (new sessions)
• Sessions whose updated_at is newer than their ingested_at in the manifest

This is usually what you want — the user ran a few new sessions and wants to capture the delta.

Full Mode

Process everything regardless of manifest. Use after a wiki-rebuild or if the user explicitly asks.

GitHub Copilot Data Layout

Copilot stores data in three locations. Scan all three.

Source 1: ~/.copilot/session-state/ (CLI sessions)

~/.copilot/session-state/
├── <session-uuid>/
│   ├── workspace.yaml           # Session metadata (id, cwd, summary_count, created_at, updated_at)
│   ├── vscode.metadata.json     # VS Code context (workspaceFolder, repositoryProperties, customTitle)
│   ├── events.jsonl             # Full event log — all turns, tool calls, reasoning
│   ├── session.db               # Per-session SQLite (todos/todo_deps only — skip for ingestion)
│   ├── index.md                 # Session summary written at session end
│   ├── checkpoints/             # Checkpoint JSON files (mid-session summaries)
│   │   └── <uuid>.json          # title, overview, history, work_done, technical_details,
│   │                            #   important_files, next_steps
│   ├── files/                   # Artifacts produced during session (plans, diagrams, etc.)
│   └── research/                # Research artifacts
└── ...

Source 2: ~/.copilot/session-store.db (Global SQLite)

The canonical cross-session database. This is the highest-value source: structured, queryable, and pre-summarised.

sessions       — id, cwd, repository, branch, summary, created_at, updated_at, host_type
turns          — session_id, turn_index, user_message, assistant_response, timestamp
checkpoints    — session_id, checkpoint_number, title, overview, history, work_done,
                 technical_details, important_files, next_steps, created_at
session_files  — session_id, file_path, tool_name, turn_index, first_seen_at
session_refs   — session_id, ref_type (commit/pr/issue), ref_value, turn_index, created_at
search_index   — FTS5 virtual table (content, session_id, source_type, source_id)

Source 3: VS Code Workspace Storage (<workspaceStorage>/<hash>/GitHub.copilot-chat/)

VS Code extension data, keyed by workspace hash. The path is platform-specific and must come from .env or user input.

<hash>/GitHub.copilot-chat/
├── transcripts/
│   └── <session-uuid>.jsonl     # Conversation transcripts (same JSONL format as events.jsonl)
├── memory-tool/
│   └── memories/
│       └── <base64-session-id>/ # Per-session saved artifacts (plan.md, etc.)
│           └── plan.md
└── codebase-external.sqlite     # Codebase index (skip — no conversation knowledge)

Key data sources ranked by value:

1. Checkpoints (session-store.db checkpoints table + per-session checkpoints/*.json) — Pre-distilled summaries with overview, work_done, technical_details, important_files, next_steps. Gold.
2. Session summaries (session-store.db sessions.summary + index.md) — One-paragraph synopsis per session.
3. Turns (session-store.db turns table + events.jsonl / transcript JSONL) — Full conversation. Rich but verbose.
4. Memory artifacts (memory-tool/memories/<id>/plan.md etc.) — Pre-written plans and structured notes the user saved explicitly. Worth importing verbatim (or lightly summarised).
5. File access patterns (session_files table + tool.execution_* events) — Which files the agent repeatedly touched — reveals high-value project files.
6. Session refs (session_refs table) — Commits, PRs, and issues linked to sessions.
7. `vscode.metadata.json` — Workspace folder path, branch, customTitle (user-set session label). Useful for grouping and naming.

Step 1: Survey and Compute Delta

Scan all three data locations and compare against .manifest.json:

# --- Source 1: per-session directories ---
# Find all session directories (each has workspace.yaml)
ls ~/.copilot/session-state/

# For each session, read workspace.yaml for id/cwd/updated_at
# and vscode.metadata.json for customTitle / repositoryProperties

# --- Source 2: global database ---
# Query session-store.db with sqlite3 (or Python sqlite3)
SELECT s.id, s.cwd, s.repository, s.branch, s.summary, s.updated_at,
       COUNT(DISTINCT t.turn_index) AS turn_count,
       COUNT(DISTINCT c.id)         AS checkpoint_count
FROM sessions s
LEFT JOIN turns t ON t.session_id = s.id
LEFT JOIN checkpoints c ON c.session_id = s.id
GROUP BY s.id
ORDER BY s.updated_at DESC;

# --- Source 3: VS Code workspace storage ---
# For each <hash> directory under workspaceStorage, check for GitHub.copilot-chat/
# Find transcript files
ls <workspaceStorage>/<hash>/GitHub.copilot-chat/transcripts/

Build a unified inventory — one entry per session UUID — and classify:

• New — not in manifest → needs ingesting
• Modified — in manifest but updated_at is newer → needs re-ingesting
• Unchanged — in manifest and not modified → skip in append mode

Report to the user: "Found X sessions in session-state, Y in session-store.db, Z VS Code transcript files. Checkpoints: A. Delta: B new, C modified."

Step 2: Ingest Checkpoints and Summaries First

Checkpoints are already distilled — process them before touching raw turns.

From session-store.db:

SELECT s.id, s.cwd, s.repository, s.branch, s.summary,
       c.checkpoint_number, c.title, c.overview, c.work_done,
       c.technical_details, c.important_files, c.next_steps,
       c.created_at
FROM checkpoints c
JOIN sessions s ON c.session_id = s.id
ORDER BY s.updated_at DESC, c.checkpoint_number ASC;

From per-session checkpoints/*.json:

Each checkpoint file has: title, overview, history, work_done, technical_details, important_files, next_steps.

Read index.md (if present) as a session-level summary — it's typically written at session end and is already concise.

What to extract:

• overview → high-level description of what the session accomplished
• work_done → concrete tasks completed (good for skills / project pages)
• technical_details → implementation specifics (good for concepts pages)
• important_files → high-value files in the project (good for project pages)
• next_steps → open threads (good for linking to ongoing project work)

Step 3: Parse Session Turns