Tech News Digest

Tech News Digest

Evidence-led tech digest with unified collection and one editorial selection shared across formats. Follow references/digest-prompt.md as the authoritative mode policy.

Editorial Contract

• Daily preserves original sections: 2–4 sentence overview; configured topics (normally LLM, AI Agent, Crypto, Frontier Tech; aim for 3–5 qualified items each); KOL viewpoints ≤3; releases ≤3 repositories; project discovery ≤3; blog picks ≤3; ≤30 unique items overall. News/blog explanations normally contain 2–3 concise sentences of change, evidence and implication. Retain empty headings with a coverage note, never filler. Optimize content within sections: a request to shorten GitHub releases must not become a whole-report redesign or one-message digest.
• Weekly ≤35 unique items: overview; configured topic sections in configured order (normally LLM, AI Agent, Crypto, Frontier Tech), with 2–4 substantial syntheses per topic as an advisory aim. Each synthesis uses 2–4 concise sentences joining week progress, evidence and implications, not concatenated headlines. Retain empty topic, key-release, project and viewpoint/reading headings with honest non-bullet coverage notes. Key releases ≤5 repositories (Chinese explanation ≤80 characters); projects ≤3; viewpoints/deep reading ≤4; optional next-week watch ≤3 pending releases, fixes or uncertainties. Do not force experiments; factual watch claims need citations, clearly proposed/not-performed experiments retain the narrow citation exemption.
• Caps are not quotas. Count evidence events inside syntheses and standalone watch items. Preserve topic sections in both modes; no global three-theme constraint. Body limits: daily ≤6500 characters, weekly ≤10000, excluding URL targets; never fill to the limit.
• Mandatory security coverage: follow the primary-alert/search and pre-delivery checks in references/digest-prompt.md. Preserve existing topics; use optional 🛡️ 安全警报 after the overview only for credible urgent incidents in either mode; weekly may also use an optional dedicated security/risk section for consequential nonurgent risks. Omit when empty and count/deduplicate within existing caps. Log checks and decisions; failed collection is not evidence of no incidents.
• Relevance, impact and evidence before internal scores. Exceptional claims need original technical evidence and independent assessment where possible, otherwise exclude/downgrade. No public scores, social metrics or operational statistics. Daily keeps substantive, nonduplicative KOL viewpoints. A material coverage gap may receive one concise caveat without raw counts; details stay in the final operational log.
• Deduplicate events and canonical URLs across the whole report. Daily repeats require incremental developments; weekly may synthesize daily coverage. Daily keeps blog picks; weekly viewpoints/deep reading may include original opinions, papers, docs, postmortems or essays.
• Project discovery is not verified trending; no lifetime-derived growth or repeated mature repos without meaningful changes. Preserve the prompt's consequential-release rules, exact versions and official links.
• Plain bullets with bold titles; concise inline Discord links [来源](<URL>). Section-aware chunks ≤1700 characters including numbering. Apply the requested language, including Simplified Chinese explanations when configured.
• Validate the canonical archive with python3 scripts/validate-digest.py --input FILE --mode daily or --mode weekly before delivery; fix all errors. Weekly production runs append one --topic-heading 'EXACT LOCALIZED HEADING' per configured topic in order; without these flags the gate cannot infer custom headings. Missing/failing validation blocks delivery. Manually verify unique-event counts inside syntheses, required non-topic headings/overview, security placement, evidence/coverage honesty and cross-format parity too.
• Discord, Markdown, email and PDF use identical selections/judgments. Operational stats/runtime attribution belong in the final log, not content.
• Legacy ITEMS_PER_SECTION, BLOG_PICKS_COUNT and EXTRA_SECTIONS are advisory parameters superseded by mode policy: ignore their counts/extra sections, especially duplicate weekly trends.

Quick Start

1. Configuration Setup: Default configs are in config/defaults/. Copy to workspace for customization: ``bash mkdir -p workspace/config cp config/defaults/sources.json workspace/config/tech-news-digest-sources.json cp config/defaults/topics.json workspace/config/tech-news-digest-topics.json ``

1. Environment Variables:

• TWITTERAPI_IO_KEY - twitterapi.io API key (optional, preferred)
• X_BEARER_TOKEN - Twitter/X official API bearer token (optional, fallback)
• TAVILY_API_KEY - Tavily Search API key, alternative to Brave (optional)
• WEB_SEARCH_BACKEND - Web search backend: auto|brave|tavily (optional, default: auto)
• BRAVE_API_KEYS - Brave Search API keys, comma-separated for rotation (optional)
• BRAVE_API_KEY - Single Brave key fallback (optional)
• GITHUB_TOKEN - GitHub personal access token (optional, improves rate limits)

1. Generate Digest: ``bash # Unified pipeline (recommended) — runs all 6 sources in parallel + merge python3 scripts/run-pipeline.py \ --defaults config/defaults \ --config workspace/config \ --hours 48 --freshness pd \ --archive-dir workspace/archive/tech-news-digest/ \ --output /tmp/td-merged.json --verbose --force ``

1. Select and Validate: Follow references/digest-prompt.md; validate the canonical Markdown before rendering the same selections to Discord, email and PDF.

Configuration Files

sources.json - Unified Data Sources

{
  "sources": [
    {
      "id": "openai-rss",
      "type": "rss",
      "name": "OpenAI Blog",
      "url": "https://openai.com/blog/rss.xml",
      "enabled": true,
      "priority": true,
      "topics": ["llm", "ai-agent"],
      "note": "Official OpenAI updates"
    },
    {
      "id": "sama-twitter",
      "type": "twitter", 
      "name": "Sam Altman",
      "handle": "sama",
      "enabled": true,
      "priority": true,
      "topics": ["llm", "frontier-tech"],
      "note": "OpenAI CEO"
    }
  ]
}

topics.json - Enhanced Topic Definitions

{
  "topics": [
    {
      "id": "llm",
      "emoji": "🧠",
      "label": "LLM / Large Models",
      "description": "Large Language Models, foundation models, breakthroughs",
      "search": {
        "queries": ["LLM latest news", "large language model breakthroughs"],
        "must_include": ["LLM", "large language model", "foundation model"],
        "exclude": ["tutorial", "beginner guide"]
      },
      "display": {
        "max_items": 8,
        "style": "detailed"
      }
    }
  ]
}

Scripts Pipeline

run-pipeline.py - Unified Pipeline (Recommended)

python3 scripts/run-pipeline.py \
  --defaults config/defaults [--config CONFIG_DIR] \
  --hours 48 --freshness pd \
  --archive-dir workspace/archive/tech-news-digest/ \
  --output /tmp/td-merged.json --verbose --force

• Features: Runs all 6 fetch steps in parallel, then merges + deduplicates + scores
• Output: Final merged JSON ready for report generation (~30s total)
• Metadata: Saves per-step timing and counts to *.meta.json
• GitHub Auth: Auto-generates GitHub App token if $GITHUB_TOKEN not set
• Fallback: If this fails, run individual scripts below

Individual Scripts (Fallback)

fetch-rss.py - RSS Feed Fetcher

python3 scripts/fetch-rss.py [--defaults DIR] [--config DIR] [--hours 48] [--output FILE] [--verbose]

• Parallel fetching (10 workers, max 2 concurrent per host), retry with jittered backoff, feedparser + regex fallback
• Timeout: 30s per feed, ETag/Last-Modified caching in $TECH_NEWS_DIGEST_STATE_DIR

fetch-twitter.py - Twitter/X KOL Monitor