TikTok Comment Scraper & Analyzer

Purpose

Scrape TikTok comments from video URLs using TikTok's internal comment API (no login required), clean and filter data by product category, infer user regions, and generate analysis reports. This is Steps 2-4 of the TikTok data pipeline.

When to Use

• User provides TikTok video URLs and wants to scrape comments
• User says "scrape comments from this TikTok video"
• User wants to clean/filter scraped TikTok data by category
• User needs sentiment + region analysis on TikTok comments
• After tiktok-search has collected video URLs, to scrape comments from them

API Details

The core API endpoints (discovered through testing) work WITHOUT authentication:

Main comment list

GET https://www.tiktok.com/api/comment/list/
  ?aid=1988
  &aweme_id={video_id}
  &count=50
  &cursor={cursor}

Comment replies (楼中楼)

GET https://www.tiktok.com/api/comment/list/reply/
  ?aid=1988
  &aweme_id={video_id}
  &comment_id={comment_id}
  &count=50
  &cursor={cursor}

Key facts:
- No login, cookie, or API key required for public videos
- Returns max ~50 comments/replies per page, paginated via cursor
- has_more: true/false indicates if more pages exist
- Without login, returns all publicly accessible comments (default max_per_video=10000, collects until has_more=false)
- user.region and user.language fields are almost always empty — region must be inferred from text
- IMPORTANT: Reply count field is reply_comment_total (NOT replyCount). The main comment list also includes inline reply_comment preview data for some comments.
- --no-replies flag disables reply fetching; --max-replies N controls max replies per comment (default 50)
- Data gap: TikTok's API total field is typically 3-10 higher than accessible comments (includes deleted/hidden). The scraper collects all publicly visible data; the gap is inherent to TikTok's no-login public API.

Architecture

The scraper processes comments in three phases to guarantee correct ordering and deduplication:

Phase 1: Main comment list API
  └─ Collects all top-level comments + inline reply_comment previews
  └─ replies_map: parent_cid → [inline_replies]
  └─ Preserves TikTok's original sort order

Phase 2: Reply API (for each comment with reply_comment_total > 0)
  └─ Calls /api/comment/list/reply/ for each parent
  └─ Deduplicates against inline replies by comment_id
  └─ Merges unique API replies into replies_map[parent_cid]

Phase 3: Interleave
  └─ Outputs: parent → replies (time-sorted) → next parent → replies → ...
  └─ Guarantees every reply appears immediately after its parent

Do NOT change this three-phase order without understanding that:
- Phase 1 + Phase 2 must complete data collection before Phase 3 orders it
- Inline reply dedup (by comment_id) happens in Phase 2
- Text-similarity dedup (by user+text) happens in scrape_from_urls
- If replies are collected but not interleaved, they'll all appear at the end — the exact bug we fixed

Scripts

1. scripts/scrape.py — Comment scraping

# From a single video URL (includes replies by default)
python scripts/scrape.py "https://www.tiktok.com/@user/video/123" ./output 200

# Without replies (top-level only)
python scripts/scrape.py "https://www.tiktok.com/@user/video/123" ./output 200 --no-replies

# With automatic insight report generation
python scripts/scrape.py "https://www.tiktok.com/@user/video/123" ./output 200 --analyze

# From a JSON file (output of tiktok-search)
python scripts/scrape.py search_results.json ./output 200

# From multiple comma-separated URLs
python scripts/scrape.py "url1,url2,url3" ./output

Outputs: tiktok_comments_{timestamp}.csv with fields: comment_text, username, user_id, nickname, user_bio, user_verified, user_followers, comment_time, likes, reply_count, comment_id, parent_comment_id, comment_type, source_url, source_author, source_video_id, sentiment, language

2. scripts/clean.py — Data cleaning

python scripts/clean.py tiktok_comments_xxx.csv ./output

Performs:
- Category filtering: removes comments from irrelevant categories (bath towels, hair products, cooking, etc.) based on keyword matching in comment text and source author
- Region inference: scans comment text for city/country keywords (Nairobi→Kenya, ug shillings→Uganda, etc.) — NOT from TikTok metadata (which is always empty)
- Sentiment re-labeling with expanded keyword library
- Purchase intent detection
- Per-region CSV exports

Outputs: tiktok_clean_{timestamp}.csv + per-region files

3. scripts/analyze.py — Context-aware insight analysis (NEW)

# Analyze an existing CSV and generate an HTML insight report
python scripts/analyze.py tiktok_comments_xxx.csv ./output

# With brand/product focus
python scripts/analyze.py tiktok_comments_xxx.csv ./output --brand "Clean Skin Club" --product "face towel"

This is a context-aware analysis engine that goes beyond per-comment sentiment. It performs:

1. Video context inference — Reverse-engineers the video topic from comment keywords
2. Conversation threading — Groups comments with their replies to understand dialogue context
3. Thread classification — Categorizes threads: diagnosis question, product question, complaint, alternative suggestion, environmental concern
4. Product signal extraction — Identifies price sensitivity, eco concerns, information gaps, brand visibility
5. Innovation opportunity generation — Produces actionable product/marketing suggestions

Outputs: insight_report_{timestamp}.html — a standalone HTML report with visualizations

Use --analyze flag on scrape.py to auto-generate the report after scraping.

4. Sentiment engine (external skill)

Sentiment analysis is handled by the standalone sentiment-analyzer skill (~/.workbuddy/skills/sentiment-analyzer/), which can also be used independently in any project.

from sentiment_engine import SentimentAnalyzer, analyze_sentiment
# See sentiment-analyzer SKILL.md for full documentation

Uses RoBERTa deep learning model with automatic VADER fallback. New CSV columns: sentiment_confidence, dominant_emotion (Ekman 6 emotions: joy/anger/sadness/fear/disgust/surprise).

Important Caveats

1. Region data is inferred, not verified. TikTok API does not return user location. Region labels come from text keyword matching only — a comment containing "Nairobi" is tagged Kenya. ~95% of comments will be "unknown".

2. Category filtering is keyword-based. The EXCLUDE_KEYWORDS list covers common irrelevant categories. For production use, review and customize the keyword lists in clean.py.

3. No login = limited but sufficient. Without authentication, all publicly visible comments are accessible. Default max_per_video=10000 collects complete comment threads.

4. Rate limiting. Add time.sleep(1-3) between API calls. Aggressive scraping triggers temporary blocks.

5. Reply count mismatch. reply_comment_total from the main API may report more replies than the reply API returns (~7-10% gap). These are deleted/hidden/moderated comments not accessible without login. The scraper collects all visible replies.

6. TikTok total field is inflated. The API's total includes deleted and hidden comments, typically 3-10 more than accessible. The scraper output count reflects actual retrieved data.

7. reply_comment_total field name is critical. The scraper uses reply_comment_total (NOT replyCount) to detect which comments have replies. The main comment list also embeds reply_comment preview data for some comments, which is extracted as inline replies.

8. Threaded replies are flat. The reply API returns all nested replies as a flat list. The thread_has_more flag on individual replies does not indicate additional accessible data.

9. Sentiment engine is separate. Emotion analysis uses the standalone sentiment-analyzer skill (RoBERTa + Ekman emotions). If the model fails to load, it falls back to keyword-based analysis automatically.

Example Workflow

# Step 1: Search TikTok (tiktok-search skill)
python ../tiktok-search/scripts/search.py "disposable face towel" 20 ./output

# Step 2: Scrape comments + auto-generate insight report
python scripts/scrape.py ./output/tiktok_search_*.json ./output 200 --analyze

# Or: Step 2a scrape, Step 2b analyze separately
python scripts/scrape.py "https://www.tiktok.com/@user/video/123" ./output 200
python scripts/analyze.py ./output/tiktok_comments_*.csv ./output --product "face towel"

# Step 3: Clean & analyze (legacy)
python scripts/clean.py ./output/tiktok_comments_*.csv ./output