Analyze Hacker News Discussions With Python
Load Hacker News stories and direct comments in pandas, separate record types, and analyze engagement without double-counting discussions.
Enable bounded comment collection in the Hacker News Scraper, export JSON, and split the record types before analysis:
import pandas as pd
items = pd.read_json("dataset_items.json")
stories = items[items["type"].isin(["story", "job"])]
comments = items[items["type"] == "comment"]
comments_per_story = comments.groupby("parent_story_id").size()Use story ID as the unit for engagement summaries. Scores and API comment counts belong to stories; collected comment rows are only the bounded subset requested in the run. Strip HTML from comment text before language analysis and keep the original HTML for traceability.
When comparing topics, control for feed, rank, and collection time. A highly visible front-page item had a different opportunity to attract comments than a new-feed item. Report that context alongside sentiment or keyword results.
Frequently asked questions
Why separate stories and comments?
They have different units and fields; mixing them inflates item-level counts.
Are comment rows nested?
The Actor emits selected direct comments as rows with parent and parent-story identifiers.