How to Scrape Reddit Posts, Subreddits and Comments

Web scraping is a method to automatically collect data from websites, letting users gather large amounts of information to analyze or use in different ways. It involves using tools or programs to pull data like text, photos, and links from publicly accessible websites. In this article, we’ll explore how to scrape data from Reddit.

Reddit is a valuable platform because it has millions of comments on a wide range of topics, making it a rich source of data. As a community-driven site, Reddit is filled with discussions on trending news, unique interests, and niche hobbies, giving deep insights. Each conversation is organized in "subreddits," which are communities focused on specific topics. Users can search, join, and follow discussions in these subreddits based on their interests, covering everything from niche hobbies to popular subjects like politics and technology. Reddit’s diversity of information makes it ideal for web scraping to gain insights.

Understanding Reddit's Structure

A subreddit is a community within Reddit where users discuss specific topics. Each subreddit operates with its own set of rules and guidelines, managed by moderators. Below are the main elements that define the structure of a subreddit:

Posts: Users submit posts in the form of links, images, videos, or text to spark discussions. These posts act as conversation starters and appear on the subreddit’s front page based on popularity and time of submission.
Comments: Users engage with posts by leaving comments, and creating a threaded discussion. Comments can be nested, allowing for detailed conversations where users reply to one another.
Flairs: These are labels applied to posts or user profiles, providing context. Post flairs can indicate the type of content (e.g., "Question," "Discussion") or relevant tags (e.g., "Advice"). User flairs often show personal identifiers like occupation or a lighthearted nickname within the community.

Subreddit dynamics are heavily influenced by voting mechanisms, moderation, and sidebar features:

Upvotes and Downvotes: These voting options allow users to rate posts and comments. Upvotes signal approval or interest, while downvotes indicate disapproval or irrelevance. The post or comment's total score (upvotes minus downvotes) determines its visibility on the subreddit.
Karma: Reddit’s point system, where users earn positive karma from upvotes on their posts or comments, and lose karma from downvotes. Karma reflects a user's contribution level to the Reddit community.
Sidebar Information: Subreddits usually feature a sidebar with rules, guidelines, important links, or related subreddits. It helps users navigate the community and understand the expectations.
Moderators: Volunteer users who ensure that subreddit rules are followed. They manage content, approve or remove posts, and sometimes initiate community discussions or events.
Awards: Users can give awards, like “Gold” or “Platinum,” to posts or comments they find particularly valuable. Awards are often purchased to highlight exceptional contributions.

Reddit offers a wealth of data for analysis and insight generation, especially when scraped from various subreddits. The available data types are diverse, ranging from post metadata to user engagement metrics. Post titles provide a concise summary of the discussion topics, while author information reveals the identity of the post creator, including their username, karma score, and account age. Upvotes and downvotes reflect the popularity and reception of posts, while comment threads allow for deeper insight into community interactions and discussions. Additionally, data such as post-creation time, awards given, and subreddit-specific metadata can be extracted, offering a comprehensive view of Reddit's ecosystem.

Here is a list of the available data types.

Post Titles: Summary of topics or questions raised in the post.
Author Information: Username, account age, karma, and flair details.
Post Content: Full text of the post, including links and media.
Engagement Metrics: Number of comments, views, and shares.
Comment Threads: Full conversations, replies, and nested discussions.

Common Methods of Scraping Reddit

There are two common methods when trying to scrape websites Reddit. They are Reddit API and web scraping. Both have their advantages in gathering data. Reddit API is a service provided by Reddit, that allows developers to access and interact with Reddit’s data and features.

Reddit API provides a structured way to get information about posts, comments, user profiles, and subreddit metadata directly from Reddit’s Server. Here are a few key aspects of using Reddit API.

Access to Reddit Data: Through the API, you can retrieve data on posts, comments, subreddit details, and user profiles. For example, you can fetch the top posts from a specific subreddit, search for posts with particular keywords, or monitor a subreddit for new posts.
Structured Data: The API returns data in JSON format, making it easier to parse and manipulate programmatically compared to raw HTML. JSON’s structured format allows you to directly access fields like the post title, timestamp, author, upvotes, and comment count without needing to sift through complex HTML.
Rate Limits: Reddit’s API has rate limits to prevent excessive usage. These limits cap the number of requests per user or app in a certain time frame. Rate limits are stricter for free access but can be adjusted for enterprise applications (at a cost).
OAuth Authentication: To access Reddit’s API, developers must register an app on Reddit and use OAuth 2.0 authentication. OAuth tokens ensure that access to Reddit’s data is controlled and that API usage is tracked for each app and user.
Avoiding HTML Changes: Unlike traditional web scraping that involves parsing the HTML of web pages, which may change over time, APIs generally maintain a consistent structure. This stability can be advantageous because it requires less maintenance over time compared to scraping HTML, which might break whenever the page structure changes.
Restrictions on Data Types: The Reddit API has limitations on historical data. It doesn’t give full access to all user interactions or certain subreddit information, particularly older data, and might require higher access levels for some sensitive data.

While web scraping extracts raw HTML and parses it to retrieve data (often using tools like BeautifulSoup in Python or Nokogiri in Ruby), the Reddit API offers a structured and legal way to access Reddit’s data in a more reliable and developer-friendly manner. But depends on the way we want to gather information from Reddit, web scraping is much better in terms of flexibility.

Web scraping using Nokogiri is a method of extracting data from websites by parsing their HTML or XML content in Ruby. Nokogiri is a popular library in Ruby for web scraping and parsing HTML/XML, offering powerful tools to navigate and retrieve data from webpages. Key aspects of web scraping using Nokogiri are as follows.

Fetching the HTML Content: First, you need to retrieve the HTML content of the webpage you want to scrape. This is usually done by sending an HTTP request to the website using libraries like Net::HTTP or HTTParty. Once the HTML is fetched, you can pass it to Nokogiri for parsing.
Parsing the HTML with Nokogiri: Nokogiri can parse HTML and XML into a structured format called a “document object model” (DOM). This structure allows you to access elements in the page by their tags, attributes, classes, IDs, and more.
Extracting Data with CSS Selectors and XPath: Nokogiri lets you use CSS selectors or XPath expressions to navigate and select elements in the HTML. For example, if you wanted to get all post titles on a Reddit page, you might use a CSS selector targeting the title’s HTML class or ID.
Cleaning and Structuring the Data: Once data is extracted, you can clean and structure it as needed. For instance, you might collect post titles, timestamps, and URLs, then store them in arrays, hashes, or export them to CSV or JSON.

Nokogiri is a fast and efficient tool for scraping large documents, but it’s limited by terms of service on certain sites. Always check a website's terms to ensure compliance.

Choosing the Right Scraping Tools For Reddit

When it comes to web scraping, Python and Ruby are among the top choices due to their simplicity, versatility, and extensive libraries. Python, especially, is preferred for its rich ecosystem of scraping libraries and ease of use, making it highly accessible for beginners and advanced users. While less commonly used, Ruby provides a clean and elegant syntax that can simplify code readability and maintainability.

Please make sure to set these tools beforehand to build effective web scraping tools in Ruby. Of course, start by installing Ruby on your machine, either by downloading the latest version from the official Ruby website or using a version manager like RVM or rbenv for easier management of multiple Ruby versions and gems. After installation, verify Ruby by running ruby -v in your terminal.

Next, equip yourself with essential libraries for scraping. Nokogiri is the primary library for parsing HTML and XML, allowing you to search and manipulate documents effectively. Typhoeus simplifies HTTP requests and API interactions, supporting efficient web requests and parallel processing.

Lastly, use libraries such as URI for managing and manipulating URLs, and DateTime for handling timestamps, which are crucial for logging and calculating time differences in your scraping scripts. By combining these tools, you will create a well-organized environment tailored to your web scraping projects.

For API scraping, libraries like PRAW (Python Reddit API Wrapper) provide structured access to Reddit’s data, which is particularly useful for accessing content without relying on traditional web scraping. But in this case, PRAW isn’t used because we don’t utilize API from Reddit, but we still utilizing request on Reddit API.

Web Scraping Techniques for Reddit

On managing JavaScript-rend Selenium to handle JavaScript-rendered pages, where static HTML falls short. Reddit’s interface often requires interaction, such as scrolling through infinite scrolling subreddits to load more content.

But in this case, we don’t use Selenium because we are facing Static Content, so in the form of Ruby Programming language, we are going to use Nokogiri.

Step-by-Step Guide to Scrape Subreddits

The first part of this script establishes a connection to Reddit by sending HTTP requests for each subreddit in the list. The headers mimic a typical browser to avoid blocks, which helps bypass certain site protections by making the requests look like they’re coming from a regular user.

require 'typhoeus'      ##### For making HTTP requests
require 'nokogiri'      ##### For parsing HTML

##### List of subreddit URLs to scrape
subreddit_urls = [
   "https://www.reddit.com/r/TravelHacks",
   "https://www.reddit.com/r/Shoestring",
   "https://www.reddit.com/r/AskHotels",
   "https://www.reddit.com/r/roadtrip",
   "https://www.reddit.com/r/TravelAdvice",
   "https://www.reddit.com/r/Travel",
   "https://www.reddit.com/r/TalesFromTheFrontDesk",
   "https://www.reddit.com/r/solotravel",
   "https://www.reddit.com/r/hotels",
   "https://www.reddit.com/r/Europetravel"
]


##### Loop through each subreddit URL and send a request

subreddit_urls.each do |community_url|

   ##### Send HTTP GET request with headers to mimic a browser
   
   request = Typhoeus.get(community_url, headers: {
       "accept" => "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
       "accept-encoding" => "gzip, deflate, br, zstd",
       "accept-language" => "en-US,en;q=0.9",
       "content-type" => "application/x-www-form-urlencoded",
       "priority" => "u=1, i",
       "referer" => community_url,
       "user-agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
   })
   
   ##### Fetch the response body as HTML for parsing
   
   response = request.body
   html = Nokogiri::HTML(response)  ##### Parse HTML with Nokogiri
end

In this section, we’ll automate the process of scrolling to load more posts. Reddit dynamically loads posts as you scroll down, so we mimic this by capturing the last_article_id (an identifier for the last post on the page) and using it to load additional posts. This part is essential for going beyond the initial set of posts and accessing older threads.

##### Loop to search for posts and load additional threads as we scroll
 html.search('article').each_with_index do |article, index| 

   ##### Extract each post element and convert it to a hash for easy data handling
   reddit_post = article.at('shreddit-post').to_h


   ##### Send a request to access the specific post by appending its permalink
request = Typhoeus.get("https://www.reddit.com#{reddit_post['permalink']}", headers: {
       "accept" => "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
       "accept-encoding" => "gzip, deflate, br, zstd",
       "accept-language" => "en-US,en;q=0.9",
       "content-type" => "application/x-www-form-urlencoded",
       "priority" => "u=1, i",
       "referer" => community_url,
       "user-agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
   })


   ##### Check if there’s a "more-posts-cursor" identifier; if present, capture it to load additional posts
   if reddit_post['more-posts-cursor']
       last_article_id = reddit_post['more-posts-cursor']
   end
end

To begin, let’s extract metadata such as the subreddit’s name, description, rules, and member count. We’ll use Nokogiri to parse the page and locate elements based on CSS selectors. This can be done directly on the main page of the subreddit.

We can find information such as post title, author, upvotes, and comment counts on the main page of the subreddit, specifically the variable reddit_post.

Handling media content such as **images, videos, or embedded links**.

To gather data about each post, such as images, videos, or embedded link, we’ll loop through each article or shreddit-post on the page.

post_content = html.at("shreddit-post div.text-neutral-content")

The simplest way to gather discussion data from Reddit, including comments and nested replies, is by using the old Reddit version. By sending requests to the .old subdomain, users can often access the same data with fewer restrictions. This approach offers advantages because:

Data on old and new domains often matches.
The main domain tends to have stricter protections against automated requests.
The old subdomain’s data structure can be more user-friendly.
More data can sometimes be accessed per request on the old domain, enhancing efficiency for large-scale scraping.

To load the maximum number of comments for each post, we send a request to the .old subdomain with the parameters limit=500 (the maximum number of comments) and sort=new (ordering from newest to oldest).

request = Typhoeus.get "https://old.reddit.com#{reddit_post['permalink']}?sort=new&limit=500", headers: {
   "accept" => "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
   "accept-encoding" => "gzip, deflate, br, zstd",
   "accept-language" => "en-US,en;q=0.9",
   "content-type" => "application/x-www-form-urlencoded",
   "priority" => "u=1, i",
   "referer" => "https://old.reddit.com#{reddit_post['permalink']}",
   "user-agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
}

For example, consider the URL. This link brings up a page containing all comments and replies, structured in a tree format. Based on how we wish to organize the data—whether as a flat list of comments and replies or as a nested chain up to a certain depth—we can choose either iteration or recursion to traverse the structure.

Iteration is simpler, while recursion offers a more elegant approach, though it can be more resource-intensive.

Here, we’ll use recursion. We will process each first-level comment by calling a function to parse it, which will then recursively call itself to parse replies. This continues until we reach all replies or achieve a specified depth (in this case, 3 levels).

To implement such step, we need to create an initial request for the comment.

request = Typhoeus.get "https://old.reddit.com#{reddit_post['permalink']}?sort=new&limit=500", headers: {
   "accept" => "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
   "accept-encoding" => "gzip, deflate, br, zstd",
   "accept-language" => "en-US,en;q=0.9",
   "content-type" => "application/x-www-form-urlencoded",
   "priority" => "u=1, i",
   "referer" => "https://old.reddit.com#{reddit_post['permalink']}",
   "user-agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
}

This code sends a GET request to the .old Reddit subdomain to retrieve comments for a specific post. The request includes parameters sort=new and limit=500 to load up to 500 of the newest comments. Also, it includes headers specifying acceptable content formats, language, user-agent, and encoding, which helps mimic a legitimate browser request and avoid bot detection. Now we need to parse the HTML content of Reddit.

html = Nokogiri::HTML(content)
html.search('div.clearleft').remove

This line loads the HTML response using Nokogiri and removes any div.clearleft elements from the page content to clean up irrelevant content. Now let’s set up some variables.

@vars = page['vars']
@topic = @vars['community_url'].split("/").last
reddit_post = @vars["reddit_post"]

These lines extract information from @vars, like community_url and reddit_post, to set up basic parameters, like @topic (the subreddit name). Next, we defined a function called fetch_more_comments. Just like the name of the function, the purpose of the function is to fetch more comments by parsing the “Load More” button.

def fetch_more_comments(more_comments_element, level, comment, r, main_data)
   link_id, sort, children, limit_children = more_comments_element.at('a').attr('onclick').scan(/\'([^"']*)\'/)
     
   return {
       url: "https://old.reddit.com/api/morechildren",
       method: "POST",
       page_type: "more_comments",
       http2: true,
       body: URI.encode_www_form({
           link_id: link_id.first,
           sort: "new",
           children: children.first,
           limit_children: "False",
           r: r,
           renderstyle: "html"
       }),
       headers: {
           "accept" => "application/json, text/javascript, */*; q=0.01",
           "accept-encoding" => "gzip, deflate, br, zstd",
           "accept-language" => "en",
           "content-type" => "application/x-www-form-urlencoded; charset=UTF-8",
           "priority" => "u=1, i",
           "user-agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36",
           "x-requested-with" => "XMLHttpRequest"
       },
       vars: @vars.merge({
           parent_comment: link_id.first,
           children: children.first,
           level: level,
           comment: comment,
           main_data: main_data
       })
   }

Next, we defined function called parse_comment to turn the HTML version of Reddit data into structured data, in this case in the form of CSV table.

main_data stores relevant details of each comment, like content, author, date, likes, etc. It checks for the presence of "more comments" links, and if found, appends requests to final_list['pages'].If a comment has replies (nested comments) and the recursion depth is below 3, the function recursively calls itself to fetch nested replies, appending each parsed comment to final_list['outputs'].

def parse_comment(main_data, comment, level, its_last_comment, final_list)
   comment_id = comment.attr('data-fullname').to_s.gsub("t1_", "")
   main_data["Comment_Level_#{level}"] = comment.at('div.usertext-body').text.split("\n").map(&:strip).reject(&:empty?).join("\n")
   main_data["Comment_Level_#{level} ID"] = comment.attr('data-fullname')
   main_data["Comment_Level_#{level} Author"] = comment.attr('data-author')
   main_data["Date_of_Commment_Level_#{level}"] = DateTime.parse(comment.at('.tagline time').attr("datetime")).strftime("%Y%m%d %H:%M")
   main_data["Number of Likes_Comment_#{level}"] = comment.at('.tagline .score.unvoted').attr('title') rescue nil
   main_data["Number of Dislikes_Comment_#{level}"] = nil
   main_data["_id"] = "#{main_data["Comment_Level_1 ID"]}_#{main_data["Comment_Level_2 ID"]}_#{main_data["Comment_Level_3 ID"]}"
   main_data["_collection"] = "comments"




   more_comments_pages = []
   more_comments_element = comment.next_element rescue nil
   has_more_comments = more_comments_element && more_comments_element.attr('class') =~ /morechildren/i




   if level < 3 && has_more_comments && !its_last_comment
       final_list['pages'] << fetch_more_comments(more_comments_element, level, comment, @topic, main_data.clone)
       # save_pages(pages)
   end

Next, we are going to loop our process for each comment with the while function and using the previous parse function.

comments_count = html.search('[data-type="comment"]').count
comment = html.search('[data-type="comment"]')[0]




while comment
   next_element = comment.next_element
   main_data = {
       "Topic" => @topic,
       "Subtopic" => "",
       "ID of Subreddit" => reddit_post["subreddit-id"],
       "User Name" => reddit_post["author"],
       "Date of Subreddit_Entry Shared" => DateTime.parse(reddit_post['created-timestamp']).strftime("%Y%m%d %H:%M"),
       "Access Date" => Time.parse(page['fetched_at']).strftime("%Y%m%d"),
       "Subreddit_page" => @vars['community_url'],
       "Subreddit_Entry_Title" => html.at('#siteTable p.title a.title').text,
       "Subreddit_Entry_Text" => (html.at("#siteTable div.usertext-body").text.split("\n").map(&:strip).reject(&:empty?).join("\n") rescue ""),
       "Number of Likes_Subreddit_Entry" => html.at('#siteTable .score.unvoted').attr('title'),
       "Number of Dislikes_Subreddit_Entry" => nil,
       "URL Subreddit_Entry" => page['url'].gsub("/?limit=500", "").gsub("old.", "www.")
   }
   its_last_comment = next_element.nil? || !(next_element.attr('class') =~ /morechildren/i).nil?  




   final_list = parse_comment(main_data, comment, 1, its_last_comment, {'outputs' => [], 'pages' => []})




   final_list['outputs'].uniq.each do  |formated_comment|
       outputs << formated_comment
       save_outputs(outputs) if outputs.count > 99
   end




   final_list['pages'].uniq.each do |more_comments_page|
       outputs << more_comments_page
       save_pages(pages) if pages.count > 99
   end




   if (!next_element.nil? && next_element.attr('class') =~ /morechildren/i)
       more_comments_element = next_element




       pages << fetch_more_comments(more_comments_element, 0, nil, @topic, main_data)




       comment = nil
   elsif (!next_element.nil? && next_element.attr('class') =~ /comment/i)
       comment = next_element
   elsif next_element.nil? || next_element.inner_html.empty?
       comment = nil
   end
end

The code snippet shows the process of scraping comments from a Reddit post on the old version of the site, focusing on handling nested replies efficiently. It begins by counting the total number of comments and setting the first comment as the starting point for processing. Within a while loop, the code checks for the existence of the current comment and identifies the next element in the HTML structure for navigation. It constructs a data structure called main_data, which aggregates essential information about the subreddit and the specific post, including the topic, subreddit ID, author, creation date, and content of the entry.

A key variable, its_last_comment, is evaluated to determine if the current comment is the last one in the section, indicating whether there are more comments to load. The function parse_comment is invoked to process the current comment, passing the necessary data and indicating the current level of recursion. As the code extracts and formats the comments, it saves unique outputs to an outputs array, triggering a save operation when the count exceeds 99.

After processing the current comment, the code examines the next element. If it is identified as a "more comments" link, the code fetches additional comments and prepares for further processing. If it is a new comment, it updates the current comment for the next iteration. If the next element is nil or empty, the loop exits. Overall, this process effectively navigates the comment structure, allowing for comprehensive retrieval of discussion data, including nested replies, while managing resource usage through recursion.

After parsing all comments, the script saves the parsed comments (outputs) and pagination data (pages) for later processing or storage, using save function.

save_outputs(outputs)
save_pages(pages)

Alternatives to Custom Scraping

If you want to scrape specific data but avoid the hassle of building your own tools, you can have DataHen handle it for you. With their custom web scraping service, DataHen delivers precisely the data you need from almost any website, be it for hotels, restaurants, automotive parts, or other unique industries. Having worked with diverse datasets across these sectors, DataHen understands the nuances and specific requirements involved in data extraction.

Their team takes care of every detail, from extraction to data cleaning and structuring, ensuring you receive information that’s ready for analysis. Unlike one-size-fits-all tools, DataHen’s service is fully customizable, adapting to your specifications while minimizing risks like downtime and compliance issues.

With DataHen managing the complexities of web scraping, you’re free to focus on leveraging insights for your business, backed by reliable, structured data tailored to drive smarter decisions. You can contact us through the website to discuss your specific needs and get started on your data journey today!

Conclusion

In summary, this guide has covered the essential steps for scraping Reddit data from subreddits, including setting up access credentials, retrieving data through the Reddit API, and handling and storing the data efficiently. The scraped data opens up a range of applications: it can be analyzed for sentiment to gauge community sentiment on various topics, used in trend analysis to track emerging discussions, and even applied in more specialized domains like market research or product feedback.

Looking forward, web scraping and Reddit data collection are set to evolve as tools and techniques continue to improve, allowing for more sophisticated data gathering and analysis. As new libraries and API updates emerge, staying informed will help you adapt and leverage these advancements in future projects.

How to Scrape Reddit Posts, Subreddits and Comments

Lanang Bagaskara

Hasibe Güler

Muhammad Rasyid

Lanang Bagaskara, Hasibe Güler, Muhammad Rasyid