How to Get Every Comment on a Reddit Thread, Including Deep Replies

The thread says 55 comments. Your code got 25. You check for an off-by-one, add a retry, run it again. Still 25.

Nothing's broken. Reddit never sends a whole thread in one go. It sends the first batch and leaves markers where the rest would be, and there are two different kinds of marker. Most people follow one and miss the other, which is how you end up with a dataset that looks complete and isn't.

We walked a real thread end to end to see exactly where the missing comments live. Here's how to get all of them.

Why the first request isn't the whole thread

A Reddit thread is a tree. Top-level comments reply to the post. Replies hang off those comments, and replies to replies hang off those, sometimes ten levels deep.

Reddit sends the first part of that tree and stubs out the rest. In our responses, that shows up in two places:

1. One cursor for the thread, at data.more.cursor, for the next batch of top-level comments.
2. One cursor per comment, at replies.more.cursor, for the next batch of replies to that comment.

Follow only the first and you get every top-level comment, but none of the replies Reddit collapsed. Follow only the second and you never get past the first page. You need both.

What we found on a real thread

We took a discussion thread that Reddit reported as having 55 comments and walked it.

The first request returned 16 top-level comments and 25 comments in total, counting the replies that came inline. Following data.more.cursor, page two added 8 more top-level comments and page three added 9. Page three came back with has_more: false, so the top level was done: 33 top-level comments, 43 in total.

That's where most scrapers stop, and they're 12 comments short.

Along the way, 15 comments had a replies.more.cursor of their own. Following those turned up a few more, and following them in turn took us to 19 reply cursors altogether. They added 13 comments we hadn't seen. Final count: 56 unique comments, against the 55 Reddit reported, from 22 requests in all.

The one extra comment isn't a bug either. The thread was live, and people kept commenting while we crawled. More on that below.

On a big thread, the top-level loop matters even more. Our docs example, the AskReddit thread "People who haven't pooped in 2019 yet, why are you still holding on to last years shit?", has 7,897 comments. Six top-level pages gave us 325 top-level comments, and the cursor was still saying there was more.

The response shape

Here's what you're working with. The first request needs only the post URL:

GET https://api.sociavault.com/v1/scrape/reddit/post/comments?url=<thread url>

The fields that matter:

• data.post.num_comments: the count Reddit reports for the thread
• data.comments: the top-level comments in this batch
• each comment's id, author, body, score, created_utc and parent_id
• each comment's replies.items: the replies that came inline
• each comment's replies.more: has_more, and a cursor for the rest of its replies
• data.more: has_more, and a cursor for the next batch of top-level comments

To get more, you call the same endpoint again with the same url plus a cursor. It's one cursor per call, and they come in two shapes. The thread-level one is a long opaque string. The reply-level ones are often short, comment IDs like pc4aldc. Don't try to parse either. Pass them back exactly as you got them.

A detail that'll bite you: some list fields come back as objects with numbered keys ("0", "1", …) instead of arrays. Normalise them before you loop, as the code below does.

The crawler

This follows both kinds of cursor, deduplicates by comment ID, and stops at a page cap so a huge thread can't run away with your credits:

const API = "https://api.sociavault.com/v1/scrape/reddit/post/comments";
const KEY = process.env.SOCIAVAULT_API_KEY;
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
const toArray = (x) => (Array.isArray(x) ? x : x ? Object.values(x) : []);

async function fetchBatch(url, cursor) {
  const params = new URLSearchParams({ url });
  if (cursor) params.set("cursor", cursor);
  const res = await fetch(`${API}?${params}`, { headers: { "x-api-key": KEY } });
  const body = await res.json();
  if (body?.error) throw new Error(body.error);
  return body.data ?? {};
}

async function getAllComments(threadUrl, { maxRequests = 60 } = {}) {
  const comments = new Map();   // id -> comment
  const replyCursors = [];      // collapsed reply branches still to fetch
  const seenCursors = new Set();
  let requests = 0;

  // Record a comment and everything under it; queue any collapsed replies.
  const walk = (c) => {
    if (!c?.id) return;
    comments.set(c.id, c);
    for (const r of toArray(c.replies?.items)) walk(r);
    const more = c.replies?.more;
    if (more?.has_more && more.cursor && !seenCursors.has(more.cursor)) {
      seenCursors.add(more.cursor);
      replyCursors.push(more.cursor);
    }
  };

  // Phase 1: every top-level page.
  let cursor = null;
  let reported = null;
  do {
    const data = await fetchBatch(threadUrl, cursor);
    requests++;
    reported ??= data.post?.num_comments ?? null;
    toArray(data.comments).forEach(walk);
    cursor = data.more?.has_more ? data.more.cursor : null;
    if (cursor) await sleep(800);
  } while (cursor && requests < maxRequests);

  // Phase 2: every collapsed reply branch, including ones discovered along the way.
  while (replyCursors.length && requests < maxRequests) {
    const data = await fetchBatch(threadUrl, replyCursors.shift());
    requests++;
    toArray(data.comments).forEach(walk);
    await sleep(800);
  }

  return { reported, found: comments.size, requests, comments: [...comments.values()] };
}

const result = await getAllComments("https://www.reddit.com/r/AskReddit/comments/ablzuq/");
console.log(`${result.found} comments (Reddit reports ${result.reported}) in ${result.requests} requests`);

And in Python:

import os, time, requests

API = "https://api.sociavault.com/v1/scrape/reddit/post/comments"
HEADERS = {"x-api-key": os.environ["SOCIAVAULT_API_KEY"]}

def to_list(x):
    return x if isinstance(x, list) else list((x or {}).values())

def fetch_batch(url, cursor=None):
    params = {"url": url, **({"cursor": cursor} if cursor else {})}
    body = requests.get(API, params=params, headers=HEADERS, timeout=120).json()
    if body.get("error"):
        raise RuntimeError(body["error"])
    return body.get("data") or {}

def get_all_comments(thread_url, max_requests=60):
    comments, reply_cursors, seen = {}, [], set()
    requests_made, reported = 0, None

    def walk(c):
        if not c or not c.get("id"):
            return
        comments[c["id"]] = c
        replies = c.get("replies") or {}
        for r in to_list(replies.get("items")):
            walk(r)
        more = replies.get("more") or {}
        if more.get("has_more") and more.get("cursor") and more["cursor"] not in seen:
            seen.add(more["cursor"])
            reply_cursors.append(more["cursor"])

    cursor = None
    while True:  # phase 1: top-level pages
        data = fetch_batch(thread_url, cursor)
        requests_made += 1
        reported = reported or (data.get("post") or {}).get("num_comments")
        for c in to_list(data.get("comments")):
            walk(c)
        more = data.get("more") or {}
        cursor = more.get("cursor") if more.get("has_more") else None
        if not cursor or requests_made >= max_requests:
            break
        time.sleep(0.8)

    while reply_cursors and requests_made < max_requests:  # phase 2: reply branches
        for c in to_list(fetch_batch(thread_url, reply_cursors.pop(0)).get("comments")):
            walk(c)
        requests_made += 1
        time.sleep(0.8)

    return {"reported": reported, "found": len(comments), "requests": requests_made}

Store parent_id with each comment and you can rebuild the tree later. Top-level comments have a parent starting with t3_ (the post); replies have one starting with t1_ (another comment).

Why your count won't match exactly

Don't write a test that asserts found === num_comments. It'll fail for reasons that have nothing to do with your code:

• Live threads keep moving. One thread we were testing grew from 37 to 55 comments over two days. Comments posted mid-crawl can push you over the reported count, and that's where our extra one came from.
• Removed and deleted comments. Reddit's count can include comments that have since been deleted or removed by moderators, which may come back as placeholders or not at all.
• The count itself lags. num_comments is Reddit's number, and it isn't always updated the instant something changes.

A practical check: within a few percent of num_comments on a quiet thread means you got everything that's there.

What it costs

Each request is one credit. Our 55-comment thread took 22 requests: 3 top-level pages and 19 reply branches. Reply branches are usually the bigger share, because every collapsed conversation is its own request.

That's why the crawler has maxRequests. For a thread of a few dozen comments, 60 is generous. For a thread in the thousands, decide upfront what you need. If top-level comments are enough, as they often are for sentiment or topic analysis, skip phase 2 entirely and save most of the cost. Our guide to handling pagination cursors across platforms covers how to budget crawls like this.

The honest limits

• Only what's public. Comments removed by moderators, or in private and quarantined communities, aren't available.
• Huge threads are expensive to finish. A thread with thousands of comments can need hundreds of reply-branch requests. Cap it, or take top-level only.
• Order isn't chronological. Comments come back in Reddit's ranking, not by time. Sort on created_utc yourself if time matters.
• Counts drift. On an active thread, your total and Reddit's will rarely match to the comment.

If you're mining threads for research, finding pain points on Reddit shows what to do with the comments once you have them.

Frequently Asked Questions

Why does my Reddit scraper only return about 20 comments?

Reddit sends threads in batches. The first response includes the first batch of top-level comments and a cursor for the rest. You have to follow data.more.cursor to get the remaining top-level comments, and each comment's replies.more.cursor to get collapsed replies.

What's the difference between the two Reddit cursors?

data.more.cursor loads the next batch of top-level comments on the thread. replies.more.cursor on an individual comment loads more replies to that comment. A complete crawl follows both.

How do I know when I've got all the comments?

When data.more.has_more is false and you've followed every replies.more cursor you found. Your total should land close to post.num_comments, though not always exactly.

Why is my comment count higher than num_comments?

The thread is probably still active. Comments posted while you crawl are included, but num_comments was read when you started. A small overshoot on a live thread is normal.

How many requests does it take to get a whole thread?

It depends on how many conversations Reddit collapsed. Our 55-comment thread took 22 requests: 3 for top-level pages and 19 for reply branches. Each request is one credit.

Can I get comments in chronological order?

Not directly. Comments come back in Reddit's ranking order. Collect them all, then sort by created_utc.

---

Have a thread you need in full? Sign up for 50 free credits and run the crawler above. It'll tell you how many comments it found against the number Reddit reports.