How to Build a Twitter (X) Thread Scraper (Step by Step)

Some of the best writing on the internet is twenty tweets stapled together. The format is great; reading it in the X app is not — ads, "show more" buttons, and reply noise constantly break the flow. Tools like ThreadReaderApp fix that, but building your own is genuinely useful if you want to save threads to Notion, turn your own threads into LinkedIn carousels, or convert them into blog posts.

This tutorial builds a thread scraper in Node.js: give it a tweet URL, get back the whole thread as clean Markdown. The catch most tutorials hand-wave — actually reconstructing the thread from replies — we'll do properly.

The cost problem (and the workaround)

X's official API got expensive and heavily rate-limited, which is hard to justify for a side project. A scraping API reads the public web view of tweets instead, at a tiny fraction of the cost. See the full landscape in X/Twitter API alternatives.

Step 1: Get the main tweet

const API_KEY = process.env.SOCIAVAULT_API_KEY;
const BASE = "https://api.sociavault.com/v1/scrape/twitter";

async function getTweet(tweetUrl) {
  const res = await fetch(`${BASE}/tweet?url=${encodeURIComponent(tweetUrl)}`, {
    headers: { "x-api-key": API_KEY },
  });
  const json = await res.json();
  if (!json.success) throw new Error(json.error);
  return json.data; // log once to confirm fields (text, user, created_at, media)
}

Step 2: Reconstruct the thread from replies

This is the part that actually matters. A "thread" is the original author replying to themselves. So you fetch the replies to the main tweet and keep only the ones written by the same author, in order. The comments endpoint takes a pid (the tweet's post ID) and a cursor for paging.

async function getReplies(postId, cursor) {
  const params = new URLSearchParams({ pid: postId });
  if (cursor) params.set("cursor", cursor);
  const res = await fetch(`${BASE}/comments?${params}`, {
    headers: { "x-api-key": API_KEY },
  });
  const json = await res.json();
  if (!json.success) return { replies: [], cursor: null };
  // Be defensive about the container and cursor field names.
  const replies =
    json.data.tweets || json.data.replies || json.data.comments || [];
  return { replies, cursor: json.data.cursor || null };
}

async function buildThread(tweetUrl) {
  const main = await getTweet(tweetUrl);
  const authorId = main.user?.id || main.user?.id_str;
  const postId = main.id || tweetUrl.split("/status/")[1]?.split(/[/?]/)[0];

  // Page through replies, keeping only the author's own (the thread)
  let cursor = null,
    own = [];
  for (let page = 0; page < 5; page++) {
    const { replies, cursor: next } = await getReplies(postId, cursor);
    own.push(
      ...replies.filter((r) => (r.user?.id || r.user?.id_str) === authorId),
    );
    cursor = next;
    if (!cursor || replies.length === 0) break;
  }

  own.sort((a, b) => new Date(a.created_at) - new Date(b.created_at));
  return [main, ...own];
}

The page cap matters — a tweet with thousands of replies would otherwise loop forever and burn credits. Five pages is plenty to capture a normal thread.

Step 3: Format as Markdown

function toMarkdown(tweets) {
  const author =
    tweets[0].user?.screen_name || tweets[0].user?.username || "author";
  let md = `# Thread by @${author}\n\n`;
  for (const t of tweets) {
    const clean = (t.text || "").replace(/https:\/\/t\.co\/\w+/g, "").trim();
    md += `${clean}\n\n`;
    for (const m of t.media || []) {
      if (m.url) md += `![media](${m.url})\n\n`;
    }
  }
  return md;
}

Step 4: Save it

import fs from "fs";

async function saveThread(url) {
  const tweets = await buildThread(url);
  const id = url.split("/status/")[1]?.split(/[/?]/)[0] || Date.now();
  fs.writeFileSync(`thread_${id}.md`, toMarkdown(tweets));
  console.log(`Saved ${tweets.length} tweets to thread_${id}.md`);
}

saveThread("https://x.com/user/status/1758529912345678900");

Step 5: Make it a tool

Wrap it in something you'll actually use: a tiny Express endpoint, a Telegram bot ("send a link, get a PDF back"), or a scheduled job that archives threads from accounts you follow into Notion. The scraping core stays the same — you're just changing the trigger and the output format.

A note on honesty: reply ordering and author matching depend on the fields the response returns, so log a real response once and adjust the field names (user.id, created_at, media) to match. The logic is sound; the exact keys are worth confirming.

Frequently Asked Questions

How do I scrape a whole Twitter/X thread, not just one tweet?

Fetch the original tweet, then fetch its replies and keep only the ones from the same author, sorted by time — that reconstructs the thread. A single tweet endpoint gives you the first post; the replies endpoint (by post ID) gives you the rest of the conversation to filter.

How do you tell which replies are part of the thread?

A thread is the original author replying to themselves, so you filter the replies down to those whose author ID matches the original tweet's author, then sort chronologically. Replies from other accounts are conversation, not the thread.

Is this cheaper than X's official API?

Considerably. The official API's paid tiers are costly and rate-limited for what a small project needs. A scraping API reads the public tweet view at a fraction of the cost, which is what makes a personal thread-reader tool viable.

Can I convert threads to other formats?

Yes — once you have the thread as an array of tweets, you can render it to Markdown (shown here), PDF, a LinkedIn carousel, or push it into Notion. The format step is independent of the scraping.

Will this capture images and videos in the thread?

It captures whatever media fields the response includes — the formatter above pulls media URLs into the Markdown. Confirm the media field shape against a real response, since exact keys can vary.

Do I need to handle pagination?

Yes for long threads — replies come in pages, so loop with the cursor until you've collected the author's tweets, with a sensible page cap so a massively-replied tweet doesn't loop endlessly or overspend credits.

The bottom line

A thread scraper is a small, satisfying build: fetch the tweet, reconstruct the thread from the author's own replies, format, save. The only real subtlety is the reconstruction step — and now you've got it.

Want to build your thread reader? Start free with SociaVault with 50 credits.