Build a Unified Social Media Data Pipeline (One API, Every Platform)

Anyone who's built a social dashboard knows the integration tax. X's API has its own auth and tiers, Instagram's Graph API wants a linked Facebook Page, TikTok has signing requirements, YouTube has quota math. Maintaining four official integrations — each with its own auth, response shape, and breaking changes — is a part-time job that has nothing to do with your actual product.

This guide builds a unified pipeline: one class that fetches from any platform through a single API and returns a standardized shape, so the rest of your app never has to care where a post came from. Code in TypeScript/JavaScript.

Step 1: Define one standard shape

First decide what a "post" looks like in your system, regardless of origin. Everything normalizes to this.

interface StandardPost {
  id: string;
  platform: "twitter" | "instagram" | "tiktok";
  content: string;
  author: string;
  stats: { views: number; likes: number; comments: number; shares: number };
  publishedAt: Date;
  url: string;
}

Step 2: The pipeline class

The whole value is in the per-platform mappers that translate each network's quirky response into StandardPost. The trick is getting each platform's field paths right — they differ a lot.

const axios = require("axios");

class SocialPipeline {
  constructor(apiKey) {
    this.client = axios.create({
      baseURL: "https://api.sociavault.com/v1/scrape",
      headers: { "x-api-key": apiKey },
    });
  }

  async fetchFeed(platform, handle) {
    if (platform === "twitter") return this._twitter(handle);
    if (platform === "instagram") return this._instagram(handle);
    if (platform === "tiktok") return this._tiktok(handle);
    throw new Error(`Unsupported platform: ${platform}`);
  }

  async _twitter(handle) {
    const { data } = await this.client.get("/twitter/user-tweets", {
      params: { handle },
    });
    return (data.data.tweets || []).map((t) => ({
      id: t.id,
      platform: "twitter",
      content: t.text,
      author: handle,
      stats: {
        views: t.views || 0,
        likes: t.likes || 0,
        comments: t.replies || 0,
        shares: t.retweets || 0,
      },
      publishedAt: new Date(t.created_at),
      url: t.url || `https://x.com/${handle}/status/${t.id}`,
    }));
  }

  async _instagram(handle) {
    const { data } = await this.client.get("/instagram/posts", {
      params: { handle },
    });
    return (data.data.items || []).map((p) => ({
      id: p.id || p.pk,
      platform: "instagram",
      content: p.caption?.text || "",
      author: handle,
      stats: {
        views: p.play_count || p.video_view_count || 0,
        likes: p.like_count || 0,
        comments: p.comment_count || 0,
        shares: 0, // Instagram doesn't expose shares publicly
      },
      publishedAt: new Date((p.taken_at || 0) * 1000),
      url: `https://instagram.com/p/${p.code}`,
    }));
  }

  async _tiktok(handle) {
    const { data } = await this.client.get("/tiktok/videos", {
      params: { handle },
    });
    return (data.data.aweme_list || []).map((v) => {
      const s = v.statistics || {};
      return {
        id: v.aweme_id,
        platform: "tiktok",
        content: v.desc,
        author: handle,
        stats: {
          views: s.play_count || 0,
          likes: s.digg_count || 0,
          comments: s.comment_count || 0,
          shares: s.share_count || 0,
        },
        publishedAt: new Date((v.create_time || 0) * 1000),
        url: `https://tiktok.com/@${handle}/video/${v.aweme_id}`,
      };
    });
  }
}

module.exports = SocialPipeline;

The field paths that matter (and that DIY versions usually get wrong): TikTok stats are at aweme_list[].statistics.play_count with create_time in Unix seconds; Instagram posts live in items[] with taken_at in seconds and like_count/comment_count; X tweets come under data.tweets. Get those right and everything downstream is uniform.

Step 3: One universal feed

Now fetch every platform in parallel and merge into a single sorted stream.

const SocialPipeline = require("./SocialPipeline");
const pipeline = new SocialPipeline(process.env.SOCIAVAULT_API_KEY);

async function brandFeed(brand) {
  const sources = [
    ["twitter", brand.twitter],
    ["instagram", brand.instagram],
    ["tiktok", brand.tiktok],
  ];

  const results = await Promise.all(
    sources.map(
      ([platform, handle]) =>
        pipeline.fetchFeed(platform, handle).catch(() => []), // one failure doesn't sink the feed
    ),
  );

  const all = results.flat().sort((a, b) => b.publishedAt - a.publishedAt);
  all
    .slice(0, 5)
    .forEach((p) =>
      console.log(
        `[${p.platform}] ❤️ ${p.stats.likes} 👁️ ${p.stats.views} — ${p.content.slice(0, 50)}`,
      ),
    );
  return all;
}

brandFeed({ twitter: "nike", instagram: "nike", tiktok: "nike" });

The .catch(() => []) is deliberate — if one platform hiccups, you still get a feed from the others instead of a total failure.

Why this pays off

Data normalization is the hardest, most thankless part of social analytics, and it's where codebases rot. Abstracting every platform's quirks behind one pipeline means the rest of your app speaks a single language (StandardPost), your code shrinks dramatically, and platform changes are contained to one mapper instead of rippling everywhere. Add a platform later and nothing downstream changes.

Frequently Asked Questions

How do I combine data from multiple social platforms?

Define one standard data shape, write a small mapper per platform that translates each network's response into that shape, and fetch them through a single API. The pipeline class here does exactly that for TikTok, Instagram, and X, returning a uniform StandardPost regardless of source.

Why is normalizing social data so hard?

Every platform structures its data differently — different field names, nesting, and timestamp formats — and each changes over time. Mapping them all into one consistent shape is the core challenge; getting the per-platform field paths right is most of the work.

What are the field paths that trip people up?

TikTok video stats sit at aweme_list[].statistics.play_count with create_time in Unix seconds; Instagram posts are in items[] with taken_at in seconds; X tweets come under data.tweets. Using the wrong paths (like a top-level stats.playCount for TikTok) is the most common bug.

Does one platform failing break the whole pipeline?

Not if you isolate failures — wrapping each platform fetch so an error returns an empty result lets the rest of the feed succeed. The example uses .catch(() => []) for exactly this, which matters for a pipeline running unattended.

Can I add more platforms later?

Yes — that's the benefit of the standard shape. Adding a platform means writing one new mapper to StandardPost; nothing downstream changes because the rest of your app only ever sees the normalized format.

Does Instagram expose shares?

No — share counts aren't part of public Instagram post data, which is why the mapper sets shares to 0 for Instagram. Likes, comments, and (for video) play counts are available; shares and saves are not.

The bottom line

A unified pipeline turns "four fragile integrations" into "one API and a handful of mappers." Define a standard shape, normalize each platform into it, fetch in parallel, and your analytics app stops caring which network a post came from.

Want to build your pipeline? Start free with SociaVault with 50 credits.