Handling Pagination Cursors Across Social Platforms (A Developer Guide)

Pagination is where clean social-data code goes to die. Each platform invented its own scheme, and if you're pulling from several, you end up with a tangle of cursor, next_max_id, continuationToken, and after scattered through your codebase, each behaving slightly differently, each with its own "you've reached the end" signal. Get it wrong and you either miss data or loop forever burning credits. Here's how to handle all of it cleanly.

Why every platform is different

There's no universal pagination standard, so providers expose whatever the underlying platform uses. In practice you'll meet a few shapes:

• Opaque cursor/token you pass back verbatim to get the next page (most common). You don't parse it; you just echo it.
• ID-based paging where you pass the last item's ID (Instagram-style next_max_id).
• Offset/page number for a few endpoints.

The mistake is scattering each platform's quirk across your app. The fix is to hide all of it behind one function and never think about it again at the call site.

The unified paginator

Because SociaVault keeps a consistent envelope, base URL https://api.sociavault.com/v1, x-api-key header, payload under data, 1 credit per call, you can write one generic paginator and just tell it which field carries the "next" token for a given endpoint:

import os, time, requests

API_KEY = os.environ["SOCIAVAULT_API_KEY"]
BASE = "https://api.sociavault.com/v1"

def paginate(path, base_params, *, page_param, next_field,
             items_field, max_pages=10, want=None, delay=1.0):
    """
    path         e.g. "/scrape/instagram/posts"
    page_param   the query param that carries the token, e.g. "next_max_id"
    next_field   where the next token lives in the response data
    items_field  where the array of items lives in the response data
    """
    out, token, pages = [], None, 0
    while pages < max_pages:                     # hard ceiling: never loops forever
        params = dict(base_params)
        if token:
            params[page_param] = token
        r = requests.get(f"{BASE}{path}", headers={"x-api-key": API_KEY},
                         params=params, timeout=60)
        r.raise_for_status()
        data = r.json().get("data", {}) or {}
        out.extend(data.get(items_field, []) or [])
        token = data.get(next_field)             # None/absent => last page
        pages += 1
        if not token or (want and len(out) >= want):
            break
        time.sleep(delay)                        # gentle pacing
    return out[:want] if want else out

Now every platform is a one-liner with the right field names plugged in:

# Instagram posts (ID-based)
ig = paginate("/scrape/instagram/posts", {"handle": "natgeo"},
              page_param="next_max_id", next_field="next_max_id",
              items_field="items", want=90)

# TikTok videos (cursor-based)
tt = paginate("/scrape/tiktok/videos", {"handle": "mrbeast"},
              page_param="max_cursor", next_field="max_cursor",
              items_field="aweme_list", want=90)

Confirm the exact field names from a real response per endpoint, log one page first, because the next-token field and items field genuinely differ across platforms. The point is that the logic is written once; only the field names change.

The three rules that keep pagination safe

Whatever you build, these three keep you out of trouble:

1. Always cap pages. The max_pages ceiling is non-negotiable. An endpoint that keeps returning a token, or a bug, will otherwise loop until your credits are gone. Two ceilings (max_pages and want) is even safer.

2. Trust the "no token" signal, but verify. The end of data is signaled by a missing/null next token. Handle both None and absent keys, and don't assume an empty items array alone means the end, some endpoints return an empty page with a token before finishing.

3. Pace and handle failures mid-run. A delay between pages is polite and reduces errors. And because a multi-page pull can fail on page 7, keep partial results and consider check-pointing the token so you can resume rather than restarting (and re-spending).

De-duplicate as you go

One subtle gotcha: paginated feeds occasionally return an overlapping item across page boundaries. If exact counts matter, dedupe by item ID as you collect:

def dedupe(items, id_field):
    seen, out = set(), []
    for it in items:
        i = it.get(id_field)
        if i not in seen:
            seen.add(i)
            out.append(it)
    return out

The honest limits

• Field names really do differ. There's no way around confirming next_field and items_field per endpoint from a live response. Don't assume; log and check.
• Caps can truncate. max_pages protects your balance but can cut a long feed short. Log when you hit the ceiling so truncation is visible, not silent.
• Deep pagination gets less reliable. The further back you page, the more platforms may thin or reorder results. Very deep historical pulls are inherently best-effort.
• Every page is a credit. Deep pulls add up fast. Set want to what you actually need and stop, don't page to the end out of habit.
• Tokens can expire. An opaque cursor may not stay valid indefinitely. For resumable jobs, don't assume a saved token works days later, be ready to restart.

Frequently Asked Questions

Why does every platform paginate differently?

Because there's no universal standard, providers expose whatever scheme the underlying platform uses, opaque cursors, ID-based paging like Instagram's next_max_id, or offsets. The practical fix is to hide those differences behind one generic paginator.

How do I avoid an infinite pagination loop?

Always set a hard max_pages ceiling, and ideally a target item count too. An endpoint that keeps returning a next token, or a bug, will otherwise loop until your credits run out. Two independent ceilings is the safe pattern.

How do I know I've reached the last page?

The next-token field is missing or null. Handle both cases, and don't rely on an empty items array alone, some endpoints return an empty page with a token before truly finishing. Trust the token signal but verify.

Can I write one pagination function for all platforms?

Yes. Because the response envelope is consistent, you can write one paginator and pass in the endpoint-specific field names (the token field and the items field). The logic is written once; only the field names change per endpoint.

Do I need to de-duplicate paginated results?

Sometimes. Paginated feeds can return an overlapping item at page boundaries. If exact counts matter, dedupe by item ID as you collect so a boundary overlap doesn't inflate your totals.

Does deep pagination cost more?

Yes, every page is one credit, so deep pulls add up. Set a target count and stop once you have enough rather than paging to the very end. You start with 50 free credits to test your pagination logic.

---

Want one clean paginator across every platform instead of a tangle of special cases? Start free with 50 credits, no card required. New to the API? Start with the indie developer guide.