How to Scrape ChatGPT: SSE Streams and Citations

To scrape ChatGPT is not the same as calling the OpenAI API. The web interface delivers its answer as a Server-Sent Events (SSE) stream. That stream is a UTF-8 text protocol, and each update arrives on a line prefixed with data: (MDN).

Two more obstacles stand in the way. The interface hides citations behind lazy-loaded UI. And the whole page sits behind Cloudflare protections.

Why does that matter for SEO and AI visibility work? Because the official API never shows the same user-facing answer. If you need citations, shopping cards, brand entities, or source panels, you have to scrape ChatGPT’s web surface instead.

This guide shows how to scrape ChatGPT end to end. You will learn how the SSE stream works and where citations hide. You will also learn when a managed ChatGPT scraper or visibility tracker is safer than DIY automation.

Why scrape ChatGPT responses?

The chat.completions API and the chatgpt.com web interface use different stacks. The API answers from a base model. The web interface does far more work.

It runs the prompt through a query-fanout step, then dispatches a web search against OpenAI’s search index. It hydrates citations from those results. And for shopping prompts, it injects a product card with merchant offers.

That post-processing is exactly where the data lives. SEO and e-commerce teams need it, yet the API never sees it. So the only way to capture it is to scrape ChatGPT’s rendered answer.

Streaming the official endpoint does not help here either. When you stream chat.completions, OpenAI returns data-only server-sent events. Each token rides in a delta field rather than a full message (OpenAI Cookbook).

What only exists in the web stream

When you scrape ChatGPT, four data types show up that the plain API never returns:

• Citations and source URLs — surfaced via button.group/footnote and lazy-hydrated into a flyout modal ([data-testid="screen-threadFlyOut"])
• Shopping cards — embedded as shopping_card events with merchant offers, prices, and ratings
• Brand entity tags — emitted as entities events with type, name, and confidence
• Search query metadata — the actual sub-queries ChatGPT issued before answering, surfaced in event metadata

When you scrape ChatGPT’s web stream, it also runs roughly 12× cheaper per response than the equivalent paid API call once you account for input and output tokens.

Use cases that only the SSE stream supports

These are the jobs teams actually scrape ChatGPT to do.

• Citation auditing. Did ChatGPT cite our docs vs. a competitor’s? Only the modal sources answer this.
• Shopping-card monitoring. Which merchants are appearing in the product card for “best running shoes”? Only the shopping_card event has the offers list.
• Entity tracking. Is your brand showing up in the entity sidebar with what confidence score? Only the entities events expose this.
• Query-fanout study. What sub-queries did ChatGPT issue? Only present in the metadata, not in the API response.

Understanding ChatGPT’s architecture

A few things make ChatGPT harder to scrape than a typical site. Knowing the pipeline tells you exactly where each piece of data appears. That, in turn, tells you what a scrape-ChatGPT workflow has to intercept.

The response generation pipeline

To scrape ChatGPT well, map its five-step response pipeline first.

1. Query processing. Your prompt is analyzed and broken down into sub-queries using query fanout.
2. Search integration. For web-enabled chats, ChatGPT performs real-time web searches.
3. Streaming generation. Responses stream as Server-Sent Events, the standardized text/event-stream wire format that the WHATWG HTML spec defines as being parsed line by line (WHATWG HTML).
4. Dynamic rendering. Content is rendered client-side using React and web components.
5. Source attribution. Citations and sources are dynamically linked and rendered.

Why ChatGPT resists scrapers

Four properties make it hard to scrape ChatGPT reliably. Each one breaks a naive HTTP-only scraper.

JavaScript-heavy interface:

// ChatGPT uses React components that require full browser rendering
const responseContainer = document.querySelector(
  '[data-message-author-role="assistant"]',
);
// Content isn't available in initial HTML

Streaming response format:

data: {"id": "chatcmpl-abc123", "object": "chat.completion.chunk"}
data: {"choices": [{"delta": {"content": "Hello"}}]}
data: [DONE]

Anti-bot detection:

• Canvas fingerprinting
• Behavioral analysis
• Request pattern monitoring
• CAPTCHA challenges

Dynamic content loading:

• Lazy-loaded source citations
• Modal-based source browsing
• Real-time content updates

The event stream parsing challenge

Most of the work to scrape ChatGPT is parsing Server-Sent Events (SSE). A browser EventSource concatenates consecutive data: lines into one message, inserting a newline between each. That is exactly the reassembly a scraper has to replicate by hand (MDN).

So what makes it tricky when you scrape ChatGPT this way?

Event stream structure:

# Raw ChatGPT event stream example
data: {"id": "chatcmpl-123", "object": "chat.completion.chunk", "created": 1677652288}
data: {"choices": [{"index": 0, "delta": {"content": "I"}}]}
data: {"choices": [{"index": 0, "delta": {"content": " recommend"}}]}
data: {"choices": [{"index": 0, "delta": {"content": " using"}}]}
data: {"choices": [{"index": 0, "delta": {"content": " Python"}}]}
data: [DONE]

Parsing challenges:

1. Mixed data types. Events contain both JSON and special markers.
2. Partial responses. Content arrives in chunks that need reconstruction.
3. Metadata extraction. Model info, citations, and search queries are embedded.
4. Error handling. Network issues can split events mid-stream.

Reassembling the stream in Python

Here is the core parser. It is the first function you write when you scrape ChatGPT from raw SSE text. It splits on data: lines, drops the [DONE] sentinel, and keeps only valid JSON objects.

import json
from typing import List, Dict, Any

def extract_raw_response(input_string: str) -> List[Dict[str, Any]]:
    """Parse ChatGPT's Server-Sent Events stream."""
    json_objects = []

    # Split by lines that start with "data: "
    lines = input_string.split("\n")

    for line in lines:
        # Skip empty lines and non-data lines
        if not line.strip() or not line.startswith("data: "):
            continue

        # Remove "data: " prefix
        json_str = line[6:].strip()

        # Skip special markers like [DONE]
        if json_str == "[DONE]":
            continue

        # Try to parse as JSON
        try:
            json_obj = json.loads(json_str)

            # Only include if it's a dictionary (object), not string or other types
            if isinstance(json_obj, dict):
                json_objects.append(json_obj)
        except json.JSONDecodeError:
            # Skip invalid JSON
            continue

    return json_objects

Reconstructing the full response:

def reconstruct_content(events: List[Dict[str, Any]]) -> str:
    """Rebuild complete response from streaming chunks."""
    content_parts = []

    for event in events:
        # Extract content from delta messages
        if 'choices' in event and len(event['choices']) > 0:
            delta = event['choices'][0].get('delta', {})
            if 'content' in delta:
                content_parts.append(delta['content'])

    return ''.join(content_parts)

Building the scraping infrastructure

The stack you need to scrape ChatGPT has two jobs. It has to defeat Cloudflare-level fingerprinting. And it has to parse a streaming response inside a 60-second window.

Fingerprinting is the first wall. A WebDriver-controlled browser exposes navigator.webdriver as true, a flag detection scripts read on the very first navigation (MDN). Cloudflare’s Turnstile then challenges anything that looks automated, verifying a visitor is human without a classic CAPTCHA (Cloudflare).

Parsing is the second wall. The payload is an SSE text/event-stream response, not a single JSON document, so the standard requires reading it line by line (WHATWG HTML). On top of that, OpenAI layers a [DONE] sentinel to mark the end. Those two facts dictate the whole stack.

The five components you need

Each part below maps to a specific failure you hit when you try to scrape ChatGPT with off-the-shelf tools.

1. Stealth-patched Playwright Chromium — vanilla Playwright is detected on first navigation. Apply playwright-stealth or equivalent canvas/WebGL/audioContext patches.
2. Residential proxy with sticky session — datacenter IPs get a CAPTCHA on first prompt. Cloudflare sets the cf_clearance cookie only after a visitor solves a challenge. It then skips further challenges while that cookie stays valid (Cloudflare). So a sticky session that preserves the cookie across the round-trip is what keeps you unblocked.
3. Network response interceptor scoped to backend-api/f/conversation — ChatGPT issues other API calls during the page lifecycle. Only this URL carries the SSE stream. Playwright surfaces every response through a page.on("response") handler. That lets you read the body of the one request you care about without touching the DOM (Playwright).
4. [DONE]-aware completion detector — DOM-based “is the response complete” checks fire false positives during streaming. The [DONE] sentinel in the SSE body is the only reliable signal.
5. Lazy-citation expander — citations don’t ship in the SSE stream itself; you have to click button.group/footnote and parse the resulting flyout modal after the response completes.

Complete scraper implementation

The class below wires those five parts together into one working scrape-ChatGPT routine. It launches a browser, captures the conversation stream, and hands the raw text to the parser.

import asyncio
from playwright.async_api import async_playwright, Page
import json
from typing import Dict, Any, List

class ChatGPTScraper:
    def __init__(self):
        self.captured_responses = []

    async def setup_page_interceptor(self, page: Page):
        """Set up network request interception."""

        async def handle_response(response):
            # Capture conversation API responses
            if 'backend-api/f/conversation' in response.url:
                response_body = await response.text()
                self.captured_responses.append(response_body)

        page.on('response', handle_response)

    async def scrape_chatgpt(self, prompt: str) -> Dict[str, Any]:
        """Main scraping function."""

        async with async_playwright() as p:
            browser = await p.chromium.launch(headless=False)
            context = await browser.new_context()
            page = await context.new_page()

            # Set up response interception
            await self.setup_page_interceptor(page)

            try:
                # Navigate to ChatGPT
                await page.goto('https://chatgpt.com/?temporary-chat=true')

                # Wait for textarea and enter prompt
                await page.wait_for_selector('#prompt-textarea')
                await page.fill('#prompt-textarea', '/search')  # Enable web search
                await page.press('#prompt-textarea', 'Enter')
                await asyncio.sleep(0.5)

                await page.fill('#prompt-textarea', prompt)
                await page.press('#prompt-textarea', 'Enter')

                # Wait for response completion
                await self.wait_for_response(page)

                # Parse the captured response
                if self.captured_responses:
                    raw_response = self.captured_responses[0]
                    return self.parse_chatgpt_response(raw_response)
                else:
                    raise Exception("No response captured")

            finally:
                await browser.close()

    async def wait_for_response(self, page: Page, timeout: int = 60):
        """Wait for ChatGPT response completion."""

        for i in range(timeout * 2):  # Check every 500ms
            # Check if we have captured responses
            if self.captured_responses:
                # Verify response is complete
                response = self.captured_responses[0]
                if '[DONE]' in response:
                    return

            # Check for content in DOM
            content_div = page.locator('[data-message-author-role="assistant"]').first
            if await content_div.count() > 0:
                content_text = await content_div.text_content()
                if content_text and len(content_text.strip()) > 50:
                    # Check if response seems complete
                    await asyncio.sleep(2)  # Allow for final updates
                    continue

            await asyncio.sleep(0.5)

        raise Exception("Response timeout")

    def parse_chatgpt_response(self, raw_response: str) -> Dict[str, Any]:
        """Parse the raw ChatGPT response into structured data."""

        # Extract streaming events
        events = extract_raw_response(raw_response)

        # Reconstruct content
        content = reconstruct_content(events)

        # Extract metadata
        model = self.extract_model_info(events)
        search_queries = self.extract_search_queries(events)

        return {
            'content': content,
            'model': model,
            'search_queries': search_queries,
            'raw_events': events
        }

    def extract_model_info(self, events: List[Dict]) -> str:
        """Extract model information from events."""
        for event in events:
            if 'model' in event:
                return event['model']
        return 'unknown'

    def extract_search_queries(self, events: List[Dict]) -> List[str]:
        """Extract search queries from the response."""
        queries = []

        # This requires analyzing the metadata in the events
        # Implementation varies based on ChatGPT's current format
        for event in events:
            if 'metadata' in event:
                metadata = event.get('metadata', {})
                if 'search_queries' in metadata:
                    queries.extend(metadata['search_queries'])

        return queries

Parsing the streaming response data

Capturing the stream is only half the job. The value comes from what you pull out of it. This is where a scrape-ChatGPT pipeline earns its keep, because citations, shopping cards, and entities each live in a different place.

Extracting citations and sources

async def extract_sources(page: Page) -> List[Dict[str, Any]]:
    """Extract source citations from ChatGPT response."""

    try:
        # Click sources button if available
        sources_button = page.locator("button.group\\/footnote")
        if await sources_button.count() > 0:
            await sources_button.first.click()

            # Wait for modal
            modal = page.locator('[data-testid="screen-threadFlyOut"]')
            await modal.wait_for(state="visible", timeout=2000)

            # Extract links from modal
            links = modal.locator("a")
            link_count = await links.count()

            sources = []
            for i in range(link_count):
                link = links.nth(i)
                url = await link.get_attribute('href')
                text = await link.text_content()

                if url:
                    sources.append({
                        'url': url,
                        'title': text.strip() if text else '',
                        'position': i + 1
                    })

            return sources

    except Exception as e:
        print(f"Source extraction failed: {e}")
        return []

Shopping card extraction

Shopping cards are the reason many teams scrape ChatGPT in the first place. The offers, prices, and merchant list all sit inside a shopping_card event.

def extract_shopping_cards(events: List[Dict]) -> List[Dict[str, Any]]:
    """Extract product/shopping information from response."""

    shopping_cards = []

    for event in events:
        if 'shopping_card' in event:
            card_data = event['shopping_card']

            # Parse product information
            products = []
            for product in card_data.get('products', []):
                product_info = {
                    'title': product.get('title'),
                    'url': product.get('url'),
                    'price': product.get('price'),
                    'rating': product.get('rating'),
                    'num_reviews': product.get('num_reviews'),
                    'image_urls': product.get('image_urls', []),
                    'offers': []
                }

                # Parse merchant offers
                for offer in product.get('offers', []):
                    product_info['offers'].append({
                        'merchant_name': offer.get('merchant_name'),
                        'price': offer.get('price'),
                        'url': offer.get('url'),
                        'available': offer.get('available', True)
                    })

                products.append(product_info)

            shopping_cards.append({
                'tags': card_data.get('tags', []),
                'products': products
            })

    return shopping_cards

Entity extraction

The last field worth pulling when you scrape ChatGPT is the brand-entity tag. It tells you which brands ChatGPT associated with the answer, and with what confidence.

def extract_entities(events: List[Dict]) -> List[Dict[str, Any]]:
    """Extract named entities from ChatGPT response."""

    entities = []

    for event in events:
        if 'entities' in event:
            for entity in event['entities']:
                entities.append({
                    'type': entity.get('type'),
                    'name': entity.get('name'),
                    'confidence': entity.get('confidence'),
                    'context': entity.get('context')
                })

    return entities

Using cloro’s managed ChatGPT scraper

The hard part of running a scrape-ChatGPT stack in-house is not the scraper code. It is keeping up with OpenAI. Three surfaces drift constantly.

The backend-api/f/conversation URL has changed twice in 2025. Cloudflare’s fingerprint check is updated on a rolling basis. The shopping-card event shape moved from a flat products array to a nested offers block last quarter. Every change is a Slack ping at 11pm and a hot-fix the next morning.

cloro’s /v1/monitor/chatgpt endpoint absorbs that maintenance. You call one stable URL and let the managed ChatGPT scraper API keep pace with OpenAI’s changes.

Simple API integration

The fastest way to scrape ChatGPT is to skip the browser entirely and call one endpoint.

import requests
import json

# Your prompt
prompt = "Compare the top 3 programming languages for web development in 2026"

# API request to cloro
response = requests.post(
    'https://api.cloro.dev/v1/monitor/chatgpt',
    headers={'Authorization': 'Bearer YOUR_API_KEY'},
    json={
        'prompt': prompt,
        'country': 'US',
        'include': {
            'markdown': True,
            'rawResponse': True,
            'searchQueries': True
        }
    }
)

result = response.json()
print(json.dumps(result, indent=2))

What the endpoint handles for you

Every item below is a maintenance burden you would otherwise carry when you scrape ChatGPT yourself. The managed endpoint owns all of it.

• Cloudflare fingerprint rotation. Stealth-patched browsers tracked against the live cf_clearance challenge.
• backend-api/f/conversation URL pinning. When OpenAI rotates the path, we ship the fix; you keep calling the same cloro endpoint.
• SSE chunk reassembly. chatcmpl- chunks parsed and stitched into one response object.
• Citation flyout extraction. The lazy-loaded sources modal parsed into a positional sources array.
• Shopping-card normalization. The event shape is normalized — products[].offers[] is stable across OpenAI’s internal refactors.
• Entity + search-query metadata. Extracted from the SSE stream and exposed as first-class fields.

Structured output you get:

{
  "status": "success",
  "result": {
    "model": "gpt-5-mini",
    "text": "When comparing programming languages for web development in 2026...",
    "markdown": "**When comparing programming languages for web development in 2026**...",
    "sources": [
      {
        "position": 1,
        "url": "https://developer.mozilla.org/en-US/docs/Learn",
        "label": "MDN Web Docs",
        "description": "Comprehensive web development documentation"
      }
    ],
    "shoppingCards": [
      {
        "tags": ["programming", "education"],
        "products": [
          {
            "title": "Python Crash Course",
            "price": "$39.99",
            "rating": 4.8,
            "offers": [...]
          }
        ]
      }
    ],
    "searchQueries": ["web development languages 2026", "popular programming frameworks"],
    "rawResponse": [...]
  }
}

Why teams pick cloro to scrape ChatGPT

• Cloudflare passes that survive across prompts. Most DIY stacks burn a cf_clearance per call; cloro’s session pool reuses one across an entire monitoring batch.
• First-class shopping-card output. Other vendors return raw HTML; cloro returns the normalized offers[].merchant shape that survives OpenAI’s internal refactors.
• Citation expansion on by default. No flyout-clicking ceremony — the modal is parsed automatically into a positional sources array.
• Entity sidebar exposed. The brand-entity events that other scrapers drop on the floor are first-class output fields.

Building a scrape-ChatGPT stack in-house typically runs $5,000–10,000/month. That figure accounts for the Cloudflare fingerprint-rotation work, the backend-api URL maintenance, the residential proxy spend, and the on-call rotation.

If you still want to scrape ChatGPT with a custom setup, the protocol map above is the starting point. Expect ongoing work as OpenAI ships UI updates. The SSE chunk shape has changed twice this year, and the shopping-card event format has moved once. That churn is the real cost of choosing to scrape ChatGPT yourself.

Get started with cloro’s ChatGPT API and skip the maintenance.