How to Scrape Copilot: WebSocket Events and Citations

Microsoft Copilot is grounded in Bing. It generates a Bing search query from a few words of your prompt and answers from live web results. But it does not expose that consumer answer through a simple search API. Microsoft retired the standalone Bing Search APIs on August 11, 2025.

The UI streams mixed WebSocket events instead. Those events carry answer text, citations, source panels, shopping cards, and maps.

That is why you cannot scrape Copilot with a plain HTTP client. To scrape Copilot, you solve a protocol problem. It is not just a browser-automation problem.

You preserve session state. You intercept live frames. You reconstruct citations in the right order.

This guide explains the WebSocket architecture and the parsing workflow. It also shows when a managed Copilot API is the better way to scrape Copilot at scale. Pair it with AI search tracking if you monitor brand visibility across engines.

Why scrape Microsoft Copilot responses?

The reason to scrape Copilot is its Microsoft-ecosystem citation bias. Run the same prompt across ChatGPT, Perplexity, and Copilot. Copilot consistently surfaces docs.microsoft.com, learn.microsoft.com, and support.microsoft.com as primary citations. That pattern holds on enterprise IT and developer-tooling queries.

Say your brand monitoring covers Microsoft 365, Azure, or Power Platform. Copilot is then the engine where your docs should be cited. The only way to verify that is the WebSocket frame stream.

The two Copilots ground differently. The enterprise, work-grounded Copilot retrieves its answer data from the Microsoft Graph and a semantic index. The consumer Copilot at copilot.microsoft.com grounds each answer in Bing web results.

This guide targets that consumer surface. It is the one you scrape Copilot data from. In the consumer UI a sources button reveals the exact query Copilot sent to Bing and the sites it used. But that panel is rendered client-side. Only the frame stream carries it in machine-readable form.

What only the WebSocket frames expose

• Inline citation pills — Copilot groups consecutive citation events between text into pills. Web-grounded Copilot answers list hyperlinked citations below the text so users can click through to the source. The pill grouping is what orders the [1][2] markers. The official Bing API never exposes that structure.
• Microsoft-source attribution rate — citation.url lets you compute what share of citations point at *.microsoft.com domains. That share is the cleanest signal of how well your docs cover a query.
• Citation density — the citations-per-text-event ratio is a proxy for how grounded a Copilot answer is. It is useful for content strategy on technical topics.
• Mode-specific behavior — Copilot exposes distinct Quick response and Think Deeper conversation modes. Think Deeper takes longer and streams its chain of thought while it answers. Each mode emits different event volumes and citation patterns over the WebSocket.

Use cases you can scrape Copilot for

• Microsoft-doc citation tracking. “Is our learn.microsoft.com PR-merged article showing up in Copilot’s pills for the target query yet?”
• Internal-vs-external citation balance. “What ratio of microsoft.com-vs-third-party citations does Copilot use for our product category?”
• Mode comparison. “Does Smart mode cite our docs at a higher rate than Quick mode?”
• Enterprise compliance auditing. “Capture the full Copilot answer + sources for every prompt our support agents send, with timestamps.”

For a similar guide on a different protocol, see how to scrape ChatGPT (SSE-based).

Understanding Copilot’s WebSocket architecture

Copilot streams answers over a real-time WebSocket channel. To scrape Copilot, you have to read that channel frame by frame. The chat traffic is a WebSocket, not an HTTP request. So you attach a frame listener instead of reading a response body.

Playwright’s WebSocket object fires a framereceived event each time the socket receives a frame. That callback is where every appendText, citation, and done payload lands.

Each frame is a small JSON object. There is no single response to await. You accumulate frames until the done frame tells you the answer is finished. This is the mental model to hold before you scrape Copilot in code.

The response generation flow

1. Page navigation. Load the Copilot web interface.
2. WebSocket connection. Intercept WebSocket messages from the chat endpoint.
3. Event collection. Capture all JSON events sent via WebSocket.
4. Response parsing. Process collected events once completion is detected.
5. Microsoft knowledge. Draws on Microsoft product and service data.

WebSocket event interception:

// Copilot sends events via WebSocket from:
// copilot.microsoft.com/c/api/chat

// Events are simple JSON objects:
{
  "event": "appendText",
  "text": "To improve team productivity..."
}

{
  "event": "citation",
  "title": "Microsoft 365 Documentation",
  "url": "https://docs.microsoft.com/..."
}

{
  "event": "done"
}

WebSocket event types to handle

Three event types carry the whole answer. Handle these and you can scrape Copilot end to end. Everything else on the socket is noise you can drop.

• appendText: text content chunks
• citation: source citations embedded inline
• done: completion marker

The appendText frames arrive in order. Concatenate their text fields to rebuild the answer. The citation frames slot between text frames and mark where a source belongs. The done frame is your only reliable signal that the stream has closed.

The WebSocket event parsing challenge

Copilot scraping comes down to two jobs. You parse real-time WebSocket events. Then you reconstruct a single structured response. Get the ordering wrong and the citations detach from the text they support.

WebSocket event stream:

# Raw Copilot WebSocket events example
{
  "event": "appendText",
  "text": "To improve team productivity using Microsoft 365"
}

{
  "event": "citation",
  "title": "Microsoft 365 Documentation",
  "url": "https://docs.microsoft.com/en-us/microsoft-365/"
}

{
  "event": "appendText",
  "text": ", I recommend implementing SharePoint for document collaboration"
}

{
  "event": "done"
}

Parsing challenges:

1. Real-time streaming. Content arrives via WebSocket events, not HTTP responses.
2. Mixed event types. Text and citation events are interleaved.
3. Citation pill grouping. Multiple citations can be grouped together.
4. Event ordering. Citation positions need to be tracked accurately.

Python WebSocket parsing implementation:

import json
from typing import List, Dict, Any

class CopilotWebSocketParser:
    def __init__(self):
        self.text_parts = []
        self.citation_pills: List[List[Dict[str, Any]]] = []
        self.current_pill: List[Dict[str, Any]] = []
        self.citation_position = 1
        self.last_event_was_citation = False
        self.is_complete = False

    def parse_websocket_events(self, events: List[Dict[str, Any]]) -> Dict[str, Any]:
        """
        Parse Copilot WebSocket events into structured response.
        """
        for event in events:
            event_type = event.get("event")

            # Collect text chunks
            if event_type == "appendText":
                text_chunk = event.get("text", "")
                self.text_parts.append(text_chunk)

                # If we were building a citation pill and now see text, save the pill
                if self.last_event_was_citation and self.current_pill:
                    self.citation_pills.append(self.current_pill)
                    self.current_pill = []

                self.last_event_was_citation = False

            # Collect citations
            elif event_type == "citation":
                citation_data = {
                    "position": self.citation_position,
                    "label": event.get("title", ""),
                    "url": event.get("url", ""),
                    "description": None
                }

                self.current_pill.append(citation_data)
                self.citation_position += 1
                self.last_event_was_citation = True

            # Check for completion
            elif event_type == "done":
                self.is_complete = True
                break

        # Don't forget to add the last citation pill if it exists
        if self.current_pill:
            self.citation_pills.append(self.current_pill)

        # Combine all text parts
        full_text = "".join(self.text_parts)

        # Flatten citation pills into unique sources
        sources = self.flatten_citation_pills()

        return {
            "text": full_text,
            "sources": sources,
            "is_complete": self.is_complete
        }

    def flatten_citation_pills(self) -> List[Dict[str, Any]]:
        """
        Flatten grouped citation pills into unique sources with corrected positions.
        """
        seen_urls = set()
        sources: List[Dict[str, Any]] = []
        position = 1

        for pill in self.citation_pills:
            for citation_data in pill:
                url = citation_data["url"]
                if url not in seen_urls:
                    seen_urls.add(url)
                    sources.append({
                        "position": position,
                        "label": citation_data["label"],
                        "url": url,
                        "description": citation_data["description"]
                    })
                    position += 1

        return sources

Citation pill grouping logic:

def group_consecutive_citations(events: List[Dict[str, Any]]) -> List[List[Dict[str, Any]]]:
    """
    Group consecutive citation events into citation pills.
    Copilot groups multiple citations that appear together.
    """
    citation_pills = []
    current_pill = []
    last_was_citation = False

    for event in events:
        if event.get("event") == "citation":
            citation_data = {
                "position": len(current_pill) + 1,
                "label": event.get("title", ""),
                "url": event.get("url", ""),
                "description": None
            }
            current_pill.append(citation_data)
            last_was_citation = True
        elif event.get("event") == "appendText" and last_was_citation:
            # Text event after citations means the pill is complete
            if current_pill:
                citation_pills.append(current_pill)
                current_pill = []
            last_was_citation = False

    # Add the final pill if it exists
    if current_pill:
        citation_pills.append(current_pill)

    return citation_pills

Building the scraping infrastructure

Copilot’s stack is what happens when an SSE-trained scraper team meets a WebSocket-only product. None of the SSE primitives carry over. Chunk parsers, data: line splitters, and [DONE] sentinel detectors are all useless here.

The component list below is what you actually need to scrape Copilot reliably. Each item targets Copilot’s WebSocket plus its Microsoft-auth defenses.

Components you need to scrape Copilot

1. page.on("websocket") interceptor with frame handler. Playwright’s page.on("response") does nothing here — Copilot’s chat traffic is exclusively WebSocket. You hook the framereceived event and parse each frame as JSON.
2. Cookie stash keyed on (proxy_ip, copilot.microsoft.com). Microsoft drops the session cookie within minutes on a fresh IP. Stashing the cookie set per IP lets the next request on that IP skip the auth dance entirely.
3. Citation-pill grouping state machine. The rule is simple. Consecutive citation events form one pill, terminated by the next appendText or done. Without this state machine, sources come out as a flat list. The [1][2] inline markers in the answer then stop lining up.
4. Mode-selection keyboard navigation. A conversation mode is chosen before you submit the prompt. Copilot’s UI does not expose a stable mode-selector selector. The team uses keyboard shortcuts (Tab, Tab, Enter) to land on Quick or Smart mode. Selector-based navigation breaks within days. Keyboard-based navigation survives months.
5. event: "done" completion detector. SSE ships a [DONE] sentinel in-band. Copilot’s done event is a discrete JSON frame instead. Check for it inside the frame handler, not in the response body.

Complete scraper implementation:

import asyncio
import json
from playwright.async_api import async_playwright, Page
from typing import Dict, Any, List, Optional

class MicrosoftCopilotScraper:
    def __init__(self):
        self.copilot_events: List[Dict[str, Any]] = []
        self.received_done_event = False

    async def setup_websocket_interceptor(self, page: Page):
        """Set up WebSocket event interception."""

        def on_websocket(ws):
            # Intercept Copilot chat WebSocket
            if "copilot.microsoft.com/c/api/chat" in ws.url:
                ws.on("framereceived", self.websocket_message_handler)

        page.on("websocket", on_websocket)

    def websocket_message_handler(self, message: Union[str, bytes]):
        """Handle incoming WebSocket messages."""
        parsed = json.loads(message)

        if parsed.get("event") == "done":
            self.received_done_event = True

        self.copilot_events.append(parsed)

    async def scrape_copilot(self, query: str, country: str = 'US') -> Dict[str, Any]:
        """Main scraping function."""

        async with async_playwright() as p:
            browser = await p.chromium.launch(headless=False)
            context = await browser.new_context()
            page = await context.new_page()

            # Set up WebSocket interception
            await self.setup_websocket_interceptor(page)

            try:
                # Navigate to Copilot
                await page.goto('https://copilot.microsoft.com/', timeout=20_000)

                # Handle landing page and mode selection
                await self.handle_copilot_landing(page)

                # Fill and submit query
                await page.wait_for_selector("#userInput", state="visible", timeout=10_000)
                await page.fill("#userInput", query)
                await page.keyboard.press("Enter")

                # Wait for response completion
                await self.wait_for_copilot_response(page)

                # Parse the captured events
                parser = CopilotWebSocketParser()
                result = parser.parse_websocket_events(self.copilot_events)

                # Extract additional data if needed
                if result.get("text"):
                    # Get HTML content for markdown conversion if needed
                    html_content = await self.extract_html_content(page)
                    result["html_content"] = html_content

                return result

            finally:
                await browser.close()

    async def handle_copilot_landing(self, page: Page):
        """Handle Copilot landing page and mode selection."""

        # Wait for mode selection buttons
        await page.wait_for_selector(
            "[data-testid='composer-chat-mode-quick-button'], [data-testid='composer-chat-mode-smart-button']",
            timeout=5_000
        )

        # Navigate to chat mode using keyboard shortcuts (matching actual code)
        for _ in range(2):
            await page.keyboard.press("Tab")
            await asyncio.sleep(0.1)
        await page.keyboard.press("Enter")
        await asyncio.sleep(1)

        # Additional navigation to input field
        for _ in range(5):
            await page.keyboard.press("Tab")
            await asyncio.sleep(0.1)
        await page.keyboard.press("Enter")
        await asyncio.sleep(0.5)

    async def wait_for_copilot_response(self, page: Page, timeout: int = 60):
        """Wait for Copilot response completion."""

        for _ in range(timeout * 2):  # Check every 500ms
            await self.solve_captcha_if_needed(page)

            # If response got captured, we can return
            if self.received_done_event:
                break

            await asyncio.sleep(0.5)
        else:
            raise Exception("Never received Copilot response after 60 seconds")

    async def solve_captcha_if_needed(self, page: Page):
        """Handle captcha challenges if encountered."""
        # Simplified captcha handling
        try:
            # This would integrate with your captcha solving service
            pass
        except Exception:
            pass

    async def extract_html_content(self, page: Page) -> str:
        """Extract HTML content from Copilot response."""
        try:
            # Get the AI message content
            html_content = await page.locator(
                "[class*='group/ai-message-item']"
            ).first.inner_html(timeout=2_000)
            return html_content or ""
        except Exception:
            return ""

Cookie management for session persistence:

# Simple cookie management based on actual implementation
from typing import List, Dict

class CookieStash:
    """Manage cookies for persistent sessions across scrapes."""

    def __init__(self):
        self.cookies_cache = {}

    async def save_cookies(self, proxy_ip: str, domain: str, cookies: List[Dict]):
        """Save cookies for reuse."""
        cache_key = f"{proxy_ip}:{domain}"
        self.cookies_cache[cache_key] = cookies

    async def get_cookies(self, proxy_ip: str, domain: str) -> Optional[List[Dict]]:
        """Retrieve cached cookies."""
        cache_key = f"{proxy_ip}:{domain}"
        return self.cookies_cache.get(cache_key)

# Usage in scraper (matching actual code)
cookie_stash = CookieStash()

# Load existing cookies before navigation
existing_cookies = await cookie_stash.get_cookies(proxy.ip, "https://copilot.microsoft.com/")
if existing_cookies:
    try:
        await page.context.add_cookies(existing_cookies)
    except Exception as e:
        print(f"Failed to load cached cookies: {e}")

# Save cookies after successful session
cookies = await page.context.cookies()
await cookie_stash.save_cookies(proxy.ip, "https://copilot.microsoft.com/", cookies)

Parsing streaming text and citations

Raw frames are not the deliverable. Once you scrape Copilot frames, you still have to turn them into clean, linked text. Two conversions matter.

First you rebuild the answer as markdown with real anchor links. Then you score the citations for insight.

Converting the HTML answer to markdown

import html2text
from bs4 import BeautifulSoup
import re

def convert_html_to_markdown_with_links(
    html_content: str, citation_pills: List[List[Dict[str, Any]]]
) -> str:
    """
    Convert Copilot HTML to markdown, replacing citation buttons with proper links.
    """
    if not html_content:
        return ""

    # Parse HTML
    soup = BeautifulSoup(html_content, "html.parser")

    # Remove unwanted elements
    reactions_div = soup.find(attrs={"data-testid": "message-item-reactions"})
    if reactions_div:
        reactions_div.decompose()

    citation_cards = soup.find(attrs={"data-testid": "citation-cards-row"})
    if citation_cards:
        citation_cards.decompose()

    # Find all citation buttons (rounded-md class)
    buttons = soup.find_all("button", {"class": "rounded-md"})

    button_index = 0
    pill_index = 0

    # Replace each citation button with actual links
    while button_index < len(buttons) and pill_index < len(citation_pills):
        pill_links = citation_pills[pill_index]
        button = buttons[button_index]

        # Create anchor elements for each link in the pill
        new_anchors = []
        for link_data in pill_links:
            source_text = link_data.get("label")
            url = link_data.get("url")

            new_anchor = soup.new_tag("a", href=url)
            new_anchor.string = source_text
            new_anchors.append(new_anchor)

        # Insert all anchors after the button and remove the button
        for anchor in reversed(new_anchors):
            button.insert_after(anchor)
        button.decompose()

        button_index += 1
        pill_index += 1

    # Convert to markdown
    h = html2text.HTML2Text()
    h.ignore_links = False
    h.ignore_images = False
    h.body_width = 0
    h.unicode_snob = True
    h.skip_internal_links = False

    markdown = h.handle(str(soup))

    # Clean up whitespace
    markdown = re.sub(r"\n\s*\n\s*\n", "\n\n", markdown)
    markdown = markdown.replace("\\n\\n", "\n\n")
    markdown = markdown.replace("\\n", "\n")

    return markdown.strip()

Analyzing citation patterns

The parsed sources are also a dataset. Once you scrape Copilot at volume, you can measure which domains it favors.

The helper below counts total citations and computes citation density. It also splits Microsoft-owned sources from third-party ones. That split is the metric most brand teams actually track.

def analyze_citation_patterns(events: List[Dict[str, Any]]) -> Dict[str, Any]:
    """
    Analyze citation patterns in Copilot responses for insights.
    """
    citations = []
    text_events = []

    for event in events:
        if event.get("event") == "citation":
            citations.append({
                "title": event.get("title", ""),
                "url": event.get("url", ""),
                "position": len(citations) + 1
            })
        elif event.get("event") == "appendText":
            text_events.append(event.get("text", ""))

    return {
        "total_citations": len(citations),
        "citation_density": len(citations) / len(text_events) if text_events else 0,
        "average_text_between_citations": len("".join(text_events)) / len(citations) if citations else 0,
        "microsoft_sources": len([c for c in citations if "microsoft.com" in c.get("url", "")]),
        "external_sources": len([c for c in citations if "microsoft.com" not in c.get("url", "")])
    }

def extract_microsoft_knowledge_focus(text: str) -> Dict[str, Any]:
    """
    Analyze text to identify Microsoft ecosystem focus areas.
    """
    microsoft_products = [
        "Microsoft 365", "Office 365", "SharePoint", "Teams", "Outlook",
        "Azure", "Visual Studio", "Power Platform", "Power BI", "Power Apps",
        "Windows", "Active Directory", "Exchange", "OneDrive"
    ]

    product_mentions = {}
    for product in microsoft_products:
        count = text.lower().count(product.lower())
        if count > 0:
            product_mentions[product] = count

    return {
        "total_product_mentions": sum(product_mentions.values()),
        "mentioned_products": product_mentions,
        "has_microsoft_focus": len(product_mentions) > 0,
        "primary_products": sorted(product_mentions.items(), key=lambda x: x[1], reverse=True)[:3]
    }

Using cloro’s managed Copilot scraper

The hard part of running your own Copilot scraper is not the protocol. It is Microsoft session-cookie expiry. The appendText, citation, and done frame shapes are stable. The cookie window is measured in minutes and changes without notice.

cloro’s /v1/monitor/copilot endpoint is the managed way to scrape Copilot. It runs the cookie-stash and IP-rotation pool that keeps sessions alive across batches. Each response returns parsed sources and citation pills.

Microsoft ships a supported alternative too. Grounding with Bing Search for Azure AI Agents incorporates real-time public web data and returns citations as links to the websites used.

That is a fine building block for an agent. But it does not reproduce the consumer UI’s ordered [1][2] pill grouping. Only the WebSocket frame stream preserves that structure.

Calling the cloro Copilot API

import requests
import json

# Your Microsoft ecosystem query
query = "How can I improve team productivity using Microsoft 365 tools?"

# API request to cloro
response = requests.post(
    'https://api.cloro.dev/v1/monitor/copilot',
    headers={'Authorization': 'Bearer YOUR_API_KEY'},
    json={
        'prompt': query,
        'country': 'US',
        'include': {
            'markdown': True,
            'html': True
        }
    }
)

result = response.json()
print(json.dumps(result, indent=2))

What the endpoint handles for you

The endpoint absorbs the parts that make people give up on a DIY Copilot scraper. You send a prompt. You get back parsed text, ordered sources, and markdown. The list below is what it manages behind that single call.

• Cookie stash + IP rotation. Sessions are pinned to IP, refreshed before expiry, rotated on detection.
• Frame-handler completion detection. No timeout-based polling — the response returns the moment the done frame lands.
• Citation-pill grouping done correctly. The [1][2] inline markers in the answer body line up with positions in the sources[] array; no off-by-one drift when Copilot bunches three citations together.
• Microsoft-domain attribution rate. Returns a microsoft_source_pct field per response — useful for tracking ecosystem citation share over time.
• Mode-aware navigation. Quick vs Smart mode selectable via parameter; the keyboard-shortcut walk is handled internally.
• Markdown reconstruction with linked citations. The HTML+citation pill markup is converted into clean markdown with inline anchor links (not just [1][2] references) — ready to feed into downstream LLM analysis.

Structured output

The response is normalized JSON, ready for a database or an LLM prompt. Text, sources, and markdown all arrive in one object. The sources[] array is already de-duplicated and ordered to match the inline markers.

{
  "status": "success",
  "result": {
    "text": "To improve team productivity using Microsoft 365 tools, I recommend implementing the following strategies: utilize SharePoint for document collaboration, leverage Teams for communication, use Power Automate for workflow automation...",
    "sources": [
      {
        "position": 1,
        "url": "https://docs.microsoft.com/en-us/microsoft-365/",
        "label": "Microsoft 365 Documentation",
        "description": "Official documentation for Microsoft 365 productivity tools and features..."
      },
      {
        "position": 2,
        "url": "https://learn.microsoft.com/en-us/sharepoint/",
        "label": "SharePoint Documentation",
        "description": "Comprehensive guide to SharePoint for document management and collaboration..."
      }
    ],
    "markdown": "**To improve team productivity using Microsoft 365 tools**, I recommend implementing the following strategies...",
    "html": "https://storage.cloro.dev/results/c45a5081-808d-4ed3-9c86-e4baf16c8ab8/page-1.html"
  }
}

Why teams pick cloro to scrape Copilot

• Cookie-stash pool that survives Microsoft’s eviction. No mid-batch session breaks; the pool is sized to outlast the eviction window.
• WebSocket frame parsing battle-tested against Quick + Smart modes. Both modes’ citation patterns produce identical output shapes from the API.
• Mode-selection keyboard walk maintained centrally. Microsoft tweaks the UI ordering — that’s our integration test, not yours.
• Microsoft-source attribution rate as a first-class field. No need to post-process URLs to compute microsoft.com share — it’s returned per response.

Building this in-house typically runs $3,000–6,000/month. That covers residential proxies, a WebSocket fleet, and the cookie-stash service. It also covers the on-call rotation for when Microsoft’s session-eviction window shifts.

You can still build your own Copilot scraper, and the protocol map above is the starting point. Just plan for ongoing session-cookie maintenance.

Microsoft tightened the eviction window twice in 2025. The Quick and Smart mode buttons also swapped tab-order positions in the latest UI update. Both changes silently break a scraper that hard-codes selectors or timeouts.

Get started with cloro’s Copilot API and skip the cookie-stash maintenance.