OpenAI SDK Architecture

This guide explains how MCPJam Inspector implements the OpenAI Apps SDK to render custom UI components for MCP tool results. This enables MCP server developers to create rich, interactive visualizations for their tool outputs.

MCPJam Inspector also supports MCP-UI for simpler component-based UIs. See the Playground Architecture docs for MCP-UI implementation details.

Overview

MCPJam Inspector provides full support for the OpenAI Apps SDK, allowing MCP tools to return custom UI components that render in iframes with a sandboxed window.openai API bridge.

This document covers the V1 Playground implementation. As of PR #773, Playground V2 (ChatTabV2.tsx) also supports OpenAI Apps with a streamlined implementation using openai-app-renderer.tsx. The V2 implementation uses MCP resources API to fetch widget templates and renders them with similar window.openai bridge capabilities.

Key Features

Custom UI Rendering: Display tool results using custom HTML/React components
Interactive Widgets: Components can call other MCP tools and send followup messages
State Persistence: Widget state persists across sessions via localStorage
Theme Synchronization: Widgets automatically receive theme updates (light/dark mode)
Secure Isolation: Components run in sandboxed iframes with CSP headers
Server-Side Storage: Widget context stored server-side with 1-hour TTL for iframe access
Modal Support: Widgets can open modal dialogs with separate view contexts
Dual Mode Support:
- ui:// URIs for server-provided HTML content
- External URLs for remotely hosted components

OpenAI Apps SDK vs MCP-UI

MCPJam Inspector supports two approaches for custom UI rendering:

Feature	OpenAI Apps SDK	MCP-UI
Specification	OpenAI proprietary	MCP-UI (open standard)
Rendering	Sandboxed iframes	RemoteDOM components
Complexity	Full web applications	Simple interactive components
Tool calls	`window.openai.callTool()`	Action handlers
State persistence	`window.openai.setWidgetState()`	Not supported
Security	Full iframe sandbox	Component-level isolation
Best for	Complex dashboards, charts, forms	Buttons, cards, simple layouts

When to use OpenAI Apps SDK:

Need full JavaScript framework support (React, Vue, etc.)
Require persistent state across sessions
Building complex interactive visualizations
Need access to external APIs and libraries

When to use MCP-UI:

Simple interactive components (buttons, cards)
Prefer open standards over proprietary APIs
Don’t need state persistence
Want faster rendering without iframe overhead

See Playground Architecture - MCP-UI Integration for MCP-UI implementation details.

Architecture Overview

Component Flow

1. Tool Execution & Detection

When a tool is executed that returns OpenAI SDK components, the system detects this in two ways: Method A: _meta["openai/outputTemplate"] field

// Tool definition includes OpenAI output template
{
  "name": "weather_widget",
  "description": "Get weather with custom UI",
  "_meta": {
    "openai/outputTemplate": "ui://weather/display"
  }
}

Method B: ui:// resource in response

// Tool result contains ui:// resource
{
  "content": [{
    "type": "resource",
    "resource": {
      "uri": "ui://weather/display",
      "mimeType": "text/html"
    }
  }]
}

ResultsPanel Detection Logic

Located in client/src/components/tools/ResultsPanel.tsx:100-104:

// Check for OpenAI component using tool metadata from definition
const openaiOutputTemplate = toolMeta?.[\"openai/outputTemplate\"];
const hasOpenAIComponent =
  openaiOutputTemplate && typeof openaiOutputTemplate === \"string\";
const uiResource = resolveUIResource(result);

The resolveUIResource function searches for ui:// URIs in:

Direct resource field at root level
content array items with type: "resource"

Before rendering, widget data must be stored server-side for iframe access: Why Store Server-Side?

Iframes need access to toolInput and toolOutput for window.openai API
Client localStorage can’t be shared across iframe sandbox boundaries
Server becomes the source of truth for widget initialization data

Storage Implementation

Located in server/routes/mcp/openai.ts:14-32:

interface WidgetData {
  serverId: string;
  uri: string;
  toolInput: Record<string, any>;
  toolOutput: any;
  toolResponseMetadata?: Record<string, any> | null;
  toolId: string;
  toolName: string;
  theme?: "light" | "dark";
  timestamp: number;
}

const widgetDataStore = new Map<string, WidgetData>();

// Cleanup expired widget data every 5 minutes
setInterval(
  () => {
    const now = Date.now();
    const ONE_HOUR = 60 * 60 * 1000;
    for (const [toolId, data] of widgetDataStore.entries()) {
      if (now - data.timestamp > ONE_HOUR) {
        widgetDataStore.delete(toolId);
      }
    }
  },
  5 * 60 * 1000,
).unref();

The system uses a clever two-stage loading process to ensure React Router compatibility: Why Two Stages?

Many widgets use React Router’s BrowserRouter which expects clean URLs
Stage 1 changes URL to ”/” before widget code loads
Stage 2 fetches actual content after URL is reset
Prevents routing conflicts and 404 errors

Stage 1: Container Page

Located in server/routes/mcp/openai.ts:81-120:

openai.get(\"/widget/:toolId\", async (c) => {
  const toolId = c.req.param(\"toolId\");
  const widgetData = widgetDataStore.get(toolId);

  if (!widgetData) {
    return c.html(\"Error: Widget data not found or expired\", 404);
  }

  return c.html(`
    <!DOCTYPE html>
    <html>
    <head>
      <meta charset=\"utf-8\">
      <title>Loading Widget...</title>
    </head>
    <body>
      <script>
        (async function() {
          // Change URL to \"/\" BEFORE loading widget
          history.replaceState(null, '', '/');

          // Fetch actual widget HTML
          const response = await fetch('/api/mcp/openai/widget-content/${toolId}');
          const html = await response.text();

          // Replace entire document
          document.open();
          document.write(html);
          document.close();
        })();
      </script>
    </body>
    </html>
  `);
});

Stage 2: Content Injection

Located in server/routes/mcp/openai.ts:123-459: Key steps:

Retrieve widget data from store
Read HTML from MCP server via readResource(uri)
Parse view mode and params from query string (view_mode, view_params)
Inject window.openai API script with bridge implementation
Add security headers (CSP, X-Frame-Options)
Set cache control headers (no-cache for fresh content)

4. window.openai API Bridge

The injected script provides the OpenAI Apps SDK API to widget code:

API Implementation

Located in server/routes/mcp/openai.ts:213-376: Core API Methods:

const openaiAPI = {
  toolInput: ${JSON.stringify(toolInput)},
  toolOutput: ${JSON.stringify(toolOutput)},
  toolResponseMetadata: ${JSON.stringify(toolResponseMetadata ?? null)},
  displayMode: 'inline',
  maxHeight: 600,
  theme: ${JSON.stringify(theme ?? "dark")},
  locale: 'en-US',
  safeArea: { insets: { top: 0, bottom: 0, left: 0, right: 0 } },
  userAgent: {
    device: { type: 'desktop' },
    capabilities: { hover: true, touch: false }
  },
  view: {
    mode: ${JSON.stringify(viewMode)},  // 'inline' or 'modal'
    params: ${serializeForInlineScript(viewParams)}
  },
  widgetState: null,

  // Persist widget state (localStorage + propagate to model)
  async setWidgetState(state) {
    this.widgetState = state;
    try {
      localStorage.setItem(widgetStateKey, JSON.stringify(state));
    } catch (err) {
      console.error('[OpenAI Widget] Failed to save widget state:', err);
    }
    // Propagate state to parent for model context
    window.parent.postMessage({
      type: 'openai:setWidgetState',
      toolId: toolId,
      state
    }, '*');
  },

  // Call another MCP tool
  async callTool(toolName, params = {}) {
    return new Promise((resolve, reject) => {
      const requestId = `tool_${Date.now()}_${Math.random()}`;

      const handler = (event) => {
        if (event.data.type === 'openai:callTool:response' &&
            event.data.requestId === requestId) {
          window.removeEventListener('message', handler);
          event.data.error ? reject(new Error(event.data.error))
                           : resolve(event.data.result);
        }
      };

      window.addEventListener('message', handler);
      window.parent.postMessage({
        type: 'openai:callTool',
        requestId,
        toolName,
        params
      }, '*');

      setTimeout(() => {
        window.removeEventListener('message', handler);
        reject(new Error('Tool call timeout'));
      }, 30000);
    });
  },

  // Send followup message to chat
  async sendFollowupTurn(message) {
    const payload = typeof message === 'string'
      ? { prompt: message }
      : message;
    window.parent.postMessage({
      type: 'openai:sendFollowup',
      message: payload.prompt || payload
    }, '*');
  },

  // Request display mode change
  async requestDisplayMode(options = {}) {
    const mode = options.mode || 'inline';
    const maxHeight = options.maxHeight;
    this.displayMode = mode;
    if (typeof maxHeight === 'number') {
      this.maxHeight = maxHeight;
    }
    window.parent.postMessage({
      type: 'openai:requestDisplayMode',
      mode,
      maxHeight
    }, '*');
    return { mode };
  },

  // Alias for compatibility
  async sendFollowUpMessage(args) {
    const prompt = typeof args === 'string' ? args : (args?.prompt || '');
    return this.sendFollowupTurn(prompt);
  },

  // Open external URL
  async openExternal(options) {
    const href = typeof options === 'string' ? options : options?.href;
    if (!href) {
      throw new Error('href is required for openExternal');
    }
    window.parent.postMessage({
      type: 'openai:openExternal',
      href
    }, '*');
    window.open(href, '_blank', 'noopener,noreferrer');
  },

  // Request modal dialog
  async requestModal(options) {
    window.parent.postMessage({
      type: 'openai:requestModal',
      title: options.title,
      params: options.params,
      anchor: options.anchor
    }, '*');
  }
};

// Make available globally
window.openai = openaiAPI;
window.webplus = openaiAPI; // Compatibility alias

Security Notes:

API is frozen with writable: false, configurable: false
30-second timeout on tool calls prevents hanging requests
Origin validation in parent ensures only iframe messages are processed

As of PR #931, widgets can request modal dialogs using window.openai.requestModal(). This enables widgets to display secondary views or detailed information in a separate modal context. Located in client/src/components/chat-v2/openai-app-renderer.tsx:380-388:

case "openai:requestModal": {
  setModalTitle(event.data.title || "Modal");
  setModalParams(event.data.params || {});
  setModalOpen(true);
  break;
}

When a widget calls window.openai.requestModal(), the parent:

Extracts modal title and params from the message
Opens a Dialog component
Mounts a new iframe with the same widget URL
Appends query params: ?view_mode=modal&view_params=<encoded_json>

View Mode Detection

Widgets receive view context via window.openai.view:

// In widget code
if (window.openai.view.mode === "modal") {
  // Render modal-specific UI
  const params = window.openai.view.params;
  renderModalView(params);
} else {
  // Render inline UI
  renderInlineView();
}

Modal and inline views share widget state automatically. When either view calls setWidgetState(), the state is propagated to the other view via openai:pushWidgetState messages. Located in client/src/components/chat-v2/openai-app-renderer.tsx:269-284:

// After state is set, push to other view
const targetWindow = isFromInline
  ? modalWindow
  : isFromModal
    ? inlineWindow
    : null;

if (targetWindow) {
  targetWindow.postMessage(
    {
      type: "openai:pushWidgetState",
      toolId: resolvedToolCallId,
      state: event.data.state,
    },
    "*",
  );
}

The receiving view updates its local state and dispatches a openai:widget_state event: Located in server/routes/mcp/openai.ts:410-428:

if (
  event.data.type === "openai:pushWidgetState" &&
  event.data.toolId === toolId
) {
  try {
    const nextState = event.data.state ?? null;
    window.openai.widgetState = nextState;

    // Update localStorage
    try {
      localStorage.setItem(widgetStateKey, JSON.stringify(nextState));
    } catch (err) {}

    // Dispatch event for widget to listen
    try {
      const stateEvent = new CustomEvent("openai:widget_state", {
        detail: { state: nextState },
      });
      window.dispatchEvent(stateEvent);
    } catch (err) {
      console.error(
        "[OpenAI Widget] Failed to dispatch widget state event:",
        err,
      );
    }
  } catch (err) {
    console.error("[OpenAI Widget] Failed to apply pushed widget state:", err);
  }
}

Example Usage:

// In inline widget
document.getElementById("details-btn").addEventListener("click", () => {
  window.openai.requestModal({
    title: "Detailed View",
    params: { itemId: 123, view: "details" },
  });
});

// Widget code handles both views
if (window.openai.view.mode === "modal") {
  const { itemId, view } = window.openai.view.params;
  renderDetailedView(itemId, view);
} else {
  renderSummaryView();
}

// State changes sync automatically
window.openai.setWidgetState({ selectedItem: 123 });
// Both inline and modal views receive the update

6. Display Mode Support

As of PR #927, widgets can request different display modes to optimize their presentation:

Inline (default) - Widget renders within the chat message flow with configurable height
Picture-in-Picture (PiP) - Widget floats at the top of the screen in a fixed overlay
Fullscreen - Widget expands to fill the entire viewport

Display Mode Implementation

The display mode system uses React state to track which widget (if any) is in PiP mode, and applies different CSS classes based on the current mode: State Management (client/src/components/chat-v2/thread.tsx:62-72):

const [pipWidgetId, setPipWidgetId] = useState<string | null>(null);

const handleRequestPip = (toolCallId: string) => {
  setPipWidgetId(toolCallId);
};

const handleExitPip = (toolCallId: string) => {
  if (pipWidgetId === toolCallId) {
    setPipWidgetId(null);
  }
};

Mode Detection (client/src/components/chat-v2/openai-app-renderer.tsx:440-442):

const isPip = displayMode === "pip" && pipWidgetId === resolvedToolCallId;
const isFullscreen = displayMode === "fullscreen";

CSS Classes (client/src/components/chat-v2/openai-app-renderer.tsx:444-476):

let containerClassName = "mt-3 space-y-2 relative group";

if (isFullscreen) {
  containerClassName = [
    "fixed",
    "inset-0",
    "z-50",
    "w-full",
    "h-full",
    "bg-background",
    "flex",
    "flex-col",
  ].join(" ");
} else if (isPip) {
  containerClassName = [
    "fixed",
    "top-4",
    "inset-x-0",
    "z-40",
    "w-full",
    "max-w-4xl",
    "mx-auto",
    "space-y-2",
    "bg-background/95",
    "backdrop-blur",
    "supports-[backdrop-filter]:bg-background/80",
    "shadow-xl",
    "border",
    "border-border/60",
    "rounded-xl",
    "p-3",
  ].join(" ");
}

Exit Button (client/src/components/chat-v2/openai-app-renderer.tsx:481-493):

{shouldShowExitButton && (
  <button
    onClick={() => {
      setDisplayMode("inline");
      onExitPip?.(resolvedToolCallId);
    }}
    className="absolute left-2 top-2 z-10 flex h-6 w-6 items-center justify-center rounded-md bg-background/80 hover:bg-background border border-border/50 text-muted-foreground hover:text-foreground transition-colors cursor-pointer"
    aria-label="Close PiP mode"
    title="Close PiP mode"
  >
    <X className="w-4 h-4" />
  </button>
)}

Key Features:

Single PiP Widget: Only one widget can be in PiP mode at a time. Requesting PiP on a different widget automatically exits the current PiP widget.
Automatic Inline Fallback: If a widget is in PiP mode but another widget becomes the active PiP, the first widget automatically returns to inline mode.
Z-Index Layering: Fullscreen widgets use z-50, PiP widgets use z-40, ensuring proper stacking order.
Transform Isolation: The chat container uses transform: translateZ(0) to create a new stacking context, preventing z-index conflicts.
Backdrop Blur: PiP widgets use backdrop blur for a modern floating effect with semi-transparent background.

Requesting Display Mode from Widgets

Widgets can request display mode changes using the window.openai.requestDisplayMode() API:

// Request Picture-in-Picture mode
await window.openai.requestDisplayMode({ mode: "pip" });

// Request Fullscreen mode
await window.openai.requestDisplayMode({ mode: "fullscreen" });

// Return to inline mode
await window.openai.requestDisplayMode({ mode: "inline" });

// Set inline mode with custom height
await window.openai.requestDisplayMode({
  mode: "inline",
  maxHeight: 800,
});

The parent component handles these requests and updates the widget’s display mode accordingly.

7. Parent-Side Message Handling

Located in client/src/components/chat-v2/openai-app-renderer.tsx:312-347:

useEffect(() => {
  const handleMessage = async (event: MessageEvent) => {
    // Accept messages from inline or modal iframe
    const inlineWindow = iframeRef.current?.contentWindow;
    const modalWindow = modalIframeRef.current?.contentWindow;
    const isFromInline = inlineWindow != null && event.source === inlineWindow;
    const isFromModal = modalWindow != null && event.source === modalWindow;

    if (!isFromInline && !isFromModal) {
      return;
    }

    switch (event.data.type) {
      case \"openai:setWidgetState\":
        localStorage.setItem(widgetStateKey, JSON.stringify(event.data.state));
        break;

      case \"openai:callTool\":
        if (onCallTool) {
          try {
            const result = await onCallTool(
              event.data.toolName,
              event.data.params || {}
            );
            const targetWindow = event.source as Window | null;
            targetWindow?.postMessage({
              type: \"openai:callTool:response\",
              requestId: event.data.requestId,
              result: result
            }, \"*\");
          } catch (err) {
            const targetWindow = event.source as Window | null;
            targetWindow?.postMessage({
              type: \"openai:callTool:response\",
              requestId: event.data.requestId,
              error: err instanceof Error ? err.message : \"Unknown error\"
            }, \"*\");
          }
        }
        break;

      case \"openai:sendFollowup\":
        if (onSendFollowup) {
          onSendFollowup(event.data.message);
        }
        break;

      case \"openai:requestDisplayMode\":
        const mode = event.data.mode;
        setDisplayMode(mode);
        if (mode === \"pip\") {
          onRequestPip?.(resolvedToolCallId);
        } else if (mode === \"inline\" || mode === \"fullscreen\") {
          if (pipWidgetId === resolvedToolCallId) {
            onExitPip?.(resolvedToolCallId);
          }
        }
        if (typeof event.data.maxHeight === \"number\") {
          setMaxHeight(event.data.maxHeight);
        }
        break;
    }
  };

  window.addEventListener(\"message\", handleMessage);
  return () => window.removeEventListener(\"message\", handleMessage);
}, [widgetUrl, onCallTool, onSendFollowup, pipWidgetId, onRequestPip, onExitPip, resolvedToolCallId]);

Display Mode Synchronization

The component automatically resets to inline mode if another widget takes over PiP mode: Located in client/src/components/chat-v2/openai-app-renderer.tsx:368-372:

useEffect(() => {
  if (displayMode === "pip" && pipWidgetId !== resolvedToolCallId) {
    setDisplayMode("inline");
  }
}, [displayMode, pipWidgetId, resolvedToolCallId]);

This ensures only one widget can be in PiP mode at a time, preventing overlapping floating widgets.

Theme Synchronization

The parent component automatically sends theme updates to widgets when the user changes between light and dark mode: Located in client/src/components/chat-v2/openai-app-renderer.tsx:293-307:

// Send theme updates to iframe when theme changes
useEffect(() => {
  if (!isReady || !iframeRef.current?.contentWindow) return;

  iframeRef.current.contentWindow.postMessage(
    {
      type: "openai:set_globals",
      globals: {
        theme: themeMode,
      },
    },
    "*",
  );
}, [themeMode, isReady]);

Widgets can listen for theme changes using the openai:set_globals event:

// In widget code
window.addEventListener("message", (event) => {
  if (event.data.type === "openai:set_globals") {
    const { theme } = event.data.globals;
    // Update widget UI based on theme
    document.body.classList.toggle("dark", theme === "dark");
  }
});

As of PR #891, widget state changes are now propagated to the LLM model as hidden assistant messages. This allows the AI to understand and reason about widget interactions. Implementation (client/src/components/chat-v2/openai-app-renderer.tsx:232-244):

case "openai:setWidgetState": {
  // Widget state is already persisted by the iframe script
  console.log("[OpenAI App] Widget state updated:", event.data.state);

  if (onWidgetStateChange && event.data.toolId === resolvedToolCallId) {
    const newState = event.data.state;
    const newStateStr = newState === null ? null : JSON.stringify(newState);

    // Dedupe: only propagate if state actually changed
    if (newStateStr !== previousWidgetStateRef.current) {
      previousWidgetStateRef.current = newStateStr;
      onWidgetStateChange(resolvedToolCallId, newState);
    }
  }
  break;
}

Chat Integration (client/src/components/ChatTabV2.tsx:281-326):

const handleWidgetStateChange = useCallback(
  (toolCallId: string, state: any) => {
    setMessages((prevMessages) => {
      const messageId = `widget-state-${toolCallId}`;

      // If state is null, remove the widget state message
      if (state === null) {
        return prevMessages.filter((msg) => msg.id !== messageId);
      }

      const stateText = `The state of widget ${toolCallId} is: ${JSON.stringify(state)}`;

      const existingIndex = prevMessages.findIndex(
        (msg) => msg.id === messageId,
      );

      if (existingIndex !== -1) {
        // Update existing state message if changed
        const existingMessage = prevMessages[existingIndex];
        const existingText =
          existingMessage.parts?.[0]?.type === "text"
            ? (existingMessage.parts[0] as any).text
            : null;
        if (existingText === stateText) {
          return prevMessages; // No change
        }

        const newMessages = [...prevMessages];
        newMessages[existingIndex] = {
          id: messageId,
          role: "assistant",
          parts: [{ type: "text", text: stateText }],
        };
        return newMessages;
      }

      // Add new state message
      return [
        ...prevMessages,
        {
          id: messageId,
          role: "assistant",
          parts: [{ type: "text", text: stateText }],
        },
      ];
    });
  },
  [setMessages],
);

UI Hiding (client/src/components/chat-v2/thread.tsx:99):

// Hide widget-state messages from UI (they're sent to model but not displayed)
if (message.id?.startsWith("widget-state-")) return null;

Key Features:

Hidden Messages: Widget state messages are prefixed with widget-state- and hidden from the UI
Deduplication: State changes are only propagated if the serialized state actually changed
Model Context: The LLM receives state updates as assistant messages, enabling it to reason about widget interactions
Null Handling: Setting state to null removes the state message entirely
Example: When a chart widget updates its selected date range, the model receives: "The state of widget tool_123 is: {\"startDate\":\"2024-01-01\",\"endDate\":\"2024-01-31\"}"

This enables powerful use cases like:

LLM understanding user interactions with widgets
Contextual follow-up questions based on widget state
Multi-turn conversations that reference widget selections
Debugging widget behavior through model awareness

Tool Execution Bridge

Located in client/src/components/ChatTab.tsx:181-207:

const handleCallTool = async (
  toolName: string,
  params: Record<string, any>
) => {
  const response = await fetch(\"/api/mcp/tools/execute\", {
    method: \"POST\",
    headers: { \"Content-Type\": \"application/json\" },
    body: JSON.stringify({
      toolName,
      parameters: params,
      // Pass serverId if only one server is connected
      ...(selectedConnectedNames.length === 1
        ? { serverId: selectedConnectedNames[0] }
        : {})
    })
  });

  const data = await response.json();
  return data.result;
};

Security Architecture

Content Security Policy

Located in server/routes/mcp/openai.ts:408-437:

const trustedCdns = [
  \"https://persistent.oaistatic.com\",
  \"https://*.oaistatic.com\",
  \"https://unpkg.com\",
  \"https://cdn.jsdelivr.net\",
  \"https://cdnjs.cloudflare.com\",
  \"https://cdn.skypack.dev\"
].join(\" \");

c.header(\"Content-Security-Policy\", [
  \"default-src 'self'\",
  `script-src 'self' 'unsafe-inline' 'unsafe-eval' ${trustedCdns}`,
  \"worker-src 'self' blob:\",
  \"child-src 'self' blob:\",
  `style-src 'self' 'unsafe-inline' ${trustedCdns}`,
  \"img-src 'self' data: https: blob:\",
  \"media-src 'self' data: https: blob:\",
  `font-src 'self' data: ${trustedCdns}`,
  \"connect-src 'self' https: wss: ws:\",
  \"frame-ancestors 'self'\"
].join(\"; \"));

Iframe Sandbox

Located in client/src/components/chat-v2/openai-app-renderer.tsx:518-530:

<iframe
  ref={iframeRef}
  src={widgetUrl}
  sandbox=\"allow-scripts allow-same-origin allow-forms allow-popups allow-popups-to-escape-sandbox\"
  title={`OpenAI Component: ${toolCall.name}`}
  allow=\"web-share\"
/>

Sandbox Permissions:

allow-scripts: Enable JavaScript execution
allow-same-origin: Allow localStorage access (required for state)
allow-forms: Support form submissions
allow-popups: Enable external link navigation
allow-popups-to-escape-sandbox: Allow popup windows to load normally

Security Trade-offs:

allow-same-origin + allow-scripts = Full JavaScript capabilities
Required for React Router and modern frameworks
Mitigated by CSP headers and origin validation
Widgets should be treated as semi-trusted code

Complete Data Flow Example

Let’s trace a complete interaction where a widget calls a tool:

Development Guide

Testing OpenAI SDK Widgets Locally

Create a test MCP server with OpenAI SDK support:

// server.ts
import { Server } from \"@modelcontextprotocol/sdk/server/index.js\";
import { StdioServerTransport } from \"@modelcontextprotocol/sdk/server/stdio.js\";

const server = new Server(
  { name: \"test-widget-server\", version: \"1.0.0\" },
  { capabilities: { tools: {}, resources: {} } }
);

// Define tool with OpenAI output template
server.setRequestHandler(ListToolsRequestSchema, async () => ({
  tools: [{
    name: \"hello_widget\",
    description: \"Test widget with custom UI\",
    inputSchema: {
      type: \"object\",
      properties: {
        name: { type: \"string\" }
      }
    },
    _meta: {
      \"openai/outputTemplate\": \"ui://hello/display\"
    }
  }]
}));

// Tool returns structured content
server.setRequestHandler(CallToolRequestSchema, async (request) => {
  if (request.params.name === \"hello_widget\") {
    return {
      content: [{
        type: \"resource\",
        resource: {
          uri: \"ui://hello/display\",
          mimeType: \"text/html\"
        }
      }],
      _meta: {
        structuredContent: {
          greeting: `Hello, ${request.params.arguments.name}!`,
          timestamp: new Date().toISOString()
        }
      }
    };
  }
});

// Serve widget HTML
server.setRequestHandler(ReadResourceRequestSchema, async (request) => {
  if (request.params.uri === \"ui://hello/display\") {
    return {
      contents: [{
        uri: \"ui://hello/display\",
        mimeType: \"text/html\",
        text: `
<!DOCTYPE html>
<html>
<head>
  <meta charset=\"UTF-8\">
  <title>Hello Widget</title>
  <style>
    body { font-family: system-ui; padding: 20px; }
    button { padding: 10px 20px; margin: 10px 0; }
  </style>
</head>
<body>
  <div id=\"app\"></div>
  <script>
    // Access tool input/output
    const input = window.openai.toolInput;
    const output = window.openai.toolOutput;

    // Render UI
    document.getElementById('app').innerHTML = \`
      <h1>\${output.greeting}</h1>
      <p>Time: \${output.timestamp}</p>
      <button id=\"refresh\">Refresh</button>
      <button id=\"followup\">Send Followup</button>
    \`;

    // Call tool on button click
    document.getElementById('refresh').addEventListener('click', async () => {
      const result = await window.openai.callTool('hello_widget', {
        name: input.name + ' (refreshed)'
      });
      console.log('Tool result:', result);
    });

    // Send followup message
    document.getElementById('followup').addEventListener('click', () => {
      window.openai.sendFollowupTurn('Tell me more about greetings');
    });
  </script>
</body>
</html>
        `
      }]
    };
  }
});

const transport = new StdioServerTransport();
server.connect(transport);

Add server to MCPJam Inspector config:

{
  \"mcpServers\": {
    \"test-widget\": {
      \"command\": \"node\",
      \"args\": [\"path/to/server.ts\"]
    }
  }
}

Test in Inspector:
- Connect to server in Servers tab
- Navigate to Chat tab
- Execute: “Call the hello_widget tool with name John”
- Widget should render with interactive buttons

Common Problems:

Widget doesn’t load (404)
- Check that widgetDataStore contains toolId
- Verify storage TTL hasn’t expired (1 hour default)
- Confirm MCP server returns valid HTML for ui:// resource
window.openai is undefined
- Verify script injection in Stage 2 content endpoint
- Check browser console for CSP violations
- Ensure <head> tag exists in HTML for injection
Widget data not found (404)
- Check that widget data was successfully stored via POST /api/mcp/openai/widget/store
- Verify toolId matches between store and load requests
- Check server logs for storage errors
Tool calls timeout
- Check network tab for /api/mcp/tools/execute failures
- Verify MCP server is connected and responsive
- Increase timeout in callTool implementation (default: 30s)
React Router 404 errors
- Confirm Stage 1 executes history.replaceState('/') before loading
- Check that widget uses BrowserRouter not HashRouter
- Verify <base href=\"/\"> is present in HTML
State doesn’t persist
- Check localStorage in browser DevTools
- Verify widgetStateKey format: openai-widget-state:${toolName}:${toolId}
- Confirm setWidgetState postMessage handler is working
Theme not updating
- Check that openai:set_globals messages are being sent from parent
- Verify widget has event listener for theme changes
- Inspect window.openai.theme value in widget console
Modal not opening
- Verify requestModal postMessage is being sent
- Check that Dialog component state is updating (modalOpen)
- Inspect modal iframe URL includes view_mode=modal query param
- Confirm window.openai.view.mode is 'modal' in modal iframe
State not syncing between inline and modal
- Check that both iframes are mounted and have valid contentWindow
- Verify openai:pushWidgetState messages are being sent
- Inspect localStorage for widget state key
- Confirm both views have openai:widget_state event listener

Debug Tools:

// Add to widget for debugging
window.addEventListener("message", (e) => {
  console.log("[Widget] Received message:", e.data);
});

// Monitor all postMessage calls
const originalPostMessage = window.parent.postMessage;
window.parent.postMessage = function (...args) {
  console.log("[Widget] Sending message:", args[0]);
  return originalPostMessage.apply(window.parent, args);
};

// Check openai API availability
console.log("openai API:", window.openai);
console.log("toolInput:", window.openai?.toolInput);
console.log("toolOutput:", window.openai?.toolOutput);

Extending the Implementation

Adding New OpenAI API Methods:

Update server-side injection script (server/routes/mcp/resources.ts:250-376)
Add postMessage handler in parent (client/src/components/chat/openai-component-renderer.tsx:118-196)
Update TypeScript types if needed

Example: Adding a custom method:

// 1. Server-side API injection (server/routes/mcp/openai.ts)
openaiAPI.customMethod = async function(options) {
  window.parent.postMessage({
    type: 'openai:customMethod',
    data: options
  }, '*');
};

// 2. Parent-side handler (client/src/components/chat-v2/openai-app-renderer.tsx)
case \"openai:customMethod\":
  handleCustomMethod(event.data.data);
  break;

Performance Considerations

TTL: 1 hour default, configurable in resources.ts:22
Cleanup: Runs every 5 minutes
Memory: Each widget stores ~1-10KB (toolInput + toolOutput)
Scale: 1000 concurrent widgets ≈ 10MB memory
Recommendation: For production, use Redis instead of Map

Iframe Rendering

Initial Load: 200-500ms (Stage 1 + Stage 2 + resource fetch)
Tool Calls: 100-300ms (postMessage + backend + MCP)
Optimization:
- Cache MCP resource reads (currently disabled with no-cache)
- Preload widget data before iframe creation
- Use service workers for offline support

postMessage Overhead

Latency: 5-15ms per message round-trip
Payload: JSON serialization for all data
Bottleneck: Large tool results (>1MB) slow down significantly
Mitigation: Use streaming or chunked responses for large data

Security Best Practices

Validate postMessage Origins:

if (event.source !== iframeRef.current?.contentWindow) return;

Sanitize Tool Parameters:

const params = JSON.parse(JSON.stringify(event.data.params)); // Deep clone
// Validate against tool schema before execution

Limit Widget Capabilities:
- Only expose necessary MCP tools to widgets
- Implement rate limiting on tool calls
- Restrict network access via CSP
Content Security Policy:
- Remove unsafe-eval if possible (breaks some frameworks)
- Whitelist only trusted CDNs
- Consider using nonces for inline scripts
Audit Widget Code:
- Widgets have semi-trusted status
- Review HTML content from MCP servers
- Scan for XSS vulnerabilities
- Monitor for suspicious postMessage patterns

client/src/components/tools/ResultsPanel.tsx - Detects OpenAI components
client/src/components/chat-v2/openai-app-renderer.tsx - Renders iframes, handles widget lifecycle, and manages display modes
client/src/components/chat-v2/thread.tsx - Manages PiP state across all widgets in the thread
client/src/components/ChatTabV2.tsx - Chat integration with transform isolation for z-index stacking
server/routes/mcp/openai.ts - Widget storage, serving, and OpenAI bridge injection
server/routes/mcp/index.ts - Mounts OpenAI routes at /openai
client/src/lib/mcp-tools-api.ts - Tool execution API

Resources

Contributing

When contributing to the OpenAI SDK integration:

Test with real MCP servers - Don’t just mock the API
Check security implications - All changes to iframe/postMessage code need review
Update this documentation - Keep architecture diagrams current
Add debug logging - Use console.log with [OpenAI Widget] prefix
Consider backwards compatibility - Existing widgets should continue working

For questions or issues, open a GitHub issue or join our Discord community.

Overview

Features

Troubleshooting

Contributing

Changelog

​OpenAI SDK Architecture

​Overview

​Key Features

​OpenAI Apps SDK vs MCP-UI

​Architecture Overview

​Component Flow

​1. Tool Execution & Detection

​ResultsPanel Detection Logic

​2. Widget Data Storage Flow

​Storage Implementation

​3. Two-Stage Widget Loading

​Stage 1: Container Page

​Stage 2: Content Injection

​4. window.openai API Bridge

​API Implementation

​5. Modal Support

​Modal Architecture

​Modal Request Flow

​View Mode Detection

​Widget State Synchronization

​6. Display Mode Support

​Display Mode Implementation

​Requesting Display Mode from Widgets

​7. Parent-Side Message Handling

​Display Mode Synchronization

​Theme Synchronization

​Widget State Propagation to Model

​Tool Execution Bridge

​Security Architecture

​Content Security Policy

​Iframe Sandbox

​Complete Data Flow Example

​Development Guide

​Testing OpenAI SDK Widgets Locally

​Debugging Widget Issues

​Extending the Implementation

​Performance Considerations

​Widget Data Storage

​Iframe Rendering

​postMessage Overhead

​Security Best Practices

​Related Files

​Resources

​Contributing

OpenAI SDK Architecture

Overview

Key Features

OpenAI Apps SDK vs MCP-UI

Architecture Overview

Component Flow

1. Tool Execution & Detection

ResultsPanel Detection Logic

2. Widget Data Storage Flow

Storage Implementation

3. Two-Stage Widget Loading

Stage 1: Container Page

Stage 2: Content Injection

4. window.openai API Bridge

API Implementation

5. Modal Support

Modal Architecture

Modal Request Flow

View Mode Detection

Widget State Synchronization

6. Display Mode Support

Display Mode Implementation

Requesting Display Mode from Widgets

7. Parent-Side Message Handling

Display Mode Synchronization

Theme Synchronization

Widget State Propagation to Model

Tool Execution Bridge

Security Architecture

Content Security Policy

Iframe Sandbox

Complete Data Flow Example

Development Guide

Testing OpenAI SDK Widgets Locally

Debugging Widget Issues

Extending the Implementation

Performance Considerations

Widget Data Storage

Iframe Rendering

postMessage Overhead

Security Best Practices

Related Files

Resources

Contributing