## JOE AI Overview This document gives Custom GPTs, Assistants, and AI agents a single mental model for how AI works inside **Json Object Editor (JOE)**: which schemas are involved, how the `chatgpt` plugin behaves, how the `joe-ai` UI connects to OpenAI, and how **MCP** exposes JOE as tools. --- ## 1. High‑level AI Architecture - **Core idea**: JOE is a schema‑driven data system. AI features are layered on top to: - Generate or transform field values (AI autofill) - Run persistent assistants and chats (in‑app + widgets) - Expose JOE data and writes via MCP tools for external agents - **Main components**: - **AI schemas** (`ai_assistant`, `ai_prompt`, `ai_tool`, `ai_response`, `ai_conversation`, `ai_widget_conversation`) - **Server plugins**: - `chatgpt.js` – **central / preferred** entry point for OpenAI (Responses API, tool calls, MCP bridge). New integrations should target this plugin. - `chatgpt-assistants.js` – legacy Assistants‑API plugin, still used by existing in‑app chat flows (`ai_conversation` / ``) but not recommended for new work. - `chatgpt-tools.js` / `chatgpt-responses.js` – helpers for tools + response shaping - **Client / UI**: - `joe-ai.js` – in‑app AI chat UI (web components) wired to `ai_conversation` + assistants - Standard JOE editor UI with **AI autofill** buttons on fields, driven by `ai_prompt` - **MCP**: - `/.well-known/mcp/manifest.json` – describes tools - `/mcp` – JSON‑RPC endpoint whose tools route into `JOE.Schemas`, `JOE.Storage`, `JOE.Cache` When in doubt: **schemas define structure**, **plugins call OpenAI + MCP**, **MCP exposes tools**, and **UI (`joe-ai`) is just a thin chat/controls layer on top. ### 1.5 End‑to‑end mental model (for agents and humans) At a high level, almost every AI flow in JOE is one of these patterns: - **Prompt / Autofill flow (schema‑driven)** 1. A human or field‑level `ai` config chooses an `ai_prompt` (and optionally an `ai_assistant`). 2. The UI calls `/API/plugin/chatgpt/` (usually `executeJOEAiPrompt`) with `ai_prompt` id, parameters, and any referenced objects/files. 3. `chatgpt.js` loads the prompt, runs any helper `functions`, builds `instructions` + `input`, and normalizes MCP config (`mcp_*`) from the prompt and/or assistant. 4. `runWithTools` calls the OpenAI **Responses API**, optionally with tools (including MCP tools), then persists the result as an `ai_response` (plus any `generated_thoughts`, `used_openai_file_ids`, etc.). 5. Optional: helpers like `compareResponseToObject` / `updateObjectFromResponse` merge results back into a target object. - **Widget / AIHub / Object chat flow (conversation‑driven)** 1. The UI renders `` (optionally scoped to an object and seeded with an `ai_assistant_id`). 2. On the first message, the widget calls `/API/plugin/chatgpt/widgetStart`, which creates an `ai_widget_conversation` with model, assistant, system text, and optional `scope_itemtype`/`scope_id`. 3. Each user turn calls `/API/plugin/chatgpt/widgetMessage`, which: - Resolves the active `ai_assistant` by `_id` and loads its `instructions` + MCP config. - Builds `systemText` (assistant instructions + MCP tool list + optional scope hints / `understandObject` snapshot). - Gathers any attached files (from scoped objects or assistant‑level files) and passes their OpenAI ids/roles into `runWithTools`. 4. `runWithTools` performs a Responses+tools call (with MCP tools where enabled) and returns the assistant’s reply; `widgetMessage` appends it to `ai_widget_conversation.messages` and returns the updated history to the widget. - **MCP‑only agent flow (Custom GPT / external agent)** 1. The agent discovers tools and named toolsets via `/.well-known/mcp/manifest.json`. 2. For reads/writes it calls `POST /mcp` with tool names like `hydrate`, `search`, `fuzzySearch`, `getObject`, `saveObject`, and `saveObjects`. 3. When the agent needs higher‑level workflows (thoughts, prompts, assistants), it can call into the same HTTP APIs that `joe-ai` uses (e.g., `/API/plugin/chatgpt/autofill`, `/API/plugin/chatgpt/executeJOEAiPrompt`, widget endpoints), treating JOE as the authoritative orchestrator of OpenAI + MCP rather than re‑implementing orchestration itself. --- ## 2. AI Schemas (Mental Model) JOE uses a small set of AI‑specific schemas. A good agent should know what each represents and how they relate. ### 2.0 Modern assistant & conversation identity (2026+) - **Assistant identity**: - Treat `ai_assistant._id` (JOE cuid) as the **canonical id** for assistants. - `ai_assistant.assistant_id` (OpenAI Assistants id) is **optional/legacy** and only present when synced to the old Assistants API. - The default assistant used by widgets/AIHub is stored in `setting.DEFAULT_AI_ASSISTANT.value` as an `ai_assistant._id`. - **Conversations**: - `ai_widget_conversation` is the **primary** conversation schema for modern chats (``, AIHub cards, object chat). It links to assistants via `assistant` (JOE cuid) and may also store a legacy `assistant_id`. - `ai_conversation` is **legacy** and only used by older `` flows that still rely on the Assistants API. - **MCP configuration**: - MCP fields (`mcp_enabled`, `mcp_toolset`, `mcp_selected_tools`, `mcp_instructions_mode`) can exist both on `ai_assistant` (assistant‑level default) and `ai_prompt` / field‑level `ai` configs (surface‑level overrides). - The server (`chatgpt.js`) uses a shared helper to resolve these configs into actual toolsets and instructions for all AI surfaces (prompts, autofill, widget/object chat). ### 2.1 `ai_assistant` - **What it is**: Configuration record for a single AI assistant linked to OpenAI. - **Key fields** (from schema + summaries): - Identity & model: `name`, `info`, `ai_model`, `assistant_id`, `openai_assistant_version` - Capabilities: `file_search_enabled`, `code_interpreter_enabled` - Prompting: `instructions`, `assistant_thinking_text`, `assistant_color` - Tools: `tools` (JSON OpenAI tools array – often imported from MCP), `datasets`, `tags`, `status` - Sync meta: `last_synced`, timestamps (`created`, `joeUpdated`) - **How it’s used**: - One `ai_assistant` usually maps 1:1 to an OpenAI Assistant. - A **DEFAULT** can be set via `setting.DEFAULT_AI_ASSISTANT` and is used by `joe-ai` as the default assistant. - The schema exposes helper methods such as: - `syncAssistantToOpenAI` – sync this record to OpenAI via `chatgpt-assistants.js`. - `loadMCPTolsIntoAssistant` – pull tools from the MCP manifest into `assistant.tools`. - `setAsDefaultAssistant` – update the `DEFAULT_AI_ASSISTANT` setting. **Agent note**: When reasoning about which assistant is active in a chat or widget, look for: - `ai_conversation.assistant` or `ai_widget_conversation.assistant` - Or the instance‑level default via the `setting` schema. ### 2.2 `ai_prompt` - **What it is**: Reusable AI **prompt configuration** – how to call the `chatgpt` plugin or Responses API. - **Key fields**: - Identity: `_id`, `name`, `info`, `itemtype:'ai_prompt'` - Integration with plugin: - `prompt_method` – name of the server method on `chatgpt.js` to call (e.g. `executeJOEAiPrompt`). - `content_items` – `objectList` of `{ itemtype, reference }` describing which JOE objects to send and under what parameter name. - Instructions: - `functions` – helper code snippet (Node‑style `module.exports = async function(...) { ... }`) used to shape instructions/input. - `instructions_format` – format hint; `instructions` – main system‑level text. - `user_prompt` – optional user‑facing prompt template or per‑call input description. - OpenAI tuning: `ai_model`, `temperature` - Meta: `status`, `tags`, `datasets`, timestamps - **Methods / behavior**: - `methods.buildURL(prompt, items)` – builds URLs like: - `/API/plugin/chatgpt/?ai_prompt=&=...` - `methods.listExamples(prompt)` – picks sample content objects for exploration/testing. - **Typical flows**: - **AI Autofill (field‑level)**: - Fields in other schemas include an `ai` config (documented in `CHANGELOG` + README). - The UI calls `/API/plugin/chatgpt/autofill`, which: - Uses `ai_prompt` definitions and schema metadata. - Sends the right JOE objects + instructions to OpenAI. - Returns structured patch JSON to update fields. - **Explicit prompts**: - UI actions or external tools call `/API/plugin/chatgpt/?ai_prompt=&...`. - The plugin locates the prompt, merges helper `functions`, and runs OpenAI. **Agent note**: Treat `ai_prompt` as the **source of truth** for how to talk to OpenAI for a particular workflow (summaries, planning, refactors, etc). Never invent `prompt_method` names; reuse existing ones or ask a human to add a new prompt record. ### 2.3 `ai_response` - **What it is**: Persistent record of a single AI response, for **auditing**, **reuse**, and **merge‑back** into JOE. - **Key fields**: - Links: `ai_prompt` (reference), `referenced_objects` (array of JOE `_id`s used), `tags` - Request context: `user_prompt`, `prompt_method`, `model_used`, `response_type` - Response data: - `response` – raw string body (often JSON or text). - `response_keys` – array of keys returned (used for patch/merge). - `response_id` – OpenAI response ID. - `usage` – token usage object `{ input_tokens, output_tokens, total_tokens }`. - **Methods**: - `compareResponseToObject(response_id, object_id, do_alert)` – validate response JSON vs object fields, optionally trigger a merge. - `updateObjectFromResponse(response_id, object_id, fields)` – apply response values into a JOE object and update tags. - `listResponses(obj)` – list AI responses referencing a given object. **Agent note**: When you are updating objects via tools, favor: - Using `ai_response` records to compare/merge changes, especially if the response contains multiple fields. - Respect existing merge flows (`compareResponseToObject` / `updateObjectFromResponse`) instead of overwriting arbitrary fields. ### 2.4 `ai_tool` - **What it is**: Definition of a reusable **function‑calling tool** that OpenAI can call and JOE can execute server‑side. - **Key fields**: - Identity: `name`, `info`, `description`, `tool_id` - OpenAI schema: `tool_properties` (JSON, typically `{ type:'function', function:{ name, description, parameters } }`) - Execution: `server_code` – Node.js async code to run when the tool is invoked. - Meta: `datasets`, `tags`, timestamps. - **Methods**: - `updateToolProperties(propObj)` – helper to keep `tool_properties.function.name` in sync with `tool_id`. **Agent note**: For tools exposed via MCP, the canonical shape is actually in the **MCP manifest**; `ai_tool` is more for: - High‑level catalog and advanced OpenAI tools, not the core MCP schema/search/save tools. ### 2.5 `ai_conversation` - **What it is**: Metadata record for **in‑app AI conversations** (staff/admin chat in JOE). - **Key fields**: - Participants: `user` (JOE user), `assistant` (`ai_assistant`), `members` (external participants) - OpenAI thread: `thread_id` - Summary: `summary` (WYSIWYG), `status`, `tags` - Meta: `_id`, `name`, `info`, timestamps - **Behavior**: - Messages are **not stored** in JOE; they’re fetched from OpenAI at runtime (via `chatgpt-assistants.js`). - Schema methods + UI: - `methods.chatSpawner(object_id)` – calls `_joe.Ai.spawnContextualChat(object_id)`. - List view shows created time, user cubes, and context objects used for that chat. - `joe-ai.js` uses `ai_conversation` as the anchor object for chat history + context. **Agent note**: If you need to understand a chat history, you: - Read `ai_conversation` for metadata and context_objects. - Use **MCP tools** or existing chat APIs to fetch live thread content; don’t invent local persistence. ### 2.6 `ai_widget_conversation` - **What it is**: Lightweight conversation record for **embeddable AI widgets** (external sites, test pages). - **Key fields**: - Participant & assistant: `user`, `assistant`, `user_name`, `user_color`, `assistant_color` - OpenAI info: `model`, `assistant_id`, `system` (effective instructions) - Conversation: `messages` (compact list, typically JSON array of `{ role, content, created_at }`), `last_message_at` - Source: `source` (widget origin), `tags`, timestamps - **Behavior**: - Accessed/updated via `chatgpt-assistants` widget endpoints: - `widgetStart`, `widgetMessage`, `widgetHistory`, etc. - Schema exposes **subsets** and **filters** for grouping by `source`, `user`, and `assistant`. **Agent note**: Treat `ai_widget_conversation` as the external/chat‑widget equivalent of `ai_conversation`. The fields are tuned for widgets and public UI, but the mental model is the same: small record pointing to assistant + messages. --- ## 3. `chatgpt` Plugin and Related Plugins ### 3.0 Modern vs legacy stacks - **Modern stack (preferred)**: - Uses `server/plugins/chatgpt.js` with the **Responses API** and MCP tools. - Conversations live in `ai_widget_conversation` and are surfaced via `` and ``. - MCP configuration is respected from `ai_assistant` and `ai_prompt` (and field‑level `ai` configs). - **Legacy stack (Assistants API)**: - Uses `server/plugins/chatgpt-assistants.js` and OpenAI Assistants/threads. - Conversations live in `ai_conversation` and are surfaced via ``. - Kept for backward compatibility only; **new integrations should use the modern stack above**. ### 3.1 `server/plugins/chatgpt.js` This is the **core server plugin** that wires JOE to OpenAI and MCP. - **Responsibilities**: - Read OpenAI API key from `setting.OPENAI_API_KEY`. - Provide shared helpers for building OpenAI clients (`OpenAI` SDK). - Implement orchestration for: - **Responses API** calls (models like `gpt-4.1`, `gpt-4.1-mini`, `gpt-5.1`). - **Tool calling** and follow‑up calls via MCP tools. - Bridge between JOE and MCP with helpers like: - `callMCPTool(toolName, params, ctx)` – call MCP tools **in‑process** (without HTTP). - Response parsing helpers such as `extractToolCalls(response)`. - Payload shrinking helpers like `shrinkUnderstandObjectMessagesForTokens(...)`. - Expose specific routes under `/API/plugin/chatgpt/...`, including: - Field autofill (`autofill`) driven by schema‑level `ai` configs. - Prompt execution handlers for `ai_prompt.prompt_method` values. - **Key patterns**: - **Slimming payloads**: When embedding object graphs into messages (e.g. `understandObject` results), `chatgpt.js` converts them into a slim representation to avoid token limits. - **Error handling**: - `isTokenLimitError(err)` identifies token/TPM error shapes for better logs and fallbacks. **Agent note**: Any time you see endpoints in docs like `/API/plugin/chatgpt/`, they resolve into functions in `chatgpt.js` or its siblings. Follow those conventions instead of inventing new paths. ### 3.2 `chatgpt-assistants.js`, `chatgpt-responses.js`, `chatgpt-tools.js` - **`chatgpt-assistants.js`**: - Manages OpenAI Assistants and threads using the **older Assistants API**. - Endpoints such as: - `syncAssistantToOpenAI` (used from `ai_assistant.methods`). - `getThreadMessages`, `runAssistantChat`, widget endpoints like `widgetStart`, `widgetMessage`, `widgetHistory`. - Used by `joe-ai.js` and widget UIs to pull/push messages for a given conversation. - **Status**: legacy bridge for existing flows; **new integrations should use the Responses‑based `chatgpt` plugin instead**, matching the README. - **`chatgpt-responses.js`**: - Helpers for working with Responses API results: - Extracting text and tool calls. - Normalizing outputs for storage in `ai_response`. - Often called from `chatgpt.js` orchestration functions. - **`chatgpt-tools.js`**: - Glue between AI tools and JOE: - Uses `ai_tool` definitions. - Executes server‑side code blocks for tools when OpenAI calls them. - Can also support MCP‑style tools exported via the manifest. **Agent note**: You typically don’t call these plugins directly from a GPT. Instead, you: - Use documented **APIs** (`/API/plugin/chatgpt/*`, widget endpoints) and/or - Use **MCP tools** for reads/writes. --- ## 4. `joe-ai.js` – In‑App AI UI The `js/joe-ai.js` file implements JOE’s in‑app AI user interface using custom elements and an `Ai` namespace. - **Key responsibilities**: - Manage open chat boxes and default assistant selection: - `Ai._openChats`, `Ai.default_ai`, `Ai.getDefaultAssistant()`. - Render **modern widget‑based chat** bound to `ai_widget_conversation`: - `` – embeddable chat component that talks to `chatgpt.js` (`widgetStart`, `widgetMessage`, `widgetHistory`). - `` – selector that applies an `ai_assistant._id` to a widget (`ai_assistant_id` attribute) and starts fresh conversations as needed. - `Ai.openObjectChatLauncher(object_id, itemtype, name)` – launches a floating shell containing a `` for **object‑scoped chat**: - Sets `source:"object_chat"`, `scope_itemtype`, `scope_id`, and passes the current user id. - Uses `DEFAULT_AI_ASSISTANT` as the initial assistant, but allows per‑conversation overrides via the picker. - (Legacy) Render **Assistants‑API chat** bound to an `ai_conversation`: - `` – older component that loads `ai_conversation` and uses `chatgpt-assistants` endpoints (`getThreadMessages`, etc.). - Still present for existing flows but not used by the new object chat or AIHub widget experiences. - Provide ergonomic chat behaviors: - Markdown rendering via `renderMarkdownSafe` (prefers `marked` + `DOMPurify`). - Greeting messages based on `user` and `context_objects`. - Keyboard handling (Enter to send). - Conversation list (``) for selecting existing `ai_widget_conversation` records and applying them to a widget. - **High‑level flow** for modern widget chat: 1. UI renders a `` with optional `ai_assistant_id`, `source`, and scope attributes. 2. On the first user message, the widget calls `/API/plugin/chatgpt/widgetStart`, which creates an `ai_widget_conversation`. 3. Subsequent messages call `/API/plugin/chatgpt/widgetMessage`, which: - Resolves the active `ai_assistant` (by `_id`), builds `systemText` (assistant instructions + optional MCP tool list + scope hints), and attaches any relevant files. - Calls `runWithTools` in `chatgpt.js` so the model can use MCP tools and respond. - Updates `ai_widget_conversation.messages` and returns the full history to the widget. **Agent note**: - When describing UI behavior or helping with debugging, remember that `joe-ai.js` is **UI only**; persistent data lives in: - `ai_widget_conversation` (modern chats: AIHub, object chat, widgets) - `ai_assistant` (config, including MCP/tool settings and assistant‑level files) - `ai_conversation` (legacy Assistants‑API chats only) --- ## 5. MCP: How JOE Exposes Tools to Agents MCP (Model Context Protocol) is how external agents (including Custom GPTs) call into JOE as a **tool provider**. ### 5.1 Endpoints and Manifest - **Endpoints**: - `GET /.well-known/mcp/manifest.json` – public manifest with: - Instance info `{ joe: { name, version, hostname } }` - Tool list (name, description, params schema) - Privacy/terms URLs (also wired into `/privacy` and `/terms` routes). - `POST /mcp` – auth‑protected JSON‑RPC 2.0 endpoint for **tool calls**. - Same auth rules as other APIs; if users exist, requires cookie or basic auth. - **Implementation note**: - The concrete MCP tool implementations live in `server/modules/MCP.js`, which is wired up by `server/init.js` and exported via the manifest and `/mcp` endpoint. - **Tools** (from README + CHANGELOG): - `listSchemas(name?)`, `getSchema(name)` - `getObject(_id, itemtype?)` - `search` (unified search across cache/storage; supports `source`, `ids`, `flatten`, `depth`, `countOnly`) - `fuzzySearch` – typo‑tolerant search across weighted fields - `saveObject({ object })` - `saveObjects({ objects, stopOnError?, concurrency? })` - `hydrate` – returns core fields, schemas, statuses, tags (for agents to bootstrap context) All tools map directly into JOE’s APIs: `JOE.Schemas`, `JOE.Storage`, `JOE.Cache`, and related helpers. Sensitive fields are sanitized before returning data. ### 5.2 Agent Best Practices with MCP - **Schema‑first**: Always: 1. Discover schemas via `listSchemas`/`getSchema`. 2. Use `hydrate` on startup to cache schemas/statuses/tags. 3. Use `search` / `fuzzySearch` with explicit `itemtype` and filters rather than arbitrary text mining. - **Reads vs writes**: - Prefer **cache** for read operations (`search` default) unless you explicitly need DB‑level freshness (set `source:"storage"`). - For writes, use: - `saveObject` for single‑item updates. - `saveObjects` for batched updates with clear error handling. - **Performance**: - Use `countOnly` before large reads to avoid pulling massive datasets. - When embedding objects into prompts, prefer **slimmed** representations (id, itemtype, name, info). **Agent note**: The MCP tools are your primary way to **inspect and modify JOE data** from a Custom GPT or other agent host. Direct HTTP calls to JOE’s internal `/API/*` routes should be treated as implementation detail unless explicitly allowed in your instructions. --- ## 6. Recommended RAG / Context Files for JOE AI Agents When configuring a JOE‑aware Custom GPT or dev agent, include the following docs in its RAG/context: - **Core architecture & instance info** - `README.md` – especially: - “Architecture & Mental Model (Server)” - MCP overview + test instructions (including references to `_www/mcp-test.html`, `_www/mcp-schemas.html`, and `_www/plugins-test.html` for live inspection) - `CHANGELOG.md` – AI/MCP‑related entries (0.10.43x+, 0.10.50x+, 0.10.62x+), already summarize important AI behavior. - **Agent / instruction docs (existing)** - `docs/JOE_Master_Knowledge_Export.md` – master context dump for schemas and concepts. - `docs/joe_agent_spec_v_2.2.md` – agent behavior/specification. - `docs/joe_agent_custom_gpt_instructions_v_3.md` – current Custom GPT / MCP prompt instructions. - `docs/schema_summary_guidelines.md` – how schema summaries are constructed and how agents should interpret/use them. - **This file** - `docs/JOE_AI_Overview.md` – **(you are here)** high‑level map of AI schemas, plugins, UI, and MCP. Optional / advanced (for deeper RAG or power users): - Selected excerpts or exports of **schema summaries** for AI‑related schemas: - `ai_assistant`, `ai_prompt`, `ai_tool`, `ai_response`, `ai_conversation`, `ai_widget_conversation` - Any **project‑specific** AI workflow docs you maintain (e.g. playbooks for content generation, planning workflows, merge policies). --- ## 7. What’s Still Missing / Future Docs to Consider For a fully self‑sufficient agent, the following additional docs can be helpful: - **AI Workflows Cookbook**: - Short task‑oriented examples: - “Summarize a project using schemas + ai_prompt + MCP search.” - “Draft a plan then write objects via `saveObjects`.” - “Use ai_response to compare vs existing data and merge.” - **AI Safety & Guardrails**: - Which schemas/fields are sensitive. - Allowed vs disallowed writes. - Expectations around **review vs autonomous changes**. - **Instance‑specific Prompts / Taxonomy**: - Any custom `ai_prompt` records or `ai_tool` definitions that embody local conventions (naming standards, planning templates, scoring rubrics, etc.). If these don’t exist yet, an agent should assume **conservative** behavior: - Prefer read + propose changes over direct writes. - Tag all AI‑generated content clearly. - Use `ai_response` + compare/merge flows rather than overwriting objects blindly.