## JOE AI Overview This document gives Custom GPTs, Assistants, and AI agents a single mental model for how AI works inside **Json Object Editor (JOE)**: which schemas are involved, how the `chatgpt` plugin behaves, how the `joe-ai` UI connects to OpenAI, and how **MCP** exposes JOE as tools. --- ## 1. High‑level AI Architecture - **Core idea**: JOE is a schema‑driven data system. AI features are layered on top to: - Generate or transform field values (AI autofill) - Run persistent assistants and chats (in‑app + widgets) - Expose JOE data and writes via MCP tools for external agents - **Main components**: - **AI schemas** (`ai_assistant`, `ai_prompt`, `ai_tool`, `ai_response`, `ai_conversation`, `ai_widget_conversation`) - **Server plugins**: - `chatgpt.js` – **central / preferred** entry point for OpenAI (Responses API, tool calls, MCP bridge). New integrations should target this plugin. - `chatgpt-assistants.js` – legacy Assistants‑API plugin, still used by existing in‑app chat flows (`ai_conversation` / ``) but not recommended for new work. - `chatgpt-tools.js` / `chatgpt-responses.js` – helpers for tools + response shaping - **Client / UI**: - `joe-ai.js` – in‑app AI chat UI (web components) wired to `ai_conversation` + assistants - Standard JOE editor UI with **AI autofill** buttons on fields, driven by `ai_prompt` - **MCP**: - `/.well-known/mcp/manifest.json` – describes tools - `/mcp` – JSON‑RPC endpoint whose tools route into `JOE.Schemas`, `JOE.Storage`, `JOE.Cache` When in doubt: **schemas define structure**, **plugins call OpenAI + MCP**, **MCP exposes tools**, and **UI (`joe-ai`) is just a thin chat/controls layer on top. --- ## 2. AI Schemas (Mental Model) JOE uses a small set of AI‑specific schemas. A good agent should know what each represents and how they relate. ### 2.1 `ai_assistant` - **What it is**: Configuration record for a single AI assistant linked to OpenAI. - **Key fields** (from schema + summaries): - Identity & model: `name`, `info`, `ai_model`, `assistant_id`, `openai_assistant_version` - Capabilities: `file_search_enabled`, `code_interpreter_enabled` - Prompting: `instructions`, `assistant_thinking_text`, `assistant_color` - Tools: `tools` (JSON OpenAI tools array – often imported from MCP), `datasets`, `tags`, `status` - Sync meta: `last_synced`, timestamps (`created`, `joeUpdated`) - **How it’s used**: - One `ai_assistant` usually maps 1:1 to an OpenAI Assistant. - A **DEFAULT** can be set via `setting.DEFAULT_AI_ASSISTANT` and is used by `joe-ai` as the default assistant. - The schema exposes helper methods such as: - `syncAssistantToOpenAI` – sync this record to OpenAI via `chatgpt-assistants.js`. - `loadMCPTolsIntoAssistant` – pull tools from the MCP manifest into `assistant.tools`. - `setAsDefaultAssistant` – update the `DEFAULT_AI_ASSISTANT` setting. **Agent note**: When reasoning about which assistant is active in a chat or widget, look for: - `ai_conversation.assistant` or `ai_widget_conversation.assistant` - Or the instance‑level default via the `setting` schema. ### 2.2 `ai_prompt` - **What it is**: Reusable AI **prompt configuration** – how to call the `chatgpt` plugin or Responses API. - **Key fields**: - Identity: `_id`, `name`, `info`, `itemtype:'ai_prompt'` - Integration with plugin: - `prompt_method` – name of the server method on `chatgpt.js` to call (e.g. `executeJOEAiPrompt`). - `content_items` – `objectList` of `{ itemtype, reference }` describing which JOE objects to send and under what parameter name. - Instructions: - `functions` – helper code snippet (Node‑style `module.exports = async function(...) { ... }`) used to shape instructions/input. - `instructions_format` – format hint; `instructions` – main system‑level text. - `user_prompt` – optional user‑facing prompt template or per‑call input description. - OpenAI tuning: `ai_model`, `temperature` - Meta: `status`, `tags`, `datasets`, timestamps - **Methods / behavior**: - `methods.buildURL(prompt, items)` – builds URLs like: - `/API/plugin/chatgpt/?ai_prompt=&=...` - `methods.listExamples(prompt)` – picks sample content objects for exploration/testing. - **Typical flows**: - **AI Autofill (field‑level)**: - Fields in other schemas include an `ai` config (documented in `CHANGELOG` + README). - The UI calls `/API/plugin/chatgpt/autofill`, which: - Uses `ai_prompt` definitions and schema metadata. - Sends the right JOE objects + instructions to OpenAI. - Returns structured patch JSON to update fields. - **Explicit prompts**: - UI actions or external tools call `/API/plugin/chatgpt/?ai_prompt=&...`. - The plugin locates the prompt, merges helper `functions`, and runs OpenAI. **Agent note**: Treat `ai_prompt` as the **source of truth** for how to talk to OpenAI for a particular workflow (summaries, planning, refactors, etc). Never invent `prompt_method` names; reuse existing ones or ask a human to add a new prompt record. ### 2.3 `ai_response` - **What it is**: Persistent record of a single AI response, for **auditing**, **reuse**, and **merge‑back** into JOE. - **Key fields**: - Links: `ai_prompt` (reference), `referenced_objects` (array of JOE `_id`s used), `tags` - Request context: `user_prompt`, `prompt_method`, `model_used`, `response_type` - Response data: - `response` – raw string body (often JSON or text). - `response_keys` – array of keys returned (used for patch/merge). - `response_id` – OpenAI response ID. - `usage` – token usage object `{ input_tokens, output_tokens, total_tokens }`. - **Methods**: - `compareResponseToObject(response_id, object_id, do_alert)` – validate response JSON vs object fields, optionally trigger a merge. - `updateObjectFromResponse(response_id, object_id, fields)` – apply response values into a JOE object and update tags. - `listResponses(obj)` – list AI responses referencing a given object. **Agent note**: When you are updating objects via tools, favor: - Using `ai_response` records to compare/merge changes, especially if the response contains multiple fields. - Respect existing merge flows (`compareResponseToObject` / `updateObjectFromResponse`) instead of overwriting arbitrary fields. ### 2.4 `ai_tool` - **What it is**: Definition of a reusable **function‑calling tool** that OpenAI can call and JOE can execute server‑side. - **Key fields**: - Identity: `name`, `info`, `description`, `tool_id` - OpenAI schema: `tool_properties` (JSON, typically `{ type:'function', function:{ name, description, parameters } }`) - Execution: `server_code` – Node.js async code to run when the tool is invoked. - Meta: `datasets`, `tags`, timestamps. - **Methods**: - `updateToolProperties(propObj)` – helper to keep `tool_properties.function.name` in sync with `tool_id`. **Agent note**: For tools exposed via MCP, the canonical shape is actually in the **MCP manifest**; `ai_tool` is more for: - High‑level catalog and advanced OpenAI tools, not the core MCP schema/search/save tools. ### 2.5 `ai_conversation` - **What it is**: Metadata record for **in‑app AI conversations** (staff/admin chat in JOE). - **Key fields**: - Participants: `user` (JOE user), `assistant` (`ai_assistant`), `members` (external participants) - OpenAI thread: `thread_id` - Summary: `summary` (WYSIWYG), `status`, `tags` - Meta: `_id`, `name`, `info`, timestamps - **Behavior**: - Messages are **not stored** in JOE; they’re fetched from OpenAI at runtime (via `chatgpt-assistants.js`). - Schema methods + UI: - `methods.chatSpawner(object_id)` – calls `_joe.Ai.spawnContextualChat(object_id)`. - List view shows created time, user cubes, and context objects used for that chat. - `joe-ai.js` uses `ai_conversation` as the anchor object for chat history + context. **Agent note**: If you need to understand a chat history, you: - Read `ai_conversation` for metadata and context_objects. - Use **MCP tools** or existing chat APIs to fetch live thread content; don’t invent local persistence. ### 2.6 `ai_widget_conversation` - **What it is**: Lightweight conversation record for **embeddable AI widgets** (external sites, test pages). - **Key fields**: - Participant & assistant: `user`, `assistant`, `user_name`, `user_color`, `assistant_color` - OpenAI info: `model`, `assistant_id`, `system` (effective instructions) - Conversation: `messages` (compact list, typically JSON array of `{ role, content, created_at }`), `last_message_at` - Source: `source` (widget origin), `tags`, timestamps - **Behavior**: - Accessed/updated via `chatgpt-assistants` widget endpoints: - `widgetStart`, `widgetMessage`, `widgetHistory`, etc. - Schema exposes **subsets** and **filters** for grouping by `source`, `user`, and `assistant`. **Agent note**: Treat `ai_widget_conversation` as the external/chat‑widget equivalent of `ai_conversation`. The fields are tuned for widgets and public UI, but the mental model is the same: small record pointing to assistant + messages. --- ## 3. `chatgpt` Plugin and Related Plugins ### 3.1 `server/plugins/chatgpt.js` This is the **core server plugin** that wires JOE to OpenAI and MCP. - **Responsibilities**: - Read OpenAI API key from `setting.OPENAI_API_KEY`. - Provide shared helpers for building OpenAI clients (`OpenAI` SDK). - Implement orchestration for: - **Responses API** calls (models like `gpt-4.1`, `gpt-4.1-mini`, `gpt-5.1`). - **Tool calling** and follow‑up calls via MCP tools. - Bridge between JOE and MCP with helpers like: - `callMCPTool(toolName, params, ctx)` – call MCP tools **in‑process** (without HTTP). - Response parsing helpers such as `extractToolCalls(response)`. - Payload shrinking helpers like `shrinkUnderstandObjectMessagesForTokens(...)`. - Expose specific routes under `/API/plugin/chatgpt/...`, including: - Field autofill (`autofill`) driven by schema‑level `ai` configs. - Prompt execution handlers for `ai_prompt.prompt_method` values. - **Key patterns**: - **Slimming payloads**: When embedding object graphs into messages (e.g. `understandObject` results), `chatgpt.js` converts them into a slim representation to avoid token limits. - **Error handling**: - `isTokenLimitError(err)` identifies token/TPM error shapes for better logs and fallbacks. **Agent note**: Any time you see endpoints in docs like `/API/plugin/chatgpt/`, they resolve into functions in `chatgpt.js` or its siblings. Follow those conventions instead of inventing new paths. ### 3.2 `chatgpt-assistants.js`, `chatgpt-responses.js`, `chatgpt-tools.js` - **`chatgpt-assistants.js`**: - Manages OpenAI Assistants and threads using the **older Assistants API**. - Endpoints such as: - `syncAssistantToOpenAI` (used from `ai_assistant.methods`). - `getThreadMessages`, `runAssistantChat`, widget endpoints like `widgetStart`, `widgetMessage`, `widgetHistory`. - Used by `joe-ai.js` and widget UIs to pull/push messages for a given conversation. - **Status**: legacy bridge for existing flows; **new integrations should use the Responses‑based `chatgpt` plugin instead**, matching the README. - **`chatgpt-responses.js`**: - Helpers for working with Responses API results: - Extracting text and tool calls. - Normalizing outputs for storage in `ai_response`. - Often called from `chatgpt.js` orchestration functions. - **`chatgpt-tools.js`**: - Glue between AI tools and JOE: - Uses `ai_tool` definitions. - Executes server‑side code blocks for tools when OpenAI calls them. - Can also support MCP‑style tools exported via the manifest. **Agent note**: You typically don’t call these plugins directly from a GPT. Instead, you: - Use documented **APIs** (`/API/plugin/chatgpt/*`, widget endpoints) and/or - Use **MCP tools** for reads/writes. --- ## 4. `joe-ai.js` – In‑App AI UI The `js/joe-ai.js` file implements JOE’s in‑app AI user interface using custom elements like `` and other helpers under a `Ai` namespace. - **Key responsibilities**: - Manage open chat boxes and default assistant selection: - `Ai._openChats`, `Ai.default_ai`, `Ai.getDefaultAssistant()`. - Render **chat UI** bound to an `ai_conversation`: - Loads `ai_conversation` via `/API/object/ai_conversation/_id/`. - Uses `chatgpt-assistants` endpoints (e.g. `getThreadMessages`) to fetch messages. - Shows assistant/user labels, context objects, and a message history. - Provide ergonomic chat behaviors: - Markdown rendering via `renderMarkdownSafe` (prefers `marked` + `DOMPurify`). - Greeting messages based on `user` and `context_objects`. - Keyboard handling (Enter to send). - **High‑level flow** for in‑app chat: 1. User opens an `ai_conversation` or clicks “Continue Conversation” from a context object. 2. `joe-ai.js` spawns a `` bound to that `ai_conversation`. 3. The chatbox: - Fetches conversation + thread from the server. - Uses `chatgpt-assistants` to send/receive messages to/from OpenAI. - Renders messages and any embedded JOE object markers into the UI. **Agent note**: When describing UI behavior or helping with debugging, remember that `joe-ai.js` is **UI only**; persistent data lives in: - `ai_conversation` (metadata) + OpenAI threads - `ai_assistant` (config) - `ai_widget_conversation` for external widgets --- ## 5. MCP: How JOE Exposes Tools to Agents MCP (Model Context Protocol) is how external agents (including Custom GPTs) call into JOE as a **tool provider**. ### 5.1 Endpoints and Manifest - **Endpoints**: - `GET /.well-known/mcp/manifest.json` – public manifest with: - Instance info `{ joe: { name, version, hostname } }` - Tool list (name, description, params schema) - Privacy/terms URLs (also wired into `/privacy` and `/terms` routes). - `POST /mcp` – auth‑protected JSON‑RPC 2.0 endpoint for **tool calls**. - Same auth rules as other APIs; if users exist, requires cookie or basic auth. - **Implementation note**: - The concrete MCP tool implementations live in `server/modules/MCP.js`, which is wired up by `server/init.js` and exported via the manifest and `/mcp` endpoint. - **Tools** (from README + CHANGELOG): - `listSchemas(name?)`, `getSchema(name)` - `getObject(_id, itemtype?)` - `search` (unified search across cache/storage; supports `source`, `ids`, `flatten`, `depth`, `countOnly`) - `fuzzySearch` – typo‑tolerant search across weighted fields - `saveObject({ object })` - `saveObjects({ objects, stopOnError?, concurrency? })` - `hydrate` – returns core fields, schemas, statuses, tags (for agents to bootstrap context) All tools map directly into JOE’s APIs: `JOE.Schemas`, `JOE.Storage`, `JOE.Cache`, and related helpers. Sensitive fields are sanitized before returning data. ### 5.2 Agent Best Practices with MCP - **Schema‑first**: Always: 1. Discover schemas via `listSchemas`/`getSchema`. 2. Use `hydrate` on startup to cache schemas/statuses/tags. 3. Use `search` / `fuzzySearch` with explicit `itemtype` and filters rather than arbitrary text mining. - **Reads vs writes**: - Prefer **cache** for read operations (`search` default) unless you explicitly need DB‑level freshness (set `source:"storage"`). - For writes, use: - `saveObject` for single‑item updates. - `saveObjects` for batched updates with clear error handling. - **Performance**: - Use `countOnly` before large reads to avoid pulling massive datasets. - When embedding objects into prompts, prefer **slimmed** representations (id, itemtype, name, info). **Agent note**: The MCP tools are your primary way to **inspect and modify JOE data** from a Custom GPT or other agent host. Direct HTTP calls to JOE’s internal `/API/*` routes should be treated as implementation detail unless explicitly allowed in your instructions. --- ## 6. Recommended RAG / Context Files for JOE AI Agents When configuring a JOE‑aware Custom GPT or dev agent, include the following docs in its RAG/context: - **Core architecture & instance info** - `README.md` – especially: - “Architecture & Mental Model (Server)” - MCP overview + test instructions - `CHANGELOG.md` – AI/MCP‑related entries (0.10.43x+, 0.10.50x+, 0.10.62x+), already summarize important AI behavior. - **Agent / instruction docs (existing)** - `docs/JOE_Master_Knowledge_Export.md` – master context dump for schemas and concepts. - `docs/joe_agent_spec_v_2.2.md` – agent behavior/specification. - `docs/joe_agent_custom_gpt_instructions_v_3.md` – current Custom GPT / MCP prompt instructions. - `docs/schema_summary_guidelines.md` – how schema summaries are constructed and how agents should interpret/use them. - **This file** - `docs/JOE_AI_Overview.md` – **(you are here)** high‑level map of AI schemas, plugins, UI, and MCP. Optional / advanced (for deeper RAG or power users): - Selected excerpts or exports of **schema summaries** for AI‑related schemas: - `ai_assistant`, `ai_prompt`, `ai_tool`, `ai_response`, `ai_conversation`, `ai_widget_conversation` - Any **project‑specific** AI workflow docs you maintain (e.g. playbooks for content generation, planning workflows, merge policies). --- ## 7. What’s Still Missing / Future Docs to Consider For a fully self‑sufficient agent, the following additional docs can be helpful: - **AI Workflows Cookbook**: - Short task‑oriented examples: - “Summarize a project using schemas + ai_prompt + MCP search.” - “Draft a plan then write objects via `saveObjects`.” - “Use ai_response to compare vs existing data and merge.” - **AI Safety & Guardrails**: - Which schemas/fields are sensitive. - Allowed vs disallowed writes. - Expectations around **review vs autonomous changes**. - **Instance‑specific Prompts / Taxonomy**: - Any custom `ai_prompt` records or `ai_tool` definitions that embody local conventions (naming standards, planning templates, scoring rubrics, etc.). If these don’t exist yet, an agent should assume **conservative** behavior: - Prefer read + propose changes over direct writes. - Tag all AI‑generated content clearly. - Use `ai_response` + compare/merge flows rather than overwriting objects blindly.