--- name: ideogram-ultra description: Build Ideogram 4 (Ideogram Ultra) txt2img and img2img workflows — local open-weights model, dual conditional/unconditional models with DualModelGuider, Qwen3-VL text encoder, and structured JSON ("compositional deconstruction") prompts for strong text rendering and layout control globs: - "**/*.json" --- # Ideogram 4 (Ideogram Ultra) Workflows ## Overview **This is a LOCAL open-weights pipeline — NOT the hosted Ideogram API.** There is no API key, no `IdeogramGenerate` API node, and no network call at generation time. Comfy-Org released the Ideogram 4 weights on Hugging Face and they run entirely on your GPU via standard `UNETLoader` / `CLIPLoader` / `VAELoader` nodes. (Note: ComfyUI *also* ships separate API/"partner" nodes that call the paid hosted Ideogram service — that is a different thing and is not what this workflow uses.) Ideogram 4 is best known for **text rendering / typography**, **poster and graphic-design layouts**, and **prompt adherence**. The hallmark of this workflow is a **structured JSON prompt** (a "compositional deconstruction" caption with bounding boxes) instead of a plain text prompt — this is what gives precise control over where text and objects land in the frame. Source workflow this skill is derived from: `IDEOGRAM_ULTRA_WORKFLOW-V2.json` (UI format, 66 nodes, 4 subgraphs), by Aitrepreneur. It provides both a **TEXT TO IMAGE** path and an **IMAGE TO IMAGE** path. ### Two unusual things to know up front 1. **Dual models.** Two UNETs are loaded: a conditional model (`ideogram4_fp8_scaled`) and an `..._unconditional_fp8_scaled` model. A `DualModelGuider` node uses both to perform asymmetric classifier-free guidance — the unconditional model provides the CFG baseline. There is **no negative text prompt**; negative conditioning is `ConditioningZeroOut`. 2. **Two text models, different jobs.** - `qwen3vl_8b_fp8_scaled` is the **actual diffusion text encoder** (loaded with `CLIPLoader`, type `ideogram4`). - `gemma4_e4b_it_fp8_scaled` is used **only inside an optional prompt-builder subgraph** (a `TextGenerate` node) that auto-writes the structured JSON from a plain idea. It is not the diffusion encoder. ## Installation ### Custom nodes The workflow needs these four custom node packs (clone into `ComfyUI/custom_nodes/`). Exact repos from the installer scripts: ```bash git clone https://github.com/ltdrdata/ComfyUI-Manager.git git clone https://github.com/rgthree/rgthree-comfy git clone https://github.com/kijai/ComfyUI-KJNodes git clone https://github.com/cubiq/ComfyUI_essentials ``` - **ComfyUI-KJNodes** (kijai) — provides `Ideogram4PromptBuilderKJ`, `ImageSharpenKJ`, `TextGenerate`, and the Ideogram 4 helper nodes. **Required.** - **rgthree-comfy** — `Power Lora Loader`, `Fast Groups Muter/Bypasser`, `Label`, `Any Switch`. (Used for UI/convenience; the core pipeline still works without them.) - **ComfyUI_essentials** (cubiq) — `ImageResize+` (used in the img2img path). - **ComfyUI-Manager** — node/model management; not required at run time. > The core nodes used in the simplified workflows below (`UNETLoader`, `CLIPLoader`, `VAELoader`, `DualModelGuider`, `SamplerCustomAdvanced`, `ModelSamplingAuraFlow`, `BasicScheduler`, `EmptyFlux2LatentImage`, `CLIPTextEncode`, `ConditioningZeroOut`, `VAEDecode`) are **built into ComfyUI** (recent versions). Only `Ideogram4PromptBuilderKJ` / `ImageSharpenKJ` require KJNodes. ### Models Five files. Folder layout and download URLs are taken verbatim from `IDEOGRAM_ULTRA-MODELS-NODES_INSTALL.bat` / `...RUNPOD.sh`: | File | Folder | Source URL | |------|--------|------------| | `ideogram4_fp8_scaled.safetensors` | `models/diffusion_models/` | `https://huggingface.co/Comfy-Org/Ideogram-4/resolve/main/diffusion_models/ideogram4_fp8_scaled.safetensors` | | `ideogram4_unconditional_fp8_scaled.safetensors` | `models/diffusion_models/` | `https://huggingface.co/Comfy-Org/Ideogram-4/resolve/main/diffusion_models/ideogram4_unconditional_fp8_scaled.safetensors` | | `qwen3vl_8b_fp8_scaled.safetensors` | `models/text_encoders/` | `https://huggingface.co/Aitrepreneur/FLX/resolve/main/qwen3vl_8b_fp8_scaled.safetensors` | | `gemma4_e4b_it_fp8_scaled.safetensors` | `models/text_encoders/` | `https://huggingface.co/Aitrepreneur/FLX/resolve/main/gemma4_e4b_it_fp8_scaled.safetensors` | | `flux2-vae.safetensors` | `models/vae/` | `https://huggingface.co/Aitrepreneur/FLX/resolve/main/flux2-vae.safetensors` | Notes / things to verify: - The two **diffusion models** come from the official `Comfy-Org/Ideogram-4` HF repo. The **text encoders + VAE** are mirrored from the third-party `Aitrepreneur/FLX` repo in these scripts; the official ones also live on Comfy-Org / Comfy-Org-adjacent repos. Both should be identical files but the FLX mirror is what the provided installer pulls. - **File sizes are uncertain.** The official ComfyUI docs page lists each diffusion model at ~13.8 GB, qwen3vl at ~8 GB, gemma4 at ~2 GB, vae at ~335 MB (~38.9 GB total). A web search result claimed `ideogram4_fp8_scaled` is ~9.28 GB. Treat sizes as approximate — confirm against the HF file listing. - `flux2-vae.safetensors` is the same VAE used by Flux 2 / Klein workflows. ### Linux / RunPod note `IDEOGRAM_ULTRA-AUTO_INSTALL-RUNPOD.sh` creates a venv and installs Torch `2.4.0` + `cu121` by default (override via env vars `CUDA_TAG`, `TORCH_VERSION`, etc.). Same five model files, same four node repos. ### JSON template pack (optional) `IDEOGRAM-TEMPLATES.zip` contains 25 ready-made structured-JSON templates for the `Ideogram4PromptBuilderKJ` node (film poster, book cover, logo board, character sheet, magazine cover, etc.). Per its README, copy the `.json` files into: ``` ComfyUI/user/default/kjnodes/ideogram4/templates ``` then pick them from the template dropdown inside the prompt-builder node. ## Key Nodes ### CLIPLoader (text encoder) Ideogram 4 uses **Qwen3-VL** as its diffusion text encoder. Load it with `CLIPLoader` and **type `ideogram4`**: ```json { "class_type": "CLIPLoader", "inputs": { "clip_name": "qwen3vl_8b_fp8_scaled.safetensors", "type": "ideogram4", "device": "default" } } ``` ### UNETLoader x2 (conditional + unconditional) ```json { "class_type": "UNETLoader", "inputs": { "unet_name": "ideogram4_fp8_scaled.safetensors", "weight_dtype": "default" }}, { "class_type": "UNETLoader", "inputs": { "unet_name": "ideogram4_unconditional_fp8_scaled.safetensors", "weight_dtype": "default" }} ``` ### ModelSamplingAuraFlow (shift) Applied to **both** models. In the source workflow `shift = 5`: ```json { "class_type": "ModelSamplingAuraFlow", "inputs": { "model": ["", 0], "shift": 5 }} ``` ### DualModelGuider The heart of the pipeline. Takes the (shifted) conditional model, the (shifted) unconditional model, positive conditioning, and negative conditioning, plus a CFG value (`5` in the source). This replaces the usual `CFGGuider`. ```json { "class_type": "DualModelGuider", "inputs": { "model": ["", 0], "model_negative": ["", 0], "positive": ["", 0], "negative": ["", 0], "cfg": 5 } } ``` > Input names for `DualModelGuider` (`model_negative`, `cfg`) are inferred from the subgraph wiring and KJNodes; verify against your installed KJNodes version — the exact widget/socket names may differ slightly. ### EmptyFlux2LatentImage Ideogram 4 uses the Flux 2 latent format, so the empty latent is `EmptyFlux2LatentImage` (not `EmptyLatentImage`): ```json { "class_type": "EmptyFlux2LatentImage", "inputs": { "width": 1024, "height": 1024, "batch_size": 1 }} ``` ### BasicScheduler + KSamplerSelect + RandomNoise + SamplerCustomAdvanced Generation uses the modular sampler stack, not `KSampler`: ```json { "class_type": "BasicScheduler", "inputs": { "model": ["", 0], "scheduler": "simple", "steps": 28, "denoise": 1 }}, { "class_type": "KSamplerSelect", "inputs": { "sampler_name": "euler" }}, { "class_type": "RandomNoise", "inputs": { "noise_seed": 42 }}, { "class_type": "SamplerCustomAdvanced", "inputs": { "noise": ["", 0], "guider": ["", 0], "sampler": ["", 0], "sigmas": ["", 0], "latent_image": ["", 0] }} ``` ### Ideogram4PromptBuilderKJ (the JSON prompt builder) A KJNodes node that outputs the structured caption JSON string (see "Prompt Style" below). On the canvas you draw bounding boxes for objects/text, set descriptions, a color palette, and background/style. Its string output feeds `CLIPTextEncode`. Source widget values include `width 1920`, `height 1080`, a high-level prompt, a background description, a `color_palette` array, and an `elements` array. #### ⚠️ Editing this node programmatically (panel_set_widget WILL NOT STICK) **This is the single most important thing to know about this node.** Its `elements_data` / `style_palette_data` widgets are **NOT the source of truth** — they are serialized *from* a live in-browser array (`node._boxes`) inside the KJNodes editor JS. Two mechanisms defeat any external widget edit: - **Queue-time re-serialization.** In `web/js/ideogram4_prompt_builder.js`, `elementsWidget.serializeValue()` regenerates the value from `node._boxes` *every time the graph is queued*. So `panel_set_widget(14, "elements_data", ...)` sets the value, but ComfyUI overwrites it with the stale editor boxes the instant you run. **The edit silently reverts on every run.** - **You can't reach `node._boxes`.** It lives in the browser tab; no panel/MCP tool can touch it. Editing `elements_data`, `ideo_editor`, or forcing `import_mode` alone does nothing durable. `node._boxes` is only ever re-seeded from `elements_data` on workflow *load* (`onConfigure`), and even then the saved `o.ideo.boxes` blob wins over the widget — so a stale saved workflow reloads stale. **The symptom:** you edit the JSON, the panel confirms the new value, but the render (and the visible builder JSON) still shows the OLD prompt — e.g. old subject/region text that "won't go away." **The correct, node-designed fix — drive it via `import_json`:** 1. Add a `PrimitiveStringMultiline` node containing the FULL caption JSON (the `high_level_description` + `style_description` + `compositional_deconstruction` shape from "Prompt Style" below). 2. Wire it into node 14's **`import_json`** input. 3. Set **`import_mode = "always"`**. The Python `execute()` then does `used_import = imported is not None and (import_mode == "always" or not boxes)` → the caption is built **entirely** from `import_json`; the poisoned `elements_data`/`node._boxes` are ignored. Bonus: running once in this mode pushes the caption back into the editor via `ui`, which re-seeds `node._boxes` and flushes the stale boxes for good. From then on, edit the prompt in the wired string node, **not** the builder's visual editor. (`import_mode = "when empty"` only seeds the editor when it has no regions, then the editor wins again — so for programmatic control it MUST be `"always"`.) ### ImageSharpenKJ (post-process) The source applies a light RCAS sharpen after decode: `sharpen_mode = rcas`, `strength = 0.55`. ## Parameters / Settings Values below are exactly what the source `IDEOGRAM_ULTRA_WORKFLOW-V2.json` ships with. | Parameter | Value | Where | |-----------|-------|-------| | sampler | `euler` | `KSamplerSelect` | | scheduler | `simple` | `BasicScheduler` | | steps | **28** (default) | `BasicScheduler` / `INTConstant STEPS` | | steps (turbo) | **12** | per workflow Note: "TURBO: 12 STEPS — quick image, lower quality" | | cfg | **5** | `DualModelGuider` | | shift | **5** | `ModelSamplingAuraFlow` (both models) | | denoise (txt2img) | 1.0 | `BasicScheduler` | | denoise (img2img) | **0.6** | `PrimitiveFloat DENOISE` | | sharpen | rcas, 0.55 | `ImageSharpenKJ` | | noise control | `RandomNoise`, fixed seed | seed sample value `1335735769456` | There is also a `CFGOverride` node in the sampler subgraph (widgets `3, 0.7, 1`); it is an optional override and is not the primary guidance path — the primary CFG is `DualModelGuider`'s `5`. ## Resolutions / Aspect Ratios The source default is **1920x1080** (set via `INTConstant WIDTH/HEIGHT`, fed into the prompt builder and latent). The bundled template README recommends: | Use case | Resolution | Aspect | |----------|-----------|--------| | Vertical posters / covers | 1440x2560 | 9:16 | | Wide landscape | 2560x1440 | 16:9 | | Square asset sheets | 2048x2048 | 1:1 | | Ultrawide / special layouts | 2880x1440 or 2048x1024 | 2:1-ish | | Source default | 1920x1080 | 16:9 | If a layout looks cramped, increase resolution while keeping the aspect ratio. Use the same width/height in the prompt-builder JSON, the latent, and (img2img) the resize node. ## Prompt Style — Structured JSON ("compositional deconstruction") Ideogram 4 in this workflow expects a **JSON caption**, not free text. Minimum required shape: ```json { "high_level_description": "one-sentence summary of the whole image", "compositional_deconstruction": { "background": "scene, environment, color palette, lighting, overall mood", "elements": [ { "type": "obj", "bbox": [top, left, bottom, right], "desc": "what this object is and how it's rendered" }, { "type": "text", "bbox": [top, left, bottom, right], "text": "ACTUAL TEXT TO RENDER", "desc": "font style, color, alignment, vintage/print treatment, etc." } ] } } ``` An optional `style_description` object (`aesthetics`, `lighting`, `medium`, `art_style`, `color_palette`) can sit at the top level for global style locking. ### bbox rules (critical) - Format is **`[top, left, bottom, right]`**, values **0–1000** (NOT pixels, NOT x/y/w/h). - **One bbox per major subject** — do not split a person into face/hair/clothes boxes; put all detail in one `desc`. - Use extra boxes only for genuinely separate items (a product, a title, a second character). - For character groups in wide images, use **vertical columns** of non-overlapping boxes. - For posters: reserve a top zone for the title, middle for the subject, bottom for subtitle/CTA. - Overlapping text boxes cause garbled text — increase spacing or remove boxes. ### Text rendering tips - Keep rendered text **short and bold** ("ORDER NOW", "COMING THIS FALL"). Long/tiny text still fails sometimes. - Lock style explicitly when needed, e.g. `"rendered as an actual live-action photograph, not anime, not illustration"` or `"high-quality Japanese anime, cel shading, not photographic"`. - 3–6 strong elements beat 20 overlapping ones. ### Generating the JSON automatically Two options the source provides: 1. **Local (Gemma4 subgraph):** the "JSON Prompt Builder (Gemma4)" subgraph runs a `TextGenerate` node on `gemma4_e4b_it_fp8_scaled` with a system prompt that converts a plain idea into the JSON. (TextGenerate widgets in source: max_tokens 2048, temperature 0.7, top_k 64, top_p 0.95, etc.) 2. **External LLM:** a workflow Note ships a full system prompt ("You are an expert Ideogram 4 structured JSON prompt assistant…") to paste into ChatGPT/Claude to produce the caption JSON, which you then paste into the manual prompt field. Either way, the resulting JSON must be valid (double quotes, no trailing commas, exact key `compositional_deconstruction`) or the builder reports "NOT A VALID IDEOGRAM 4 CAPTION JSON". ## Complete Workflow: Text to Image (API format, simplified) This is the core txt2img path derived from the source, written in API format. Put your structured JSON caption into node `5`'s `text`. ```json { "1": { "class_type": "UNETLoader", "inputs": { "unet_name": "ideogram4_fp8_scaled.safetensors", "weight_dtype": "default" }}, "2": { "class_type": "UNETLoader", "inputs": { "unet_name": "ideogram4_unconditional_fp8_scaled.safetensors", "weight_dtype": "default" }}, "3": { "class_type": "CLIPLoader", "inputs": { "clip_name": "qwen3vl_8b_fp8_scaled.safetensors", "type": "ideogram4", "device": "default" }}, "4": { "class_type": "VAELoader", "inputs": { "vae_name": "flux2-vae.safetensors" }}, "5": { "class_type": "CLIPTextEncode", "inputs": { "clip": ["3", 0], "text": "" }}, "6": { "class_type": "ConditioningZeroOut", "inputs": { "conditioning": ["5", 0] }}, "7": { "class_type": "ModelSamplingAuraFlow", "inputs": { "model": ["1", 0], "shift": 5 }}, "8": { "class_type": "ModelSamplingAuraFlow", "inputs": { "model": ["2", 0], "shift": 5 }}, "9": { "class_type": "DualModelGuider", "inputs": { "model": ["7", 0], "model_negative": ["8", 0], "positive": ["5", 0], "negative": ["6", 0], "cfg": 5 }}, "10": { "class_type": "EmptyFlux2LatentImage", "inputs": { "width": 1920, "height": 1080, "batch_size": 1 }}, "11": { "class_type": "BasicScheduler", "inputs": { "model": ["7", 0], "scheduler": "simple", "steps": 28, "denoise": 1 }}, "12": { "class_type": "KSamplerSelect", "inputs": { "sampler_name": "euler" }}, "13": { "class_type": "RandomNoise", "inputs": { "noise_seed": 42 }}, "14": { "class_type": "SamplerCustomAdvanced", "inputs": { "noise": ["13", 0], "guider": ["9", 0], "sampler": ["12", 0], "sigmas": ["11", 0], "latent_image": ["10", 0] }}, "15": { "class_type": "VAEDecode", "inputs": { "samples": ["14", 0], "vae": ["4", 0] }}, "16": { "class_type": "ImageSharpenKJ", "inputs": { "image": ["15", 0], "sharpen_mode": "rcas", "strength": 0.55 }}, "17": { "class_type": "SaveImage", "inputs": { "images": ["16", 0], "filename_prefix": "IDEOGRAM" }} } ``` If you don't have KJNodes / want to skip sharpening, drop node `16` and feed `["15", 0]` straight into `SaveImage`. ## Image to Image (notes) The source img2img path adds, before sampling: 1. `LoadImage` → `ImageResize+` (ComfyUI_essentials; `keep proportion`, lanczos, e.g. target 1024) → `VAEEncode` (with `flux2-vae`) to produce the input latent. 2. Feed that latent into `SamplerCustomAdvanced.latent_image` instead of `EmptyFlux2LatentImage`. 3. Set **denoise `0.6`** on `BasicScheduler` (this is the `PrimitiveFloat DENOISE` value in the source). Lower denoise = closer to the input image. Everything else (dual models, DualModelGuider, scheduler, sampler, decode, sharpen) is identical to txt2img. > The exact `VAEEncode` wiring for img2img is inferred (the source routes it through an rgthree `Any Switch` and subgraph I/O); confirm sockets in your build. ## Pipeline at a Glance ``` UNETLoader (ideogram4_fp8_scaled) ─────► ModelSamplingAuraFlow(shift=5) ─┐ UNETLoader (ideogram4_unconditional) ──► ModelSamplingAuraFlow(shift=5) ─┤ ├─► DualModelGuider(cfg=5) CLIPLoader (qwen3vl_8b, type=ideogram4) ─► CLIPTextEncode(JSON) ──────────┤ │ └► ConditioningZeroOut ──────┘ │ EmptyFlux2LatentImage (1920x1080) ─────────────────────────────────────────────────┤ BasicScheduler (simple, 28, denoise=1) ─────────────────────────────────────────────┤ KSamplerSelect (euler) ─────────────────────────────────────────────────────────────┤ RandomNoise (fixed seed) ───────────────────────────────────────────────────────────┘ └► SamplerCustomAdvanced ─► VAEDecode (flux2-vae) └► ImageSharpenKJ (rcas 0.55) ─► SaveImage Optional upstream: Ideogram4PromptBuilderKJ OR TextGenerate(gemma4) ─► CLIPTextEncode.text ``` ## VRAM Considerations - Both diffusion models plus the Qwen3-VL encoder are large. Loading **two** ~13.8 GB UNETs is the main cost; expect this to be heavy on 24 GB GPUs. The official docs cite a 16 GB minimum for the FP8 models, but that assumes ComfyUI swaps models in/out rather than holding both resident. - **Always `clear_vram`** before switching to Ideogram 4 from another model family. - If you OOM: rely on ComfyUI's automatic model offloading, run `--lowvram`, or reduce resolution. - Exact VRAM numbers for the dual-model setup are **not verified** in the source files — treat the above as guidance, not measured figures. ## Troubleshooting - **Prompt edits to `Ideogram4PromptBuilderKJ` won't stick / stale prompt keeps coming back** — `elements_data` is re-serialized from the browser editor's `node._boxes` at queue time, so `panel_set_widget` reverts on every run and you can't reach `node._boxes` externally. Fix: wire a `PrimitiveStringMultiline` (full caption JSON) into the node's `import_json` input and set `import_mode = "always"`. See "Editing this node programmatically" under Key Nodes. This is the ONLY reliable way to drive the prompt from outside the browser. - **"NOT A VALID IDEOGRAM 4 CAPTION JSON"** — JSON is malformed. Use double quotes, no trailing commas, exact key `compositional_deconstruction`, matched brackets. Validate in any JSON linter. - **Garbled / overlapping text in the image** — text bboxes overlap or text is too long. Increase spacing, shorten text, remove boxes. - **Style drift (anime when you wanted photo, etc.)** — add explicit style-lock language in the element/style description. - **Wrong element placement** — remember bbox is `[top, left, bottom, right]` 0–1000, not pixels and not x/y/w/h. - **Same seed, different image across machines** — expected; differs by GPU, drivers, PyTorch/CUDA/ComfyUI versions (per template README). - **CLIPLoader type missing `ideogram4`** — update ComfyUI; the `ideogram4` CLIP type and the Flux2/Ideogram nodes require a recent build (installer pins ComfyUI portable `v0.24.0`). - **`Ideogram4PromptBuilderKJ` / `DualModelGuider` not found** — update KJNodes (`git pull` in `custom_nodes/ComfyUI-KJNodes`); these are recent additions. ```