--- name: huggingface-datasets description: "Use when Use this skill for Hugging Face Dataset Viewer API workflows that fetch subset/split metadata, paginate rows, search text, apply filters, download parquet URLs, and read size or statistics." version: 6.0.0 last-updated: 2026-09-29 tools: Read, Grep, Glob, Bash, Edit, Write scripts-binding: - .agent/scripts/lint_runner.js - .agent/scripts/verify_all.js inputs: task: "string" target_file: "string" outputs: result: "string" verification_status: "boolean" trigger: - "huggingface" - "skill" - "hugging" --- # Hugging Face Dataset Viewer ## Mandatory Pre-Flight Context Inspection Before reading, generating, or refactoring code in the `huggingface-datasets` domain, inspect these 5 critical parameters: 1. **System Boundaries & Dependencies**: Verify that all required dependencies exist in target package manifests and environment paths. 2. **Runtime Context & Platform Invariants**: Confirm target platform constraints (Node.js, Browser, Mobile OS, Edge runtime) before applying APIs. 3. **Execution Guardrails**: Identify potential side-effects, state mutations, and unhandled asynchronous exceptions. 4. **Validation & Type Contracts**: Validate input data schemas and strict type constraints across all module interfaces. 5. **Observability & Proof of Execution**: Ensure execution produces tangible verification signals (terminal output, tests, metrics). ## Activation Boundaries - **Activate when:** Use when Use this skill for Hugging Face Dataset Viewer API workflows that fetch subset/split metadata, paginate rows, search text, apply filters, download parquet URLs, and read size or statistics. - **DO NOT activate when:** The task falls outside the `huggingface-datasets` domain or is managed by a different dedicated specialist agent. ## ๐Ÿ” Multi-Pass Execution Protocol | Pass | Phase | Core Action | Adaptive Depth | |:---|:---|:---|:---| | **Pass 1** | **Understand** | Deconstruct the user's explicit objective, implicit requirements, and platform constraints. | Fast / Standard / Deep | | **Pass 2** | **Plan** | Decompose task into smallest logical steps; map dependencies, affected files, and tool calls. | Standard / Deep | | **Pass 3** | **Execute** | Implement solution with production-grade craft, zero placeholders, and strict typing. | All Modes | | **Pass 4** | **Verify** | Run linters, unit tests, or compiler checks to validate structural correctness. | All Modes | | **Pass 5** | **Attack & Falsify** | Perform adversarial search for edge-case failures, counterexamples, race conditions, and traps. | Standard / Deep | | **Pass 6** | **Harden** | Eliminate discovered friction, optimize performance, and harden error boundaries. | Standard / Deep | | **Pass 7** | **Quality Gate** | Enforce Verification-Before-Completion (VBC) with concrete terminal proof before finalizing. | All Modes | Use this skill to execute read-only Dataset Viewer API calls for dataset exploration and extraction. ## Core workflow 1. Optionally validate dataset availability with `/is-valid`. 2. Resolve `config` + `split` with `/splits`. 3. Preview with `/first-rows`. 4. Paginate content with `/rows` using `offset` and `length` (max 100). 5. Use `/search` for text matching and `/filter` for row predicates. 6. Retrieve parquet links via `/parquet` and totals/metadata via `/size` and `/statistics`. ## Defaults - Base URL: `https://datasets-server.huggingface.co` - Default API method: `GET` - Query params should be URL-encoded. - `offset` is 0-based. - `length` max is usually `100` for row-like endpoints. - Gated/private datasets require `Authorization: Bearer `. ## Dataset Viewer - `Validate dataset`: `/is-valid?dataset=` - `List subsets and splits`: `/splits?dataset=` - `Preview first rows`: `/first-rows?dataset=&config=&split=` - `Paginate rows`: `/rows?dataset=&config=&split=&offset=&length=` - `Search text`: `/search?dataset=&config=&split=&query=&offset=&length=` - `Filter with predicates`: `/filter?dataset=&config=&split=&where=&orderby=&offset=&length=` - `List parquet shards`: `/parquet?dataset=` - `Get size totals`: `/size?dataset=` - `Get column statistics`: `/statistics?dataset=&config=&split=` - `Get Croissant metadata (if available)`: `/croissant?dataset=` Pagination pattern: ```bash curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=0&length=100" curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=100&length=100" ``` When pagination is partial, use response fields such as `num_rows_total`, `num_rows_per_page`, and `partial` to drive continuation logic. Search/filter notes: - `/search` matches string columns (full-text style behavior is internal to the API). - `/filter` requires predicate syntax in `where` and optional sort in `orderby`. - Keep filtering and searches read-only and side-effect free. For CLI-based parquet URL discovery or SQL, use the `hf-cli` skill with `hf datasets parquet` and `hf datasets sql`. ## Creating and Uploading Datasets Use one of these flows depending on dependency constraints. Zero local dependencies (Hub UI): - Create dataset repo in browser: `https://huggingface.co/new-dataset` - Upload parquet files in the repo "Files and versions" page. - Verify shards appear in Dataset Viewer: ```bash curl -s "https://datasets-server.huggingface.co/parquet?dataset=/" ``` Low dependency CLI flow (`npx @huggingface/hub` / `hfjs`): - Set auth token: ```bash export HF_TOKEN= ``` - Upload parquet folder to a dataset repo (auto-creates repo if missing): ```bash npx -y @huggingface/hub upload datasets// ./local/parquet-folder data ``` - Upload as private repo on creation: ```bash npx -y @huggingface/hub upload datasets// ./local/parquet-folder data --private ``` After upload, call `/parquet` to discover `//` values for querying with `@~parquet`. ## Agent Traces The Hub supports raw agent session traces from Claude Code, Codex, and Pi Agent. Upload them to Hugging Face Datasets as original JSONL files and the Hub can auto-detect the trace format, tag the dataset as `Traces`, and enable the trace viewer for browsing sessions, turns, tool calls, and model responses. Common local session directories: - Claude Code: `~/.claude/projects` - Codex: `~/.codex/sessions` - Pi: `~/.pi/agent/sessions` Default to private dataset repos because traces can contain prompts, file paths, tool outputs, secrets, or PII. Preserve the raw `.jsonl` files and nest them by project/cwd instead of uploading every session at the dataset root. ```bash hf repos create / --type dataset --private --exist-ok hf upload / ~/.codex/sessions codex/ --type dataset ``` ## ๐Ÿšจ Edge-Case & Failure Mode Matrix | Scenario | Risk | Production Mitigation | |:---|:---|:---| | **Empty or Null Inputs** | Unhandled exception or unexpected rendering collapse | Enforce fallback guards, optional chaining, and explicit empty state handlers | | **Network Timeout / Latency** | Hanging operations or duplicate side-effects | Implement bounded abort controllers, exponential backoff, and idempotency keys | | **Concurrency / Race Conditions** | Stale state overwrite or inconsistent data mutations | Use atomic transactions, mutex locking, or cancel-on-resubmit controls | | **Invalid Schema / Malformed Payload** | Downstream runtime errors or security injection | Validate boundary payloads with Zod/Pydantic schemas prior to execution | | **Resource / Memory Saturation** | OOM errors, frame drops, or memory leaks | Clean up listeners, cancel active timers, and enforce pagination/virtualization | ## ๐Ÿค– LLM-Specific Traps Table | Anti-Pattern | What AI Commonly Does Wrong | What Is Actually Correct | |:---|:---|:---| | **Platform-Dependent Shell Commands** | Hardcoding $(cat ~/.cache/huggingface/token) or piping to jq | Use platform-agnostic token checks via env var HF_TOKEN and native JSON parsing | | **VRAM OOM Crash** | Loading 70B model weights on a consumer 16GB GPU without quantization | Calculate parameter bounds (fp16 = VRAM/2, Q4 = VRAM*2) and load with bitsandbytes/GGUF | | **Blocking Batch Generation** | Generating large batches synchronously without streaming or yield | Use TextIteratorStreamer or asynchronous generate workers to prevent request timeouts | ## ๐Ÿ›๏ธ Tribunal Verification & Guardrails **Active Reviewers:** `ai-code-reviewer` ยท `python-pro` ยท `performance-optimizer` **Slash Command:** `/review` or `/tribunal-full` ### ๐Ÿ”ฌ Evidence Standard (Tri-State Verification) Every finding, audit statement, or completion claim must classify its factual certainty: - **`[OBSERVED]`**: Directly confirmed in the codebase or verified via executed terminal command. - **`[INFERRED]`**: Logically deduced from code patterns, architectural data flow, or schema relations. - **`[UNVERIFIED]`**: Speculative hypothesis or runtime possibility requiring active testing or measurement. ### โœ… Pre-Flight Self-Audit Checklist ``` โœ… Are model architectures and weights verified to fit within target hardware VRAM budgets? โœ… Are pipeline requests guarded with bounded timeouts and retry backoffs? โœ… Are dataset loading scripts operating in streaming mode to prevent out-of-memory errors? โœ… Are API tokens and cache paths handled portably without assuming UNIX shell environments? โœ… Are tokenizers and generation parameters (max_new_tokens, temperature) strictly bounded? ``` ### ๐Ÿ›‘ Verification-Before-Completion (VBC) Protocol **CRITICAL:** You must follow a strict "evidence-based closeout" state machine. - โŒ **Forbidden:** Declaring a task complete because the output "looks correct." - โœ… **Required:** You are explicitly forbidden from finalizing any task without providing **concrete evidence** (terminal output, passing test suites, compiler success, or equivalent operational proof) that your output works as intended.