# Overview

right now, workflows are structured like:

```
mydir/
mydir/sets/
mydir/tasks/
mydir/workflow.yaml
```
and you can run `cmte cmte-dir` and it will run the workflow

But sometimes a core workflow is really useful for lots of types of things that just require different contexts. in fact, a workflow is really for configuring a context for tasks!

So what if we did this:

```
mydir/sets/
mydir/tasks/
mydir/something.context.yaml
mydir/somethingelse.context.yaml
mydir/another.context.yaml
mydir/onemore.context.yaml
```
All in one dir and sets and tasks are just a common dir by default so you can use them for anything based on the configuration you set up in. the workflows are named instead of called `workflow.yaml` and if you said `cmte mydir` it would run all workflows in parallel or you could run them individually and if you need a specific custom set or task, you just make one and keep it in the set/task dir

## Subsequent discussion

> Does each .context.yaml need a unique outputPath
yes

> Parallel Execution?
we already have the ability to execute multiple workflows in one command

> Mandating unique outputPath in each *.context.yaml.
Just set outputPath by convention to a dir with the name of the context like: `myworkflow.context.yaml` » `myworkflow-output/outputfiles.md`

This is the dir structure. workflows (contexts), sets, and tasks stay where they are:

mydir/sets/
mydir/tasks/
mydir/something.context.yaml
mydir/somethingelse.context.yaml
mydir/another.context.yaml
mydir/onemore.context.yaml

> Your suggestion is to "just make one and keep it in the set/task dir". This works if the custom task/set has a unique name. 
it's simpler than this --- if you need a custom task (even slightly modified -- unless that mod can be handled in an iterable_object, whch frankly should be able to do a lot!), then you need a custom set file. that's it. name them both uniquely. you can still use the others. maybe eventually we add parameters for configuring slightly different variants of sets, but for now let's keep it simple. 

 > Error Handling in Parallel
how it works now is fine -- it's imperfect but it's fine

> Cross-context dependencies seem out of scope and undesirable
You can't cross contexts. That's it. If you want sets to have a shared context, run them in the same context. this is actually part of why i want to name them 'contexts' instead of 'workflows' -- they set the boundaries of the context used.

> migration path?
It's pretty straightforward. in literally the only current use case of this (mine) it would allow us to delete like 40 files that largely duplicate other files!

# Plan

Here is a plan for implementing the shift from self-contained `workflow.yaml` directories to shared `sets`/`tasks` directories with multiple `*.context.yaml` files:

**Implementation Plan: Context-Based Workflow Execution**

1.  **Refine Terminology & Configuration Schema:**
    *   Internally and in documentation, consistently use "context" file instead of "workflow" file where appropriate.
    *   Define the standard file pattern for context files (e.g., `*.context.yaml`).
    *   Confirm the structure within `*.context.yaml` remains identical to the current `workflow.yaml` (containing `name`, `description`, `outputPath` (optional), `files`, `global_variables`, `iterable_objects`, `sets`).

2.  **Update CLI Handling:**
    *   Modify the CLI entry point to accept either a directory path or a specific context file path (`*.context.yaml`).
    *   **Directory Input (`cmte mydir`):**
        *   Implement logic to scan the specified directory (`mydir`) for files matching the `*.context.yaml` pattern.
        *   Pass the list of discovered context file paths to the execution engine.
    *   **File Input (`cmte mydir/something.context.yaml`):**
        *   Validate the input is a file matching the pattern.
        *   Pass this single file path to the execution engine.
    *   Update CLI help messages to reflect the new usage patterns.

3.  **Adapt Execution Engine:**
    *   Modify the engine that currently processes a single `workflow.yaml` to accept a list of context file paths.
    *   Leverage the existing parallel execution capabilities to process each context file concurrently.
    *   Ensure each parallel execution instance is correctly initialized with the data loaded from its corresponding `*.context.yaml`.

4.  **Implement Output Path Strategy:**
    *   For each context file being processed:
        *   Check if `outputPath` is explicitly defined within the `.context.yaml`. If yes, use it.
        *   If `outputPath` is *not* defined, automatically generate a unique path based on the context filename (e.g., `myworkflow.context.yaml` results in output being directed to `./myworkflow-output/` relative to the context file's location).
        *   Ensure the file/directory operations within the execution respect this determined output path for each parallel context run.

5.  **Verify/Solidify Path Resolution:**
    *   **Crucial:** Confirm that when processing `mydir/something.context.yaml`, any `useSet: my-set` or `useTask: my-task` reference correctly and consistently resolves to `mydir/sets/my-set.set.yaml` and `mydir/tasks/my-task.md` respectively.
    *   If the current logic doesn't behave this way (e.g., if it resolves relative to CWD or the context file itself), adjust the path resolution logic to always base `sets/` and `tasks/` lookups on the *directory containing* the `.context.yaml` file being processed.

6.  **Documentation Overhaul:**
    *   Update `README.md` and any other documentation:
        *   Introduce the new recommended project structure.
        *   Explain the concept of shared `sets/` and `tasks/` directories.
        *   Clearly define the role and content of `*.context.yaml` files.
        *   Document the new CLI commands (`cmte <dir>` and `cmte <context_file>`).
        *   Explain the automatic output path generation and how to override it.
        *   Provide updated examples reflecting the new structure.
        *   Reiterate the strategy for custom logic (unique set/task names, leveraging `iterable_objects`).
        *   Explicitly state that contexts are execution boundaries and cross-context dependencies are not supported.

7.  **Testing:**
    *   Develop test cases covering:
        *   Execution via directory path (multiple contexts).
        *   Execution via single context file path.
        *   Correct path resolution for shared sets/tasks from different context files.
        *   Default output path generation.
        *   Explicit `outputPath` usage.
        *   Error handling in parallel runs (confirming it matches the existing behavior).
        *   A simple migration test (renaming `workflow.yaml`, running).

This plan focuses on leveraging existing parallel capabilities while adjusting discovery, configuration loading, output handling, and ensuring correct path resolution for the shared logic components.