# Entity Pipeline Pattern (Source → KG → Process)

> Loaded on demand from the Agentled skill. Read this whenever the user wants
> to find leads, source companies, collect contacts, or discover entities to
> act on later. Pairs with `13-entity-pipeline-lifecycle`
> (`agentled examples 13-entity-pipeline-lifecycle`).

When a user asks about **finding leads, sourcing companies, collecting contacts, or discovering any entities they want to act on later**, propose this two-workflow architecture instead of a single monolithic workflow.

## Why

Sourcing and processing have different cadences and costs. Decoupling them lets you:
- Source from many places (LinkedIn, web scrape, Crunchbase, email, webhooks) into one canonical list
- Process (enrich, score, outreach) only new entities, on a schedule, without re-processing already-handled ones
- Retry or re-run either phase independently without touching the other

## Workflow 1 — Sourcing (runs on trigger or schedule)

Finds entities from one or more sources and writes them into a shared KG list with `status: "new"`. Uses `kg.upsert-rows` with a caller-supplied `userKey` on each row so re-runs dedup (O(1), no table scan).

```json
// Step: write sourced entities to KG
{
  "id": "save-to-kg",
  "type": "appAction",
  "name": "Save to KG",
  "app": { "id": "kg", "actionId": "kg.upsert-rows", "source": "native" },
  "stepInputData": {
    "listKey": "sourced-startups",
    // rows must be [{ userKey, rowData }, ...] — userKey is the dedup contract.
    // Use any stable caller-defined id: URL, domain, LinkedIn URL, etc.
    "rows": "{{steps.extract.items}}",
    "mergeStrategy": "merge",   // preserve fields added downstream (scores, notes)
    "status": "new"
  },
  "next": { "stepId": "done" }
}
```

**Key rules:**
- Always include a `status` on sourcing writes (default `"new"`) so the processing workflow can filter on it
- Give every row a stable `userKey` (URL, domain, LinkedIn URL — any caller-defined id that won't change). Same `userKey` in the same list = same DynamoDB row, forever
- Use `mergeStrategy: "merge"` so downstream-added fields (scores, status updates) survive re-upserts from the source
- Multiple sourcing workflows can write to the same `listKey` — they converge into one canonical list
- Use `kg.add-rows` only for one-shot, never-re-run writes where duplicates are acceptable

## Workflow 2 — Processing (runs on schedule, e.g. weekly)

Reads only `status: "new"` rows, processes them (enrich, score, outreach, etc.), then updates status to `"processed"` (or `"scored"`, `"outreached"`, etc.) so they're never picked up again.

```json
// Step 1: read new entities
{
  "id": "read-new",
  "type": "appAction",
  "name": "Read New Entities",
  "app": { "id": "kg", "actionId": "kg.read-list", "source": "native" },
  "stepInputData": {
    "listKey": "sourced-startups",
    "filters": "{\"status\": \"new\"}",
    "limit": "50"
  },
  "next": { "stepId": "process-loop" }
}

// Step 2: loop → enrich / score / outreach each entity
// ... your enrichment and scoring steps here, using {{currentItem.url}} etc. ...

// Step N: mark as processed
{
  "id": "mark-processed",
  "type": "appAction",
  "name": "Mark Processed",
  "app": { "id": "kg", "actionId": "kg.update-rows", "source": "native" },
  "stepInputData": {
    "listKey": "sourced-startups",
    "rowIds": "{{steps.process-loop.processedIds}}",
    "fieldUpdates": "{\"status\": \"processed\"}"
  },
  "next": { "stepId": "done" }
}
```

## Status values (suggested convention)

| Value | Meaning |
|-------|---------|
| `new` | Sourced, not yet processed |
| `scored` | Enriched and scored, not yet outreached |
| `outreached` | Outreach sent |
| `rejected` | Filtered out during scoring |
| `processed` | Generic "done" for non-outreach pipelines |

Use whatever values make sense for the use case — the pattern is the same.

## When to propose this pattern

Suggest it whenever the user says any of:
- "find leads / companies / contacts"
- "source startups / investors / candidates"
- "collect entities from multiple sources"
- "build a list I can act on later"
- "score / enrich / outreach to a list"
- "run this once a week on new items"

The default answer is **two workflows**: one that sources into KG, one that processes from KG. A single do-everything workflow is only appropriate when the user has a fixed one-shot input and no recurring need.