Registry by 514

Pipeline Quickstart

Quickstart Example

Follow this 15–30 minute quickstart to run a minimal pipeline using the Factory structures. It extracts sample data from a public API and writes it to local files, showcasing hooks, logging, and error handling.

What you’ll build

  • A tiny TypeScript example under examples/quickstart
  • A pipeline.json definition
  • A connector that reads from the public jsonplaceholder.typicode.com API (no credentials)
  • A runner with a logging hook and basic error handling

Prerequisites

pnpm install

1) Clone and install

git clone https://github.com/514-labs/registry.git
cd registry
pnpm install

2) Explore the example

  • examples/quickstart/pipeline.json — pipeline spec
  • examples/quickstart/src/run.ts — runner
  • examples/quickstart/src/connector/jsonplaceholder.ts — connector
  • examples/quickstart/src/hooks/logging.ts — hooks
  • examples/quickstart/.env.example — env template

3) Run the pipeline

pnpm -C examples/quickstart start

This will:

  • Initialize the connector
  • Fetch a small set of posts
  • Write files under examples/quickstart/data/
  • Log structured request/response events via the logging hook

4) Modify and extend

  • Change the stream from posts to users in pipeline.json
  • Add a transform in src/run.ts to project fields (e.g., id, title)
  • Attach custom hooks (metrics, retries) in src/hooks/logging.ts

5) Troubleshooting

  • Ensure Node 20+ and pnpm are installed
  • Delete examples/quickstart/data/ to reset outputs

Notes

  • This example avoids credentials for zero‑friction usage
  • For real APIs, add environment variables (copy .env.example)