Pipeline Quickstart

Quickstart Example

Follow this 15–30 minute quickstart to run a minimal pipeline using the Factory structures. It extracts sample data from a public API and writes it to local files, showcasing hooks, logging, and error handling.

What you’ll build

• A tiny TypeScript example under examples/quickstart
• A pipeline.json definition
• A connector that reads from the public jsonplaceholder.typicode.com API (no credentials)
• A runner with a logging hook and basic error handling

Prerequisites

pnpm install

1) Clone and install

git clone https://github.com/514-labs/registry.git
cd registry
pnpm install

2) Explore the example

• examples/quickstart/pipeline.json — pipeline spec
• examples/quickstart/src/run.ts — runner
• examples/quickstart/src/connector/jsonplaceholder.ts — connector
• examples/quickstart/src/hooks/logging.ts — hooks
• examples/quickstart/.env.example — env template

3) Run the pipeline

pnpm -C examples/quickstart start

This will:

• Initialize the connector
• Fetch a small set of posts
• Write files under examples/quickstart/data/
• Log structured request/response events via the logging hook

4) Modify and extend

• Change the stream from posts to users in pipeline.json
• Add a transform in src/run.ts to project fields (e.g., id, title)
• Attach custom hooks (metrics, retries) in src/hooks/logging.ts

5) Troubleshooting

• Ensure Node 20+ and pnpm are installed
• Delete examples/quickstart/data/ to reset outputs

Notes

• This example avoids credentials for zero‑friction usage
• For real APIs, add environment variables (copy .env.example)