Running Scheduled Pipelines

Many workflows need to run automatically.

Examples include:

• ingesting labour records from field systems
• updating financial ledgers
• reconciling payroll totals

These workflows are executed using scheduled pipelines.

Scheduling Model

Blueprint workflows can be triggered using a cron schedule.

Example configuration:

trigger_type: SCHEDULED
schedule_cron: "0 2 * * *"

This schedule executes the workflow every day at 02:00 UTC.

The scheduler checks for eligible workflows once per minute.

Scheduler Execution Flow

When the scheduler detects a workflow ready to run:

1. The Blueprint is loaded
2. A BlueprintRun record is created
3. StepRun records are generated
4. The job is enqueued for execution

The worker runtime then processes the workflow.

Preventing Overlapping Runs

If a workflow is still executing when the next schedule occurs, the scheduler skips the run.

This prevents duplicate operations such as:

• double financial postings
• duplicate data ingestion

Example Pipeline

A typical scheduled pipeline might look like this:

FETCH → assignar.site_diaries
TRANSFORM → extract_labour_entries
TRANSFORM → normalize_cost_categories
PUSH → business_central.project_ledger

This pipeline ensures labour costs are recorded in the ERP each day.

Observability

Each scheduled workflow produces execution records including:

• run start time
• step execution results
• runtime duration
• failure details

These records allow operators to diagnose problems quickly.

Best Practices

When designing scheduled pipelines:

• ensure workflows are idempotent
• validate records before posting to ERP
• monitor pipeline failures
• run test pipelines in sandbox environments

Following these practices ensures reliable automated execution.

Next Steps

Once scheduled pipelines are configured, organizations typically expand their workflow library.

Common pipelines include:

• payroll reconciliation
• project cost variance detection
• document ingestion for AI retrieval

Over time these pipelines form the operational backbone of system alignment.

---