How to Get Started with Prefect 3

Prefect 3 lets you run any Python function as a monitored, retryable workflow without changing how you write code. Add two decorators, run one CLI command, and your script goes from something you trigger by hand to something that runs on a schedule, logs every execution, and retries on failure automatically. This guide walks through setting up Prefect 3 locally, writing your first flow, and scheduling it to run on its own.

What Prefect 3 Does

Prefect is an open-source workflow orchestration framework. It wraps Python functions with observability and resilience features: automatic retry logic, execution logging, scheduling, and a web UI to monitor runs. Version 3, released in late 2024, added transactional semantics, native event-driven triggers, and a significantly faster local server.

The core mental model is simple: a flow is the top-level orchestration function. A task is any Python function inside the flow that you want to track independently. Both are created with decorators. Everything else, from scheduling to retries, is configuration.

Prefect is used by over 30,000 teams. According to the project's GitHub repository, it has accumulated 17,000+ stars. The framework positions itself specifically against Apache Airflow for teams that want Python-first orchestration without a Kubernetes cluster.

Step 1: Install Prefect

Prefect 3 requires Python 3.10 or newer. Start by creating a virtual environment:

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install prefect

Verify the install:

prefect version

You should see output showing version 3.x. No database setup, no Docker, no config files required at this stage. Prefect starts with a local SQLite database by default.

Step 2: Write Your First Flow

Create a file called pipeline.py:

from prefect import flow, task
import requests

@task(retries=3, retry_delay_seconds=10)
def fetch_data(url: str) -> dict:
    response = requests.get(url)
    response.raise_for_status()
    return response.json()

@task
def process_data(data: dict) -> int:
    # Example: count records in a response
    records = data.get("results", [])
    print(f"Processed {len(records)} records")
    return len(records)

@flow(name="my-first-pipeline", log_prints=True)
def run_pipeline():
    data = fetch_data("https://jsonplaceholder.typicode.com/posts")
    count = process_data(data)
    return count

if __name__ == "__main__":
    run_pipeline()

Run it:

python pipeline.py

Prefect logs each task's state: Running, Completed, or Failed. The retries=3 parameter on fetch_data means if the HTTP request fails, Prefect waits 10 seconds and tries again up to three times before marking the task as failed.

The log_prints=True flag on the flow captures any print() calls into the Prefect log stream, so you can see them in the UI without switching to a different logging system.

Step 3: Open the Local UI

Prefect includes a local web server for monitoring runs:

prefect server start

Open http://127.0.0.1:4200 in a browser. You will see a dashboard with your recent runs, task states, and logs. For teams running flows in production, this local server can be self-hosted on any machine.

While the server is running in one terminal, run your flow again from a second terminal. The run will appear in the UI in real time, with task-level state transitions and log output.

Step 4: Schedule a Flow

To run a flow on a schedule, convert it to a deployment. A deployment is a registered version of your flow with scheduling and configuration attached.

Add a serve call to your script:

from prefect import flow, task
from prefect.schedules import CronSchedule

@task(retries=3, retry_delay_seconds=10)
def fetch_data(url: str) -> dict:
    import requests
    response = requests.get(url)
    response.raise_for_status()
    return response.json()

@task
def process_data(data: dict) -> int:
    records = data.get("results", [])
    print(f"Processed {len(records)} records")
    return len(records)

@flow(name="my-first-pipeline", log_prints=True)
def run_pipeline():
    data = fetch_data("https://jsonplaceholder.typicode.com/posts")
    count = process_data(data)
    return count

if __name__ == "__main__":
    run_pipeline.serve(
        name="daily-pipeline",
        cron="0 9 * * *",  # Every day at 9 AM
    )

Run the script. Prefect registers the deployment and keeps the process alive as a worker, executing the flow according to the cron schedule. You can verify the scheduled run in the local UI under Deployments.

Step 5: Add Parameters

Flows can accept parameters, which lets you run the same flow against different inputs from the CLI or the UI:

@flow(name="my-first-pipeline", log_prints=True)
def run_pipeline(url: str = "https://jsonplaceholder.typicode.com/posts"):
    data = fetch_data(url)
    count = process_data(data)
    return count

You can now trigger a run with a specific URL from the CLI:

prefect deployment run 'my-first-pipeline/daily-pipeline' --param url=https://jsonplaceholder.typicode.com/users

Parameters appear in the UI as a form, which means non-technical users can trigger parameterized runs without touching code.

Step 6: Connect to Prefect Cloud (Optional)

Prefect Cloud is the managed version with persistent history, team access controls, and a free tier of 500 flow runs per month. To connect:

1. Create a free account at app.prefect.cloud.
2. Generate an API key in your workspace settings.
3. Authenticate locally:

prefect cloud login --key YOUR_API_KEY

All subsequent flow runs will appear in Prefect Cloud instead of the local server. Your code does not change. The switch is purely at the authentication layer.

The free tier is sufficient for most individual analysts or small teams running fewer than 500 automated jobs per month. Paid plans add SSO, audit logs, and higher run limits.

What Prefect 3 Adds Over Version 2

The most significant addition in Prefect 3 is transactional semantics. You can now wrap a group of tasks in a Transaction block, and if any task inside it fails, the whole group rolls back. This matters for pipelines that write to databases or external APIs, where partial success can leave data in an inconsistent state.

Event-driven triggers are also new in version 3. A flow can be configured to fire when a specific Prefect event occurs, such as another flow completing or a data asset being updated, rather than only on a fixed cron.

If you are already on Prefect 2, migration to Prefect 3 is mostly backward-compatible for the core @flow and @task decorator patterns. The Deployment API has changed, so deployments need to be updated, but existing flow logic runs without modification.

When VSLZ Is Faster

If you are running Prefect to automate data analysis, reporting, or exploration, VSLZ handles the analytical layer without requiring a pipeline at all. Upload a file or connect a data source, describe what you need in plain English, and the agent produces the analysis in one prompt. For teams that need repeatable pipeline logic across many systems, Prefect is the right tool. For exploratory or ad-hoc data work, the pipeline overhead is unnecessary.

Summary

Prefect 3 is the fastest way to turn a Python script into a production-grade, scheduled workflow. The install is one command, the core abstractions are two decorators, and the free Prefect Cloud tier covers most individual use cases. Start by wrapping an existing script you run manually, add retries where network calls are involved, and schedule it. From there you can add parameters, event triggers, and transactions as complexity grows.