Linux From Cron to DAGs: Why You Need an Orchestrator and Why Airflow

[dEEpEst]

Created for Hack Tools Dark Community

From Cron to DAGs: Why You Need an Orchestrator and Why Airflow

Imagine a simple scenario:

1. Every night you need to:
- Unload your database;
- Generate a report;
- Send the result to S3.

With just a few steps, this seems easy to manage with crontab. But sooner or later:
- One task goes slower than usual,
- Another script starts too early,
- A report is generated empty...

And now you’re duct-taping the process with `sleep`, `if-else`, and manual alerting scripts. This quickly snowballs into a fragile mess that is hard to control or debug.

Enter Airflow. What makes it different?

Instead of writing independent scheduled jobs, Airflow models your workflow as a DAG (Directed Acyclic Graph):
- Each task is a node,
- Each dependency is an arrow,
- Scheduling becomes unified and centralized.

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta

with DAG(
    dag_id="nightly_pipeline",
    schedule_interval="0 2 * * *",
    start_date=datetime(2025, 4, 1),
    catchup=False,
) as dag:

    dump = BashOperator(
        task_id="dump_db",
        bash_command="/scripts/dump.sh",
        retries=2,
        retry_delay=timedelta(minutes=10),
        on_failure_callback="notify_telegram"
    )

    transform = BashOperator(
        task_id="make_report",
        bash_command="/scripts/report.sh"
    )

    upload = BashOperator(
        task_id="upload_s3",
        bash_command="/scripts/upload.sh"
    )

    dump >> transform >> upload

What changes in practice:

Automatic Retries: No more modifying shell scripts — just declare `retries`.

Visual UI for Monitoring: Know exactly which task is running (yellow), succeeded (green), or failed (red) in the browser.

Built-in Notifications: on_failure_callback lets you instantly notify via Telegram, Slack, etc.

Event Sensors: Stop using `while sleep 30`. You can wait until a file appears, a partition is ready, or an API responds.

Let’s visualize a real failure:

Suppose the DB dump normally takes 5 minutes. One night, due to high load, it takes 30 minutes. If you’re using `cron`, the `report.sh` might run at 02:05 — reading an incomplete dump and generating an empty report.

With Airflow, `make_report` won’t even start until `dump_db` finishes. If `dump_db` fails twice, the whole DAG is marked as failed and you get a Telegram alert with full logs.

Why Cron Isn’t Enough:
- Great for standalone jobs.
- Fails silently unless you add manual checks.
- No centralized logging or visualization.

Why Airflow Shines:
- Centralized control and dependency resolution via DAG.
- Written entirely in Python.
- Logs, retries, alerting, backfilling — all built-in.

Next Post Teaser:
We'll deploy Airflow using Docker Compose and build our first “Hello DAG” workflow.

Disclaimer:
This post is intended for educational and research purposes only. Any misuse of orchestration tools in production environments without proper monitoring, authorization, or security controls may lead to serious consequences.

Join the discussion below and share your Airflow tips, nightmares, or success stories!
​