Stop Polling: Build File-Driven Workflows with Unmeshed

What is file watching?

File watching is the ability to detect when something changes on a filesystem — a file is created, modified, or deleted — and react to it immediately. Instead of periodically asking "is there a new file yet?", a file watcher subscribes to OS-level filesystem events and gets notified the instant something happens.

Under the hood, most file watchers rely on kernel APIs like inotify on Linux, FSEvents on macOS, or ReadDirectoryChangesW on Windows. These APIs push events to your process rather than requiring your process to repeatedly scan a directory. The result is zero-latency detection with near-zero CPU cost while waiting.

File watching is used everywhere you need to bridge the gap between a file appearing and a process acting on it — log pipelines, batch imports, SFTP drop zones, report generation, audit trails, and more.

Why do teams reach for file watching?

The common thread across all file-watching use cases is the same: a file is the signal. Some upstream system — a legacy job, a vendor feed, an internal service — produces a file, and your system needs to know about it fast.

A few real scenarios where this comes up:

• An overnight batch job writes a CSV to a shared directory at 02:00. A downstream pipeline needs to pick it up the moment it lands.
• An order management service writes a .log file for every processed order. A monitoring system needs to catch those logs in real time.
• An SFTP client deposits inbound documents from a partner. Processing should start immediately, not at the next scheduled run.
• A payment processor generates an invoice file after charging a customer. That file needs to trigger a confirmation email and an audit entry.

In every case, the requirement is the same: react the instant the file appears, not minutes later.

How it's done traditionally

Most teams solve this with one of two approaches, and both have real problems.

The cron job

The classic approach: set up a cron job that runs every minute (or every few seconds if you push it), lists a directory, and processes any new files it finds.

# /etc/cron.d/watch-orders
* * * * * /usr/local/bin/check-for-new-orders.sh

This works, but it introduces a latency floor. If your cron runs every 60 seconds, you are always up to a minute behind. Push it to every 10 seconds and you are burning CPU on a tight loop that almost always finds nothing. More importantly, the logic lives outside your orchestration layer — in a shell script somewhere that nobody's watching.

The polling loop

The more aggressive version: a long-running process that wakes up every few seconds and scans the directory.

import os, time

while True:
    files = os.listdir("/logs/orders")
    for f in files:
        if f.endswith(".log") and not already_processed(f):
            process(f)
    time.sleep(5)

This reduces latency but introduces new problems. The script needs to track what it has already seen. It needs to handle crashes and restarts. It consumes resources constantly. And it has no visibility into the rest of your workflow — there's no execution history, no retry logic, no alerting that fits naturally into the wider system.

Why both fall short

The deeper problem with cron jobs and polling loops is that the orchestration logic ends up outside the orchestrator. You end up with a parallel system of shell scripts, cron entries, and one-off daemons that nobody has full visibility into. When something breaks, it's hard to trace. When something needs to change, it's hard to find.

How it's done with Unmeshed

Unmeshed treats file watching as a first-class workflow step, not an external trigger bolted on the side.

Instead of a cron job pointing at a script, you add a WORKER step with filewatcher.agent directly inside your process definition. The step blocks on a real OS-level filesystem event. When the file arrives, the step completes and the workflow continues — exactly like any other step.

The configuration is minimal:

{
  "name": "filewatcher.agent",
  "type": "WORKER",
  "ref": "watch_log_creation",
  "input": {
    "type": "FILE_WATCHER",
    "directory": "/logs/orders",
    "fileNamePattern": "*.log",
    "watchCriteria": "ENTRY_CREATE",
    "watchDuration": 20000
  }
}

Because this is a step inside a workflow, it automatically gets everything else Unmeshed provides: execution history, retry policies, parallel branching, secret management, and observability. No separate infrastructure to maintain.

Example: the order_processing workflow

Let's walk through a real workflow that shows file watching in the context of a complete process. The order_processing workflow handles the full lifecycle of an order — generating an ID, validating it, charging the customer, and then running three things in parallel: watching for the order log, writing the log, and generating the invoice.

Workflow shape

generate_order_id
  → validate_order
  → process_payment
  → parallel:
      ├── watch_log_creation   (file watcher — blocks until *.log appears)
      ├── write_order_log      (bash — writes the order log file)
      └── generate_invoice     (bash — writes the invoice file)

The file watcher runs in parallel with the steps that produce the files it's watching for. This is the key design pattern: start listening before writing, so the watcher never misses the event.

Step 1 — Generate an order ID

The first step creates a unique, timestamp-based order ID that flows through the rest of the workflow.

(steps, context) => {
  const orderId = 'ORD-' + Date.now();
  return { orderId };
}

Every downstream step references this ID — the log file is named after it, the invoice references it, and the payment call sends it to the gateway. Starting with a deterministic ID keeps the workflow traceable from start to finish.

Step 2 — Validate the order

Before touching any external system, the workflow validates the order. If validation fails, the process throws and stops here — no payment is attempted, no files are written.

(steps, context) => {
  const isValid = true; // replace with real validation logic
  if (!isValid) throw Error('Invalid Order');
  return { status: 'VALID' };
}

In a real implementation this step would check inventory, verify the customer account, validate the cart total, and so on. The pattern — validate before acting — keeps the rest of the workflow clean.

Step 3 — Process payment

Once the order is valid, the workflow calls the payment gateway over HTTP. Notice how the order ID and amount flow in from earlier steps via template expressions, and the API token comes from secrets — never hardcoded.

{
  "name": "process_payment",
  "type": "HTTP",
  "ref": "process_payment",
  "input": {
    "method": "POST",
    "url": "https://api.payment-gateway.com/pay",
    "headers": {
      "Content-Type": "application/json",
      "Authorization": "Bearer {{ secrets.apiGatewayToken }}"
    },
    "body": {
      "type": "json",
      "content": {
        "orderId": "{{ steps.generate_order_id.output.result.orderId }}",
        "amount": "{{ context.input.amount }}"
      }
    }
  }
}

Only after this step completes successfully does the workflow move into the parallel phase.

Step 4 — Parallel: watch, write, invoice

This is where it gets interesting. After payment succeeds, three things need to happen — and the order within those three things matters: the watcher should start before the log is written, but all three can run in the same parallel block because Unmeshed handles the sequencing via event timing.

Branch A — Watch for the order log

{
  "name": "filewatcher.agent",
  "type": "WORKER",
  "ref": "watch_log_creation",
  "input": {
    "type": "FILE_WATCHER",
    "directory": "/logs/orders",
    "fileNamePattern": "*.log",
    "watchCriteria": "ENTRY_CREATE",
    "watchDuration": 20000
  }
}

This step subscribes to ENTRY_CREATE events in /logs/orders. It will complete the moment any *.log file appears in that directory, or time out after 20 seconds if nothing arrives.

Branch B — Write the order log

#!/bin/bash
LOG_DIR="/logs/orders"
ORDER_ID="{{ steps.generate_order_id.output.result.orderId }}"
mkdir -p "$LOG_DIR"
echo "Order processed: $ORDER_ID" >> "$LOG_DIR/$ORDER_ID.log"
echo "Log written"

This bash step creates the log file that Branch A is watching for. Because both branches start at the same time, the watcher is already listening by the time this step writes the file. The OS ENTRY_CREATE event fires as soon as the file is closed, and the watcher step completes.

Branch C — Generate the invoice

#!/bin/bash
INVOICE_DIR="/invoices"
ORDER_ID="{{ steps.generate_order_id.output.result.orderId }}"
mkdir -p "$INVOICE_DIR"
echo "Invoice for $ORDER_ID" > "$INVOICE_DIR/$ORDER_ID.txt"
echo "Invoice generated"

Invoice generation runs independently of the log watcher. The parallel block is configured with failIfAnyBranchFails: false, which means a problem in one branch does not kill the others — invoice generation and log watching are independent concerns.

Full process definition

Here is the complete order_processing workflow you can import directly into Unmeshed:

{
  "orgId": 1,
  "namespace": "default",
  "name": "order_processing",
  "version": 1,
  "type": "API_ORCHESTRATION",
  "steps": [
    {
      "name": "generate_order_id",
      "type": "JAVASCRIPT",
      "ref": "generate_order_id",
      "input": {
        "script": "(steps, context) => {\n  const orderId = 'ORD-' + Date.now();\n  return { orderId };\n}"
      }
    },
    {
      "name": "validate_order",
      "type": "JAVASCRIPT",
      "ref": "validate_order",
      "input": {
        "script": "(steps, context) => {\n  const isValid = true;\n  if (!isValid) throw Error('Invalid Order');\n  return { status: 'VALID' };\n}"
      }
    },
    {
      "name": "process_payment",
      "type": "HTTP",
      "ref": "process_payment",
      "input": {
        "method": "POST",
        "url": "https://api.payment-gateway.com/pay",
        "headers": {
          "Content-Type": "application/json",
          "Accept": "application/json",
          "Authorization": "Bearer {{ secrets.apiGatewayToken }}"
        },
        "body": {
          "type": "json",
          "content": {
            "orderId": "{{ steps.generate_order_id.output.result.orderId }}",
            "amount": "{{ context.input.amount }}"
          }
        }
      }
    },
    {
      "name": "parallel",
      "type": "PARALLEL",
      "ref": "parallel_1",
      "input": {
        "failIfAnyBranchFails": false
      },
      "children": [
        {
          "name": "filewatcher.agent",
          "type": "WORKER",
          "ref": "watch_log_creation",
          "input": {
            "type": "FILE_WATCHER",
            "directory": "/logs/orders",
            "fileNamePattern": "*.log",
            "watchCriteria": "ENTRY_CREATE",
            "watchDuration": 20000
          }
        },
        {
          "name": "bash",
          "type": "WORKER",
          "ref": "write_order_log",
          "input": {
            "type": "BASH_COMMAND",
            "command": "#!/bin/bash\nLOG_DIR=\"/logs/orders\"\nORDER_ID=\"{{ steps.generate_order_id.output.result.orderId }}\"\nmkdir -p \"$LOG_DIR\"\necho \"Order processed: $ORDER_ID\" >> \"$LOG_DIR/$ORDER_ID.log\"\necho \"Log written\"",
            "streamOutput": true
          }
        },
        {
          "name": "bash",
          "type": "WORKER",
          "ref": "generate_invoice",
          "input": {
            "type": "BASH_COMMAND",
            "command": "#!/bin/bash\nINVOICE_DIR=\"/invoices\"\nORDER_ID=\"{{ steps.generate_order_id.output.result.orderId }}\"\nmkdir -p \"$INVOICE_DIR\"\necho \"Invoice for $ORDER_ID\" > \"$INVOICE_DIR/$ORDER_ID.txt\"\necho \"Invoice generated\"",
            "streamOutput": true
          }
        }
      ]
    }
  ]
}

Tips and pointers

Start the watcher before the writer. The parallel block starts all branches simultaneously, so the watcher is already listening when the bash step writes the file. If you put the watcher after the writer in a sequential flow, you risk missing the event entirely. Always subscribe before producing.

Use watchDuration as a safety net, not a timer. Set watchDuration to a generous timeout (20–60 seconds) that represents a reasonable upper bound for how long the file should take to appear. If the watcher times out, treat it as a signal that something upstream went wrong — not a normal code path.

Match your pattern precisely. *.log will catch ORD-1234567890.log, but it will also catch anything-else.log. If you are watching a shared directory used by multiple processes, use a tighter pattern like ORD-*.log to avoid false triggers.

failIfAnyBranchFails: false is deliberate here. Invoice generation and log watching are independent concerns. If the invoice step fails, you still want the watcher to complete and vice versa. Use this setting when your parallel branches represent separate responsibilities that should not block each other.

File watching belongs in the workflow, not next to it. The moment you move the watcher out of the workflow — into a cron job, a separate daemon, or a shell script — you lose execution history, retry handling, and visibility. Keep it inside Unmeshed so the event shows up on the same timeline as every other step.

Pair with a cleanup step for idempotency. In production, consider adding a step after the parallel block to archive or move processed files. This prevents the watcher from re-triggering on the same file if the workflow is re-run, and keeps the drop zone clean for the next order.

Where to take this next

The pattern shown here — validate, act, then watch and react in parallel — extends naturally to a wide range of file-based integrations:

• Swap /logs/orders for an SFTP mount and *.log for *.csv to build an inbound file processing pipeline.
• Add a downstream HTTP step after the parallel block to notify an external system once the log is confirmed written.
• Combine with wait steps when the file arrival is one of several signals you need to coordinate before continuing.
• Use secret inputs via webhook to trigger the whole workflow from an external event and pass credentials securely.

Final thoughts

File-based events are everywhere — order logs, payment receipts, batch exports, SFTP drops. The difference between a fragile setup and a clean one is not the complexity of the code. It is whether the file event is a first-class part of your workflow or an afterthought living in a cron tab nobody owns.

With filewatcher.xyz inside Unmeshed, you get instant detection, full execution history, and zero polling infrastructure — all wired directly into the same orchestration layer that handles your API calls, secrets, and branching logic.

No polling. No cron jobs. The workflow simply waits for the file and continues.