DocFlow

OCR and model calls take seconds to minutes; an HTTP request shouldn’t. DocFlow is a reusable reference implementation of the asynchronous pattern I ran in production at IFAD, rebuilt from scratch in the open.

A FastAPI backend where users upload PDFs, DOCX files or images and ask for a summary, invoice extraction or contract metadata. The upload returns straight away; Celery workers on Redis do the heavy work, and PostgreSQL tracks every request and attempt.

• JWT authentication, per-user documents and files, and email-based password reset.
• Originals stored in MinIO (S3-compatible); metadata, requests, job attempts and results in PostgreSQL, with Alembic migrations.
• Text extraction with pypdf, python-docx and Tesseract OCR, then OpenAI for the requested analysis.
• Everything runs locally with Docker Compose: API, worker, PostgreSQL, Redis and MinIO.

The questions a reviewer should ask of any job system, and how DocFlow answers them today:

• Why Celery and Redis? A mature Python task queue with retries and backoff built in, and Redis doubles as broker and result backend, so the local stack stays small.
• How does the client know a job finished? Each request moves QUEUED → PROCESSING → COMPLETED or FAILED in PostgreSQL. Clients poll the status endpoint, and the owner can get an email when it finishes.
• What happens when a job fails? It re-raises, so Celery retries it up to three times with backoff; errors a retry can’t fix fail at once. Every attempt is recorded with its number and a failure reason.
• How are duplicate results prevented? The database allows exactly one stored result per processing request, and invoice and contract output is validated against Pydantic models.
• What happens when a worker dies mid-job? Today the request stays in PROCESSING: tasks are acknowledged when received, so a crashed worker’s task isn’t redelivered. That is the first thing on the list below.

Written down deliberately. These are the gaps between a working reference and a production service:

• Make the task claim its request atomically, so running it twice is harmless, then add a sweeper that recovers requests stuck in PROCESSING or QUEUED.
• Bound every job: a page cap and page-by-page OCR for scanned PDFs, and soft and hard time limits on the task.
• Idempotency keys on upload, so a client retrying a slow request can’t create a second job.
• Push completion to integrations with signed webhooks instead of polling.