DocFlow

OCR and model calls take seconds to minutes; an HTTP request shouldn’t. DocFlow is a reusable reference implementation of the asynchronous pattern I ran in production at IFAD, rebuilt from scratch in the open.

A FastAPI backend where users upload PDFs, DOCX files or images and ask for a summary, invoice extraction or contract metadata. The upload returns straight away; Celery workers on Redis do the heavy work, and PostgreSQL tracks every request and attempt.

• JWT authentication, per-user documents and files, and email-based password reset.
• Originals stored in MinIO (S3-compatible); metadata, requests, job attempts and results in PostgreSQL, with Alembic migrations.
• Text extraction with pypdf, python-docx and Tesseract OCR, then OpenAI for the requested analysis.
• Everything runs locally with Docker Compose: API, worker, PostgreSQL, Redis and MinIO.

The questions a reviewer should ask of any job system, and how DocFlow answers them today:

• Why Celery and Redis? A mature Python task queue with retries and backoff built in, and Redis doubles as broker and result backend, so the local stack stays small.
• How does the client know a job finished? Each request moves QUEUED → PROCESSING → COMPLETED or FAILED in PostgreSQL. Clients poll the status endpoint; the result endpoint returns 404 until the result exists.
• How are duplicate results prevented? The database allows exactly one stored result per processing request, and every attempt is recorded with its attempt number.
• What happens when a worker dies mid-job? Today the request stays in PROCESSING: tasks are acknowledged when received, so a crashed worker’s task isn’t redelivered. That is the first thing on the list below.

Written down deliberately. These are the gaps between a working reference and a production service:

• Late acknowledgement (acks_late + reject_on_worker_lost) plus a sweeper that re-queues requests stuck in PROCESSING.
• Let failures reach Celery’s retry policy: the task currently records a failure instead of re-raising it, so its declared retries never fire.
• Idempotency keys on upload, so a client retrying a slow request can’t create a second job.
• Push completion (webhook or server-sent events) instead of polling, and validated structured output for invoices and contracts.