Ramy Atalla
← All work

Data · Backend · 2026

Time-records sync, reporting and data pipeline

A platform that syncs a firm's time records and reports billable hours against target, later rebuilt on a data pipeline that matched it entry for entry.

  1. 01Time-tracking systemThird-party source of every time entry.
  2. 02ConnectorSix streams; checkpoints let a five-hour backfill resume.
  3. 03SQL viewsTransform in one place, owned by the pipeline.
  4. 04AppReads only; its database role cannot write pipeline data.
  5. 05ReportsGatePages, email, Excel and PDF from one target engine. Unclassified hours stop the send.
The rebuilt pipeline. Each layer has one owner, and a report that might be wrong is not sent.
routine sync, down from a 2–4 hour full pull
~30 s
streams landed the source's record count
6 of 6
test coverage across 1,170+ tests
97%

Context

The firm logs time in a third-party time-tracking system. Targets follow each person's working calendar, holidays and leave, and a wrong report is worse than none.

My role

Solo, including the pipeline rebuild and its Docker and Kubernetes setup.

What I built

  • Architecture: One target engine feeds pages, emails, Excel and PDF, so numbers agree. The rebuild gives each layer one owner: Airbyte loads, SQL views transform, the app reads.
  • Frontend: Server-rendered pages that work on phones; charts compare any two of the last 26 weeks.
  • Backend: Incremental sync and scheduled Celery report pipelines; email and PDF share one template.
  • Data: A six-stream Python connector whose checkpoints let a five-hour backfill resume after interruption.
  • Security: Role rules decide what each user sees and sends; the PDF renderer fetches only bundled assets; the app's database role cannot write pipeline data.
  • Infrastructure & DevOps: Docker throughout; Airbyte on local Kubernetes, set up by idempotent scripts.
  • Testing: 1,170+ tests pass at 97% coverage; mutation testing exposed two missing tests.

Hard problems and decisions

  • Fail closed. One task holds the lock from sync to last email, and unclassified hours stop the send. Cost: a stopped report is resent by hand.
  • Silent truncation. Stopping at the first empty page lost two-thirds of invoices while the run reported success. The connector now reads until it has the API's stated total, or fails.

Outcome

Routine syncs take about 30 seconds, not a 2–4 hour full pull. The rebuild landed the source's record count on every stream, and over a test week every existing entry matched with identical hours.

Stack

Python, Django, PostgreSQL, Celery, Redis, WeasyPrint, Airbyte, Convex, Docker, pytest, mutmut

Areas

Architecture, Frontend, Backend, Data, Security, Infrastructure & DevOps, Testing

Period

Jul 2026 – present

Next casePermission-aware AI search