NYC Taxi Ingestion Pipeline

Overview

A Dockerised ingestion job that loads monthly NYC TLC yellow-taxi CSVs into PostgreSQL in fixed-size chunks, so peak memory stays flat regardless of file size. Built while working through the DataTalksClub Data Engineering Zoomcamp Docker module, then tidied into a standalone job.

Problem

Each monthly TLC file contains millions of rows. Reading one into a single dataframe and writing it in one transaction loads the whole month into memory and fails as the files grow.

Technical solution

pandas yields 100,000-row chunks. An explicit dtype map — including nullable Int64 for integer columns that may contain nulls — keeps every chunk’s schema identical, which is what stops a mid-job insert from failing when inference would have changed a type. The first write creates the table from the schema without inserting rows; later chunks append. The image builds from a committed uv lockfile.

What is in the repository

• Chunked CSV streaming over HTTP from the public TLC release files
• Explicit dtypes and parsed timestamps so chunk schemas stay stable
• Docker Compose stack: PostgreSQL 18 with a healthcheck, plus pgAdmin
• Credentials from environment variables, not argv (visible in the process table)
• Reproducible image via a committed uv lockfile

Technical decisions

Chunk size is a memory budget, not a tuning hobby

100,000 rows is large enough to keep the database busy and small enough that the process stays predictable on a laptop.

Replace once, then append

Re-running the job rebuilds the table instead of duplicating rows. That is the right default for a workshop-sized pipeline.