← The Upskill Initiative · ResourcesSupport labs

Support lab · tested 12 Oct 2026

Tenant Todo Lab

Build one multi-tenant to-do API, then run it, cache it, ship it to Kubernetes, watch it in Grafana, shard it with Vitess, add a Temporal workflow and stream events through Kafka. Every step lists the errors you will actually hit and how to fix them.

FastAPIPostgresRedisDockerKubernetesPrometheus + GrafanaLokiTempo + OpenTelemetryPgBouncerVitess MySQLTemporalKafka

8sessions (Day 4 is split in two)

~21.5 hin total

12tools

1app, grown daily

Before you start

What ~21.5 hours gets you. You will be able to build this stack, explain how each piece works, and fix the 6–10 most common failures in each tool. Being truly advanced takes months of real incidents. This plan gives you the mental model and the first incidents.
Memory is the hard limit. Everything was tested with Docker limited to 4 GB. Run one day's stack at a time, and stop the previous day's containers before starting the next. On Windows, Docker's memory is set in WSL (see Set up your computer).
Temporal and Kafka have their own days. Day 6 builds a real feature with Temporal: an admin bulk-assigns todos from a CSV, and the workflow survives crashes and retries without duplicates. Day 7 adds Kafka: an activity feed that survives consumers going down. Day 5 is Vitess only.
Works on macOS, Windows and Linux. Commands are written for a bash terminal: Terminal on macOS, your shell on Linux, and an Ubuntu terminal under WSL 2 on Windows. Where Windows PowerShell differs, a Windows note says so.
Day 4 has two parts. 4a covers metrics and database incidents (3 h). 4b covers logs, stack traces and traces (2 h).
Database skills are spread across the days. Each one appears where the app first needs it (table below). Adding them made each day run about 5–10 minutes over 3 hours.
Tested, not guessed. On 12 Oct 2026 every day of this lab was run end to end on macOS (Apple Silicon, Docker Desktop, 4 GB for Docker). Linux-only steps were checked in Ubuntu 24.04, and the PowerShell commands were checked with PowerShell 7.6. Green-dot boxes are real output from those runs; your ids, times and row counts will differ. Grey-dot boxes describe what to look for.

Database concepts, and where you practise them

Concept	What you actually do	Day
Transactions, row-level security	Isolate tenants inside Postgres; commit vs rollback	1
Indexes + EXPLAIN ANALYZE	500k rows; take a query from ~19 ms to ~0.1 ms; find indexes Postgres can't use	1
Pagination	OFFSET vs keyset (cursor), measured	1
Connection limits + pooling	Run out of connections, hit the app pool timeout, add PgBouncer, see the tenant-setting leak	2
Locks + deadlocks	Find the session blocking everyone; lock_timeout; force a deadlock	4
MVCC, dead rows, VACUUM	Watch the table double in size from one UPDATE; reclaim it	4
Slowest-query report	`pg_stat_statements`: which query costs the most in total	4
Sharding	Shard key, routing, scatter queries, global ids, cross-shard writes	5
Idempotent writes	A unique key + `ON CONFLICT DO NOTHING` so retried batches never duplicate rows	6, 7
Event streaming	Partitions, consumer groups, offsets and lag, replay, dead-letter topic, the outbox pattern	7

Set up your computer

You need Docker, Python 3.12 or newer, kind and kubectl. k9s is optional but recommended. Pick your system:

macOS

Install Docker Desktop. In Settings → Resources, give it at least 4 GB of memory.
Install Homebrew, then the tools:

Python 3 from python.org or Homebrew both work.

Windows (WSL 2)

Open PowerShell as administrator, run the command below, then restart (Microsoft guide). It installs Ubuntu.

Install Docker Desktop with the WSL 2 backend. In Settings → Resources → WSL Integration, turn it on for Ubuntu.
Open the Ubuntu app and run the Linux commands in the next card there. From then on, run every command on this page in that Ubuntu terminal.

Keep your project in the Linux home folder (~/tenant-todo), not under /mnt/c. Docker recommends this for speed.

Memory: with WSL 2, Docker's memory limit comes from %UserProfile%\.wslconfig (default: half your RAM), not from Docker Desktop's settings. To set 6 GB, put the lines below in that file, then run wsl --shutdown and reopen Ubuntu (Microsoft guide).

Linux (and Ubuntu on WSL)

On Linux, install Docker Engine and run sudo usermod -aG docker $USER, then log out and in again (Docker guide). On WSL, Docker Desktop provides Docker, so skip this step.
Python's venv module is a separate package on Ubuntu. Without it, python3 -m venv fails with "ensurepip is not available".

On an ARM machine, use the arm64 files instead; see the kind, kubectl and k9s guides.

Using PowerShell instead of WSL? Equivalent commands

These were checked with the PowerShell 7.6 parser, not run on a Windows machine. Use PowerShell 7: Windows PowerShell 5.1 can strip the quotes inside JSON arguments. The Linux-only parts of Days 4b–6 (the Docker socket for Alloy, bash loops and heredocs) are easiest in WSL. Install kind with winget install Kubernetes.kind and k9s with scoop install k9s or choco install k9s.

On this page (bash)	PowerShell 7	Note
source .venv/bin/activate	`.venv\Scripts\Activate.ps1`	If scripts are blocked: `Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser` (Python docs)
docker exec -i pg psql -U postgres -d todo < sql/01-schema.sql	`Get-Content sql/01-schema.sql \| docker exec -i pg psql -U postgres -d todo`	PowerShell has no `<` input redirection
a line ending in \	end the line with a backtick `	Line continuation
curl -s ...	`curl.exe -s ...`	Plain `curl` can be an alias for Invoke-WebRequest
lsof -i :5432	`Get-NetTCPConnection -LocalPort 5432 -State Listen`	Who is using a port
for i in $(seq 1 105); do ...; done	`1..105 \| ForEach-Object { ... }`	Loops
-o /dev/null	`-o NUL`	Discard output (curl.exe)

What you are building

Pick a day to see what it adds

Press “Send a request” to follow one GET /todos call.

Day 1 · FastAPI + Postgres

Two companies, one database, zero leaks

By tonight: an API where tenant 1 and tenant 2 share one table and the database itself stops them seeing each other's rows.

Idea 15

Setup 20

Schema 25

RLS 20

API 50

Indexes 35

Break it 25

What "multi-tenant" means

One app serves many customer companies (tenants). Their data must never mix.

15 min

Approach	How	Used by
Shared table + tenant_id	Every row has a `tenant_id`. Cheapest. One missed filter leaks data.	Most SaaS, Supabase (with RLS). We use this.
Schema per tenant	Same tables copied into one schema per tenant.	Mid-size B2B apps
Database per tenant	Strong isolation, painful at 10,000 tenants.	Regulated customers
Shard by tenant	Shared tables, but each tenant's rows live on one shard.	Vitess / PlanetScale (Day 5)

Row-level security (RLS) is a Postgres rule that filters rows for you. It is the main thing that keeps Supabase customers apart, so it is worth knowing well.

Project folder, Python env, Postgres container

20 min

What to look for

pg   postgres:16   Up 5 seconds   0.0.0.0:5432->5432/tcp
... LOG:  database system is ready to accept connections

Port already in use? If docker run says Bind for 0.0.0.0:5432 failed: port is already allocated, another Postgres is running on your computer (it happened on our test Mac). Find it with lsof -i :5432 (Windows PowerShell: Get-NetTCPConnection -LocalPort 5432 -State Listen), or use -p 5433:5432 and port 5433 in every URL.

Tables: tenants and todos

Save as a file so Day 2 and Day 3 can reuse it.

25 min

Windows PowerShell: < doesn't work there. Use Get-Content sql/01-schema.sql | docker exec -i pg psql -U postgres -d todo. The same applies to every < file on this page.

Inside psql, try \dt (list tables), \d todos (describe), SELECT * FROM tenants;, then \q to quit.

Row-level security: the database guards each tenant

20 min

USING filters what a query can see. WITH CHECK blocks writing a row for another tenant.
app.tenant_id is a setting the API puts on the connection for each request.
The true in current_setting(..., true) means "return nothing if unset" instead of raising an error. Nothing set means no rows.

The API

Type it yourself. Read the two comments, they hold the key ideas.

50 min

app/main.py (Day 1 version)

The queries have no WHERE tenant_id = .... Row-level security adds that filter for you. Many teams keep the WHERE as well, as a second safety net.

Real output · test run 12 Oct 2026 (status codes and notes added)

{"id":1,"title":"buy milk","done":false}            201
{"id":2,"title":"globex secret plan","done":false}  201
[{"id":1,"title":"buy milk","done":false}]          200   ← tenant 1 sees only its row
[{"id":2,"title":"globex secret plan","done":false}] 200  ← tenant 2 sees only its row
{"detail":"todo not found"}                         404   ← tenant 2 can't touch tenant 1's todo
{"detail":[{"type":"missing","loc":["header","x-tenant-id"],"msg":"Field required"...}]}  422

Also open localhost:8000/docs. FastAPI builds a clickable test page for free.

Indexes: make "my latest todos" fast at 500k rows

An index is a sorted copy of some columns, so Postgres can jump to rows instead of reading the whole table.

35 min

Round 1. The question every todo app asks: "tenant 7's newest 20". Run it with the single-column index from 01-schema.sql:

Real output (trimmed)

Limit  (actual time=18.817..18.820 rows=20)
  ->  Sort  Sort Key: created_at DESC          ← sorts all 10,000 of tenant 7's rows
        Sort Method: top-N heapsort  Memory: 26kB
        ->  Bitmap Heap Scan on todos (actual rows=10000)
              ->  Bitmap Index Scan on todos_tenant_idx
Execution Time: 18.941 ms

Round 2. Drop that index (DROP INDEX todos_tenant_idx;) and run it again: Parallel Seq Scan, 17.7 ms in our test. Almost the same. The index found the rows, but the sort still did most of the work.

Round 3. Build an index that matches both the filter and the order:

Real output

Limit  (actual time=0.058..0.073 rows=20)
  ->  Index Scan using todos_tenant_created_idx on todos
        Index Cond: (tenant_id = 7)          ← no Sort step: rows come out already ordered
Execution Time: 0.109 ms                     ← ~170× faster on our machine

Now replace the index line in sql/01-schema.sql with this one, so Days 2–4 get it automatically.

How to read a plan, in 4 checks

Scan type: Seq Scan = read every row. Index Scan / Bitmap = jumped to rows. Index Only Scan = didn't even touch the table.
rows= estimate vs actual: far apart means stale statistics. Run ANALYZE.
Rows Removed by Filter: work thrown away. Big numbers mean a missing index.
Sort / Hash steps with a large "Memory" or "Disk": an index in the right order can remove them.

Three ways an index gets ignored (all tested)

Two of these need an index on title, so create it first: CREATE INDEX todos_title_idx ON todos (title); then ANALYZE todos;

Query	Plan	Fix
WHERE created_at > now() - interval '1 hour'	Seq Scan, 41 ms. The index starts with `tenant_id`, like a phone book sorted by surname that you search by first name.	Put the filtered column first, or add a separate index
WHERE title LIKE '%4242'	Seq Scan, 27 ms (with a title index!). A B-tree can't search from the middle of a word.	For "contains": `CREATE EXTENSION pg_trgm;` + `CREATE INDEX ON todos USING gin (title gin_trgm_ops);` → 0.8 ms (tested). Even `LIKE 'task%'` skips a plain index under the default `en_US.utf8` collation; it needs `(title text_pattern_ops)` → 0.4 ms (tested).
WHERE lower(title) = '...'	Seq Scan, 47 ms. The index holds `title`, not `lower(title)`.	`CREATE INDEX ON todos (lower(title));` → 0.1 ms (tested)

Pagination: OFFSET vs keyset

Real output

OFFSET 9000:  Sort (quicksort, 1088kB) over 10000 rows    Execution Time: 39.456 ms
keyset:       Index Scan using todos_tenant_created_idx    Execution Time: 0.071 ms

Your cursor value will differ: use a created_at from your own data. OFFSET gets slower with every page; keyset stays flat.

Indexes cost something: every INSERT/UPDATE must update each one, and they use disk. Run \di+ todos*. In our test the title index alone was 19 MB on a 33 MB table. Only add indexes that real queries need.

Break it on purpose

Guess what will happen before each one, then check.

25 min

Log in as the superuser

Restart with DATABASE_URL=postgresql+psycopg://postgres:dev@localhost:5432/todo uvicorn app.main:app. Tenant 2 now sees tenant 1's todos, because superusers skip RLS. Supabase's service_role key works the same way.

Forget to set the tenant

Comment out the set_config line. Lists come back empty, and POST fails with new row violates row-level security policy. Read the full error in terminal 1.

Kill the database

docker stop pg, call /health (500, "connection refused"), then docker start pg. It recovers by itself because pool_pre_ping tests connections before using them.

An index Postgres skips on purpose

CREATE INDEX ON todos (done); then EXPLAIN SELECT id FROM todos WHERE done = false;. Two thirds of rows match, so reading the whole table is cheaper. Indexes only pay off for selective filters. Drop it afterwards.

Issues you will hit on Day 1

Symptom	Cause	Fix
password authentication failed for user "postgres"	`POSTGRES_PASSWORD` is only used the first time the volume is created. You changed it later.	`docker rm -f pg && docker volume rm pgdata`, then run it again
port is already allocated / bind: address already in use	Something else is on 5432 (a Postgres installed on your computer, an old container)	`lsof -i :5432` (PowerShell: `Get-NetTCPConnection -LocalPort 5432 -State Listen`); stop it or use `-p 5433:5432` and change the URL
connection refused	Container not running, or still starting	`docker ps -a`, `docker logs pg`
ModuleNotFoundError: psycopg2	URL starts with `postgresql://`. SQLAlchemy 2.0 then looks for the old psycopg2 driver (2.1 and later default to psycopg 3, but Day 4b pins 2.0)	Use `postgresql+psycopg://`
The virtual environment was not created successfully because ensurepip is not available	Ubuntu/WSL ships venv separately	`sudo apt install -y python3-venv`
ModuleNotFoundError: fastapi	venv not active in this terminal	`source .venv/bin/activate` (PowerShell: `.venv\Scripts\Activate.ps1`)
422 Unprocessable Content	Missing header or bad JSON body	Read the `detail` field. It names the missing field.
permission denied for sequence todos_id_seq	The app role can insert rows but can't take the next id	`GRANT USAGE ON SEQUENCE todos_id_seq TO app;`
LEAKRLS is on but everyone sees everything	Connected as the superuser or the table owner, and both skip RLS	Connect as `app`, or run `ALTER TABLE todos FORCE ROW LEVEL SECURITY`
Lists are empty even though rows exist	The tenant setting was set on a different connection or transaction (connection pool)	Set it inside the same transaction as the query, as `tenant_tx` does
INSERT "worked" but the row is gone	Used `engine.connect()` without `commit()`, so it rolled back	Use `engine.begin()` for writes
Index exists but the query is still slow	Index doesn't match the ORDER BY, or the query wraps the column in a function or a leading `%`	EXPLAIN ANALYZE; build the index to match the query's filter + order
Plan estimates rows=1 but actual rows=200000	Statistics are stale after a big load	`ANALYZE todos;`
Page 500 of a list takes seconds	OFFSET reads and discards every earlier row	Keyset pagination

Exit check: can you answer these?

1. Why can't the app connect as postgres? Superusers bypass RLS.

2. Why set_config(..., true) and not SET app.tenant_id? A plain SET stays on the pooled connection and leaks into the next request for a different tenant.

3. What does WITH CHECK stop that USING doesn't? Writing a row that belongs to another tenant.

4. Why (tenant_id, created_at) and not (created_at, tenant_id)? The query filters on tenant_id with equality, then reads in created_at order. Equality columns go first.

Day 2 · Redis + Docker Compose

Make it fast, then make it one command

By tonight: per-tenant caching and rate limits in Redis, the whole stack starting with docker compose up, and PgBouncer pooling database connections.

Idea 15

redis-cli 20

Cache + limit 40

Compose 50

Pooling 35

Break it 25

Cache-aside, in one picture

15 min

Request comes in → look in Redis under tenant:1:todos.
Hit → return it. The database is never touched.
Miss → query Postgres → store the result in Redis for 60 s → return.
Any write for tenant 1 → delete tenant:1:todos so the next read is fresh.

Every key starts with the tenant id. A key without it is a data leak, because tenant 2 would read tenant 1's cached list.

Play with Redis by hand

20 min

Open a second terminal and run docker exec -it redis redis-cli MONITOR. It prints every command live. Keep it open while you test the API.

Add the cache and a per-tenant rate limit

40 min

app/main.py: the parts that change (full file is tested)

In create_todo, mark_done and delete_todo, add cache.delete(cache_key(t)) after the with tenant_tx(...) block and before the return.

Real output · test run 12 Oct 2026

MONITOR, 1st call:  "INCRBY" "tenant:1:rl:29863330" "1"   ← rate-limit counter (redis-py sends INCR as INCRBY)
                    "GET" "tenant:1:rl:29863330"
                    "GET" "tenant:1:todos"                  ← miss
                    "SETEX" "tenant:1:todos" "60" "[{...}]"  ← store for 60 s
MONITOR, 2nd call:  INCRBY, GET (counter), GET "tenant:1:todos" ← hit: no SETEX, no Postgres
loop:               200 200 200 ... (96 × 200) then 429 429 429 ... (9 × 429)
keys '*':           tenant:1:todos   tenant:1:rl:29863330   hits

The counter key ends in "minutes since 1970". The loop hit 429 after 96 because the two earlier calls in the same minute counted too.

Package it: Dockerfile + Compose

50 min

Inside Compose, services find each other by name: the API reaches db and cache, not localhost.
Files in ./sql run once, in name order, only when the volume is empty. That is why they are numbered 01, 02.
max_connections=20 is set low on purpose for the pooling lab (the default is 100). pg_stat_statements is loaded now for Day 4.
Your computer's Python may be a different version from the image's 3.12. That's fine here. Version gaps like this explain many "works on my machine" bugs.

Connection pooling: run out of connections, then fix it

Every Postgres connection is a whole server process using several MB. You can't have thousands.

35 min

App pool (SQLAlchemy): each API process keeps up to pool_size (5) + max_overflow (default 10) connections open and reuses them.
The math that causes outages: 3 replicas × (5 + 10) = 45 possible connections. With max_connections=20, scaling up the API breaks the database.
PgBouncer sits between them: hundreds of app connections share a few real ones. In transaction mode, you get a real connection only for the length of one transaction.

A. Hit the server limit.

Real output (max_connections=20)

connection #18 failed: ... FATAL:  remaining connection slots are reserved for roles with the SUPERUSER attribute

Why #18? 20 slots minus 3 kept for superusers (so an admin can still log in and fix things) = 17. If the API is also connected, yours fails sooner: with the API running, ours failed at #17. Look at who's connected: SELECT usename, state, count(*) FROM pg_stat_activity GROUP BY 1, 2;

B. Hit the app's pool limit. In Python: create_engine(URL, pool_size=2, max_overflow=0, pool_timeout=3), check out 2 connections and keep them, then ask for a third.

Real output

TimeoutError: QueuePool limit of size 2 overflow 0 reached, connection timed out, timeout 3.00

The database was fine; the app was waiting for its own pool. A common cause in real life is a slow query or a forgotten open transaction holding connections.

C. Add PgBouncer. Add this service and point the API at it:

In the api service: set DATABASE_URL: postgresql+psycopg://app:app@pgbouncer:5432/todo and add pgbouncer: {condition: service_started} under depends_on. Then docker compose up -d --build.

Windows: the next two blocks use bash loops and a bash function. Run them in your WSL Ubuntu terminal.

Real output

1          ← 50 clients, 1 real Postgres connection

D. The leak that explains tenant_tx. Two separate psql sessions through PgBouncer:

Real output (psql's SET / RESET / BEGIN / COMMIT echo lines removed)

client B sees tenant = 1           ← plain SET leaked to a different client
client B sees tenant = (nothing)   ← set_config(..., true) ended with the transaction

With a transaction pooler, anything a session sets (SET, temp tables, advisory locks) can land on another client's request. With RLS, that's a cross-tenant data leak. Supabase uses Supavisor, a pooler with the same transaction mode, which is why this matters there too. Set DEFAULT_POOL_SIZE back to "5".

Break it on purpose

25 min

localhost inside a container

Change the API's URL host to localhost, then docker compose up -d. /health fails: inside a container, localhost means that container itself.

Stale cache

Remove cache.delete from mark_done. Mark a todo done and list it. It still shows false for up to 60 s.

Cross-tenant cache leak

Change cache_key to return "todos". List tenant 1, then tenant 2. Tenant 2 gets tenant 1's list. Revert right away.

Where did my data go?

docker compose down keeps data. docker compose down -v deletes the volume. Try both and see which keeps your todos.

Issues you will hit on Day 2

Symptom	Cause	Fix
Connection refused to localhost:5432 (from the api container)	localhost means the container itself	Use the service name: `db`, `cache`
My SQL files didn't run / role "app" does not exist	The volume already had data, so init scripts were skipped	`docker compose down -v`, then up
Code change isn't showing up	The old image is still running	`docker compose up --build -d`
Every build reinstalls all packages (slow)	App code is copied before `pip install`, which breaks the layer cache	Copy `requirements.txt` and install first, then copy the code (as above)
Old data after an update	Cache not deleted on write	Delete the key in every write path
LEAKTenant sees another tenant's list	Cache key without tenant id	Always `tenant:{id}:...`
OOM command not allowed when used memory > 'maxmemory'	Memory full and the eviction policy is `noeviction`	Use `allkeys-lru` for a cache
Redis freezes for seconds	`KEYS *` on a big dataset blocks Redis	Use `SCAN`
Load test gets 429s	Your own rate limiter is working	Expected. Raise the limit or spread the load across tenants.
remaining connection slots are reserved… / sorry, too many clients already	Postgres hit `max_connections`	`pg_stat_activity` to see who holds them; add a pooler; lower per-app pool sizes
QueuePool limit of size 5 overflow 10 reached, connection timed out	The app's own pool is empty: slow queries or connections never returned	Find what holds connections; always use `with engine...` blocks
prepared statement "_pg3_0" already exists	PgBouncer in transaction mode without prepared-statement support (added in 1.21, on by default only since 1.24 via `max_prepared_statements`) + psycopg 3's automatic prepared statements	Use PgBouncer 1.24+ (1.25.2 worked in testing), set `max_prepared_statements`, or pass `prepare_threshold=None`
LEAKA request sees another tenant's setting	Session-level `SET` through a transaction pooler	`set_config(..., true)` inside the transaction
Many connections "idle in transaction"	Code opened a transaction and never finished it	Fix the code; set `idle_in_transaction_session_timeout`

Exit check

1. Name two ways caching can leak or serve wrong data. Missing tenant prefix; missing invalidation.

2. Why does depends_on need condition: service_healthy? Without it, Compose only waits for the container to start, not for Postgres to be ready.

3. Image vs container vs volume, one line each.

4. 4 API replicas, pool_size 5, max_overflow 10, max_connections 50. Safe? No: up to 60 connections. Lower the pools or add PgBouncer.

Day 3 · Kubernetes on kind

Same app, now self-healing

By tonight: API (2 copies), Redis and Postgres running in a local cluster, with probes, a rolling update and a rollback.

Idea 15

Cluster 20

DB + Redis 35

API 40

Update 25

Break it 45

Compose words → Kubernetes words

15 min

In compose.yaml	In Kubernetes	What it adds
a service (api)	Deployment + Pods	Keeps N copies running, replaces dead ones
service name as hostname	Service	Stable name + load balancing across pods
a database service + volume	StatefulSet + PersistentVolumeClaim	Stable identity, disk that outlives the pod
environment:	ConfigMap / Secret	Config kept outside the image
healthcheck / depends_on	readiness + liveness probes	No traffic until ready; restart if stuck

Cluster up, image in

20 min

k9s comes from Set up your computer. Right after kind create the node shows NotReady for a few seconds; the wait line handles that. The cluster can't see your computer's Docker images. kind load copies the image into the cluster's node. Open k9s in a spare terminal and leave it running all day.

Postgres (StatefulSet) and Redis (Deployment)

35 min

k8s/db.yaml

k8s/redis.yaml

PgBouncer is left out on Kubernetes to save memory; the API connects straight to Postgres here.

What to look for

db-0                    0/1  ContainerCreating → Running 0/1 → Running 1/1
cache-xxxxxxxxxx-xxxxx  1/1  Running

The API: 2 replicas, probes, limits

40 min

k8s/api.yaml

Older guides use kubectl get endpoints. On Kubernetes 1.33+ it still works but prints Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice.

Now run the Day 1 curl commands again in another terminal. Same results, but now they go through a Service to one of 2 pods.

Why two probes: if liveness checked the database, a 30-second DB outage would make Kubernetes restart every API pod, making the outage worse. Readiness only removes the pod from traffic.

Scale, update, roll back, self-heal

25 min

Break it on purpose: the classic pod failures

45 min

ImagePullBackOff

kubectl set image deploy/api api=tenant-todo:9.9. Read the Events with kubectl describe pod.

Running but not Ready

Set a wrong password in the Secret's DATABASE_URL, apply, then kubectl rollout restart deploy/api. The new pod runs but stays 0/1 (Events: Readiness probe failed: HTTP probe failed with statuscode: 500). The rollout stalls and the old pods keep serving: the rolling update protected you. Put the right password back.

Service with no endpoints

Change the Service selector to app: apii. kubectl get endpointslices -l kubernetes.io/service-name=api shows <unset>, and port-forward waits 60 s, then fails with error: timed out waiting for the condition.

OOMKilled

kubectl set resources deploy/api -c api --requests=memory=16Mi --limits=memory=20Mi. Lower the request too, or Kubernetes rejects the change ("must be less than or equal to memory limit"). The new pod goes into CrashLoopBackOff; kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, Exit Code: 137.

Pending forever

kubectl set resources deploy/api -c api --requests=memory=8Gi --limits=memory=8Gi. Events say 0/1 nodes are available: 1 Insufficient memory. With 4 GB for Docker, 8 GB can never fit.

CrashLoopBackOff

Change the CMD to app.mian:app (typo). Read the crashed container's output with kubectl logs POD --previous, using the crashing pod's name: logs deploy/api may pick a healthy pod. You'll see ERROR: Error loading ASGI app. Could not import module "app.mian".

Debug order, every time: get pods → describe pod (read Events at the bottom) → logs / logs --previous → get endpoints → exec in and test the connection.

Issues you will hit on Day 3

Symptom	Cause	Fix
ErrImagePull / ImagePullBackOff	Forgot `kind load`, or used `:latest` (which makes k8s try to download it)	Use a real tag + `kind load docker-image`
CrashLoopBackOff	The process exits on start	`kubectl logs POD --previous`
Pending	Not enough CPU or memory, or the PVC isn't bound	`kubectl describe pod`, read Events
Running 0/1	Readiness probe failing	Events show the probe error; `curl` /health from inside the pod
port-forward hangs, then "timed out waiting for the condition"	Selector doesn't match pod labels, or no pod is Ready	`kubectl get endpointslices`; compare labels
OOMKilled, exit code 137	Used more memory than its limit	Raise the limit, or find the memory hog
Changed the Secret but nothing changed	Env vars are read only when the pod starts	`kubectl rollout restart deploy/api`
role "app" does not exist	The PVC already had data, so init SQL didn't run (the Day 2 trap again)	`kubectl delete sts db && kubectl delete pvc data-db-0`, re-apply
Whole computer slows down	kind + compose + other containers are over Docker's memory limit	`docker stats`; stop what you aren't using

Day 4a · Prometheus + Grafana

See the incident before the customer reports it

By tonight: a dashboard with request rate, errors, p99 latency per endpoint, todos per tenant, plus Postgres and Redis health. You'll also diagnose the database incidents support engineers see most: blocking locks, deadlocks, table bloat and the costliest query. Runs on Compose to fit in memory.

Idea 15

Metrics 25

Stack 30

Dashboard 40

DB incidents 45

App incidents 30

How it fits together

15 min

Metrics are numbers over time (requests/sec). Logs are events (one error line). Traces follow one request across services. Today is metrics.
Prometheus pulls (scrapes) /metrics from each target every 5 s. Grafana only draws charts from what Prometheus has.
RED for any API: Rate, Errors, Duration. p99 = 99% of requests were faster than this. Averages hide the slow 1%, and those are the customers who complain.

Make the API expose /metrics

25 min

In create_todo, before the return: TODOS_CREATED.labels(tenant=str(t)).inc()

Real output · curl localhost:8000/metrics (filtered)

todos_created_total{tenant="1"} 1.0
todos_created_total{tenant="2"} 1.0
http_requests_total{handler="/metrics",method="GET",status="2xx"} 1.0
http_requests_total{handler="/health",method="GET",status="2xx"} 1.0
http_requests_total{handler="/todos",method="POST",status="2xx"} 2.0
http_requests_total{handler="/todos",method="GET",status="2xx"} 2.0
http_requests_total{handler="/todos/{todo_id}/done",method="PATCH",status="4xx"} 1.0

Status codes are grouped (2xx, 4xx, 5xx), not 500. That matters when you write queries.

Add Prometheus, Grafana and two exporters

30 min

Also add grafana: under the bottom volumes: list, next to pgdata:, so dashboards survive restarts.

Then open localhost:9090/targets in your browser.

What to look for

api        UP
postgres   UP
redis      UP      ← any DOWN shows its error message right there

Build the dashboard

Grafana at localhost:3000, login admin / admin (it asks you to choose a new password).

40 min

Connections → Data sources → Prometheus → URL http://prometheus:9090 → Save & test.
Dashboards → New → add one panel per query below. Test each query in Prometheus first (localhost:9090) before pasting it into Grafana.

Windows: run this loop in WSL Ubuntu, or in PowerShell 7: 1..600 | ForEach-Object { curl.exe -s -o NUL -H "X-Tenant-ID: $(Get-Random -Minimum 1 -Maximum 3)" localhost:8000/todos; Start-Sleep -Milliseconds 100 }

Database incidents: locks, deadlocks, bloat, top queries

Use the 500k-row table from Day 1 (re-run the seed if the volume is new). Open two psql terminals: docker compose exec db psql -U postgres -d todo.

45 min

1. "Everything is hanging": a blocking lock.

Real output (trimmed)

 pid |        state        | wait_event_type |  wait_event   | blocked_by | query
 516 | idle in transaction | Client          | ClientRead    | {}         | UPDATE todos SET done = true WHERE id = 1   ← the blocker
 532 | active              | Lock            | transactionid | {516}      | UPDATE todos SET title = 'renamed' WHERE id =

Fix options: COMMIT in terminal 1, or end it with SELECT pg_terminate_backend(516); (use your pid). Prevention: SET lock_timeout = '2s'; makes waiters give up instead of hanging:

Real output

ERROR:  canceling statement due to lock timeout

2. Deadlock. Two transactions each wait for the other's row.

Real output · same steps run from a Python script

A -> deadlock detected
B committed          ← Postgres cancels one transaction (after deadlock_timeout, 1 s by default) so the other can finish

Postgres log:  ERROR:  deadlock detected
               DETAIL:  Process 643 waits for ShareLock on transaction 767; blocked by process 644.
                        Process 644 waits for ShareLock on transaction 766; blocked by process 643.

Fix: always lock rows in the same order (for example, sort ids first), and retry when you get a deadlock error.

3. "The table doubled in size but row count didn't change": MVCC + VACUUM. An UPDATE in Postgres writes a new copy of the row and leaves the old one as a dead row. VACUUM later frees the space for reuse.

Real output

before UPDATE      33 MB   live 500004   dead 6
after UPDATE       65 MB   live 500004   dead 500004   ← every row now has a dead copy
after VACUUM       65 MB   live 500004   dead 0        ← space reusable, file not smaller
after VACUUM FULL  33 MB   live 500004   dead 0        ← file rewritten; blocks all access meanwhile

Watch the n_dead_tup panel in Grafana while you do this. A long-open transaction (step 1) also stops VACUUM from cleaning up, so the two problems often show up together.

4. "What's slow overall?": pg_stat_statements.

Real output

 calls | mean_ms | query
     2 |    20.7 | SELECT count(*) FROM todos WHERE title LIKE $1
     1 |     0.2 | SELECT pg_stat_statements_reset()
     1 |     0.2 | SELECT id FROM todos WHERE tenant_id = $1 ORDER BY crea

Sort by total time, not mean. A 5 ms query called a million times hurts more than one 2-second report. Values show as $1, so similar queries are grouped together.

App incidents: find them on the dashboard

30 min

Database down

While traffic runs: docker compose stop db. Watch the 5xx panel and the Postgres target go DOWN. Then docker compose start db. How long did recovery take?

Slow query hidden by the cache

Drop todos_tenant_created_idx, then docker compose exec cache redis-cli FLUSHALL. p99 jumps only on cache misses, so the cache was hiding the problem. Fix: recreate the index.

Rate-limit 4xx spike

Make the loop hit only tenant 1 with no sleep. The 4xx panel rises. Is that an incident or a working limit? Write one sentence to the "customer".

Too many label values

Imagine labelling a metric with todo_id. Count series with count({__name__=~".+"}). Each unique label value is a new series, and too many crash Prometheus.

Issues you will hit on Day 4

Symptom	Cause	Fix
Target DOWN: connection refused / no such host	Wrong name or port in prometheus.yml	Use Compose service names and the container's own port
Grafana: "No data"	Time range too old, or the data source URL is `localhost`	Last 15 min; data source URL `http://prometheus:9090`
rate() returns nothing	Range window shorter than 2 scrapes	Use `[1m]` or more with a 5 s scrape
p99 line looks wrong or flat	Forgot `by (le)` inside histogram_quantile	Always keep `le`
Error query is always 0 or "No data"	Matching `status="500"`, but labels are grouped; or no error has happened yet, so the series doesn't exist	`status="5xx"`; add `or vector(0)`
Counter dropped to 0	App restarted; counters reset	Normal. `rate()`/`increase()` handle it. Never chart a raw counter.
Dashboard gone after restart	No volume for Grafana	The `grafana` volume above; also export the dashboard JSON
Prometheus memory keeps growing	High-cardinality labels (user id, URL with ids)	Only use labels with few possible values
Queries hang, CPU is idle	Waiting on a lock held by another session	`pg_blocking_pids()` in pg_stat_activity; commit or terminate the blocker; set `lock_timeout`
deadlock detected	Two transactions lock the same rows in opposite order	Lock in a consistent order; retry on this error
Table keeps growing, row count doesn't	Dead rows from UPDATE/DELETE; autovacuum behind or blocked by a long transaction	Check `n_dead_tup`, `last_autovacuum`; end long transactions
Everything froze during "maintenance"	`VACUUM FULL` or `CREATE INDEX` locked the table	Plain VACUUM; `CREATE INDEX CONCURRENTLY`
relation "pg_stat_statements" does not exist	Library preloaded but extension not created	`CREATE EXTENSION pg_stat_statements;`

Day 4b · Logs, stack traces, traces

From "p99 went up" to the exact line of code

By the end: JSON logs with tenant and trace id on every line, full Python tracebacks searchable in Loki, and one click from a log line to the request's trace in Tempo. Builds on the Day 4a Compose stack.

Idea 10

JSON logs 30

Loki 25

Traces 25

Link 10

Break it 20

Three signals, one id

10 min

Signal	Answers	Tool today
Metrics	Is something wrong? How much, since when?	Prometheus (4a)
Logs	What exactly happened, to which tenant? Includes the stack trace.	Loki, fed by Alloy
Traces	Where did the time go inside one request? Cache, SQL, Redis.	Tempo, fed by OpenTelemetry

A trace is one request. It is made of spans, one per step (the HTTP call, each SQL query, each Redis command), each with a start and duration.
Every log line written during a request carries that request's trace_id. That id is what lets you jump between logs and traces.
Labels vs fields. Loki indexes only labels, so give labels few possible values (service, level). tenant_id, trace_id and status stay as JSON fields you filter at query time. Making tenant_id a label is the most common Loki mistake.

JSON logs with the stack trace in one field

30 min

app/observability.py (new file)

app/main.py: imports, the request-log middleware, and a route with a bug

In the Dockerfile, add "--no-access-log" to the uvicorn CMD; the middleware now writes a better access log. In compose.yaml, add OTEL_EXPORTER_OTLP_ENDPOINT: http://tempo:4318 to the api's environment.

Real output (stack trimmed; it was 62 lines)

{"time": "2026-10-12T12:49:39", "level": "info", "logger": "tenant_todo", "msg": "request",
 "trace_id": "371756b7df9a03bc0c4990334b4a4661", "method": "GET", "path": "/todos/stats",
 "tenant_id": "1", "status": 200, "duration_ms": 122.3}
{"time": "2026-10-12T12:49:39", "level": "error", "logger": "tenant_todo", "msg": "unhandled error",
 "trace_id": "3ebe2f518e13bb0de627930815b150b4", "method": "GET", "path": "/todos/stats",
 "tenant_id": "2", "status": 500, "duration_ms": 22.2,
 "stack": "Traceback (most recent call last):\n  File \"/app/app/main.py\", line 41, in access_log\n
   ... frames inside starlette, fastapi, anyio ...
   File \"/app/app/main.py\", line 146, in todo_stats\n
     return {"total": total, "done": done, "percent_done": round(100 * done / total, 1)}\n
 ZeroDivisionError: division by zero"}

Read a Python stack trace from the bottom. The last line is the error. Just above it is the last frame in your code (app/main.py, line 146 in our file; your line number depends on your file). The library frames in between are almost never the cause.

Ship logs to Loki with Alloy

Alloy reads every container's output through the Docker socket. The app needs no log library changes.

25 min

config.alloy

Add loki-data: (and tempo-data: for the next step) to the bottom volumes: list. Run docker compose up -d, make a few requests, and wait about 20 seconds. Loki waits 15 s after starting before it accepts logs.

In Grafana: Explore → pick Loki (added in the Link step below, or add it by hand now with URL http://loki:3100). Try these LogQL queries:

Real output (from Loki's API)

errors for tenant 2:   2 lines, path /todos/stats, status 500, stack ends: ZeroDivisionError: division by zero
by status (last 10m):  200 → 16   404 → 1   500 → 2
by tenant:             {tenant_id="2"} → 2

If you see a {} group with no tenant, that's an error from outside a request, like a startup warning (see Issues).

Traces in Tempo

OpenTelemetry records each request, SQL query and Redis call automatically. You already set it up in observability.py.

25 min

Grafana → Explore → Tempo → Search, or use a TraceQL query like { duration > 15ms } or { status = error }. Open the newest GET /todos.

Real output · one GET /todos with a cache miss, spans in time order

 129.05 ms  GET /todos               ← the whole request (tenant 1 had 10,002 rows here)
   2.60 ms  connect                  ┐
   1.47 ms  SELECT todo              ┘ tenant_id(): does tenant 1 exist?
   1.36 ms  INCRBY                   ┐
   0.19 ms  EXPIRE                   │ rate limiter
   0.18 ms  GET                      ┘
   0.13 ms  GET                      ← cache lookup: miss
   0.27 ms  connect
   0.56 ms  SELECT todo              ← set_config (RLS tenant)
  47.69 ms  SELECT todo              ← the actual query: the long bar
   1.56 ms  SETEX                    ← store in cache for 60 s
   0.10 ms  GET /todos http send     ← (3 small spans for sending the response)

The /todos/stats trace for tenant 2 has an error status on its top span, and { status = error } finds those traces (2 in our run).

Link them in Grafana

A provisioning file means you never click through data source settings again.

10 min

In compose, give grafana a second volume: "./grafana/provisioning:/etc/grafana/provisioning:ro", then docker compose up -d grafana. Prometheus is left out here because you added it by hand on Day 4a; a second copy with the same name causes a conflict. The $$ is intentional: Grafana reads $$ as a literal $.

Real check · Grafana's API after provisioning

loki: OK - Data source successfully connected.
loki derived field:  "trace_id": "(\w+)"  → tempo   url ${__value.raw}     ← $$ became $
tempo → logs query:  {service="api"} |= "${__trace.traceId}"
regex tested on a real log line → captures 5e62dd2ad0e47d22976297a8c1a76980

What to look for in the UI

Explore → Loki → expand an error line → a "trace_id" field with a Tempo button → opens the trace
In the trace → a span's "Logs for this span" → back to that request's log lines

This is the support workflow to practise until it's automatic: metric spikes (4a dashboard) → find a slow or failed trace → open its logs → read the stack → name the tenant and the line of code.

Break it on purpose

20 min

Bug hunt from a ticket

"Tenant 2 says their stats page is broken." Find it using only Grafana: the error log, its stack, its trace. Then fix it (if total == 0) and confirm the errors stop.

Slow request

Hold a row lock in psql (Day 4a, incident 1) and PATCH that todo. Search Tempo for { duration > 1s }. The UPDATE span is the long bar; the log line shows the same duration_ms.

Where did my logs go?

docker compose stop alloy, make 5 requests, start it again. Do those 5 lines show up in Loki? Logs depend on their shipper, so the shipper needs watching too.

Would you log this?

Imagine adding the request's Authorization header to fields. Anyone with Grafana access could now read every customer's token. Log ids, never secrets or personal data.

Issues you will hit on Day 4b

Symptom	Cause	Fix
Loki exits: mkdir /loki/chunks: permission denied	Loki runs as a normal user; the new volume belongs to root (hit in testing)	`user: "0"` locally, or a volume with the right owner
SQLAlchemyInstrumentor only instruments "sqlalchemy >= 1.0.0, < 2.1.0"…	Instrumentation is behind the newest SQLAlchemy, so traces silently lack SQL spans (hit in testing)	Pin `sqlalchemy<2.1`; read startup logs after every upgrade
Bind for 0.0.0.0:3000 failed: port is already allocated	Another stack still holds the port. In testing it was a kind cluster mapping 3000 and 9090.	`docker ps`; stop the old stack or change the left-hand port
Ingester not ready: waiting for 15s after being ready	Loki just started	Wait 15–20 s
Loki shows nothing	Wrong time range, wrong label (`service` not `container`), or Alloy not running	Alloy UI at localhost:12345 shows each pipeline step and its errors
One traceback = 30 separate log lines	Traceback printed as plain text, not inside the JSON log	Log it with `log.exception(...)` through the JSON formatter
Logs full of GET /metrics every 5 s	Prometheus scrapes and health probes get logged too (seen in testing)	Skip those paths in the middleware
Loki slow, or "maximum active stream limit exceeded"	High-cardinality labels (tenant_id, trace_id, user id)	Keep them as JSON fields; filter with `\| json`
No Tempo button on log lines	Derived-field regex doesn't match. `json.dumps` writes `"trace_id": "…"` with a space.	Test the regex on a real line
No traces in Tempo	Wrong port (4317 is gRPC, 4318 is HTTP) or the endpoint env var is missing	Check `OTEL_EXPORTER_OTLP_ENDPOINT`; spans are sent in batches, so wait ~5 s
An error with no tenant ({} in "by tenant")	Errors outside a request: startup, background jobs	Expected. Read them; this one was the SQLAlchemy warning above.

Exit check

1. Why is level a label but tenant_id isn't? Few values vs thousands; each label combination is a separate stream Loki must index.

2. What connects a log line to a trace? The trace_id field, plus the derived-field link in Grafana.

3. A customer reports "slow sometimes". Which signal do you open first, and then which? Metrics (when, how bad, which endpoint) → traces (which step is slow) → logs (which tenant, what error).

Day 5 · Vitess MySQL

Shard by tenant

By tonight: todos split across 2 MySQL shards by tenant, and the query plans that prove it. This day is shorter on purpose (about 1 h 35). Use the rest for the stretch task, or to finish anything left from Days 1–4b.

Idea 15

Start 15

VSchema 25

Routing 20

Python 20

Stretch / catch-up 45

Vitess in one picture

15 min

Your app talks MySQL to vtgate, a proxy. It looks like one database.
Behind it, a keyspace (logical database) is split into shards. Each shard is a normal MySQL.
The VSchema says how to route rows. A vindex on tenant_id hashes it to pick a shard. All of a tenant's todos land on one shard.
A query with tenant_id goes to one shard. A query without it goes to every shard (a "scatter"). That costs more and is the most common Vitess performance ticket.
Postgres RLS doesn't exist here. In MySQL, tenant isolation is the WHERE tenant_id filter in every query.
Choosing the shard key is the big decision. tenant_id works here because almost every query is about one tenant. A bad key turns every query into a scatter and is very expensive to change later.
Pooling is built in: each shard's vttablet keeps a connection pool to its MySQL, doing PgBouncer's job.

Start a full Vitess in one container

First download takes a few minutes. Used about 830 MB of memory in our test.

15 min

--platform linux/amd64 is there because this image is only built for Intel/AMD (amd64) chips. On an Apple Silicon Mac or an ARM machine, Docker emulates it: slower, but fine for learning. On a normal Intel/AMD Windows or Linux machine it runs natively. todo gets 2 shards; lookup is unsharded and holds the id sequence.

Windows: Day 5 uses a bash function and heredocs (<<'SQL'). Run it in WSL Ubuntu.

Table, sequence, vindex

25 min

Real output · the error you are meant to see

ERROR 1105 (HY000) at line 1: table todos not found

The table exists in MySQL, but vtgate doesn't know how to route it yet. Tell it:

Real output

+----+-----------+-----------+------+
| id | tenant_id | title     | done |
+----+-----------+-----------+------+
|  1 |         1 | buy milk  |    0 |
|  2 |         1 | call bank |    0 |
|  3 |         2 | ship v2   |    0 |
|  4 |         3 | hire sre  |    0 |
|  5 |         4 | fix ci    |    0 |
+----+-----------+-----------+------+

Prove the routing

20 min

Real output

shard -80                    shard 80-
| tenant_id | title     |    | tenant_id | title  |
|         1 | buy milk  |    |         4 | fix ci |
|         1 | call bank |
|         2 | ship v2   |
|         3 | hire sre  |

Real output (trimmed)

"OperatorType": "Route", "Variant": "EqualUnique", "Vindex": "hash"   ← one shard
"OperatorType": "Route", "Variant": "Scatter"                          ← every shard

Talk to Vitess from Python

20 min

Real output

new id: 6
[(6, 'from python')]

The sequence hands out ids in reserved blocks of 100 (the cache column), held by the tablet serving lookup.todo_seq. When that tablet restarts or fails over, the unused part of its block is skipped, so the next id can jump (for example to 101). Gaps are normal. Never promise customers ids without gaps.

Stretch: point the whole API at Vitess

Change three things and you learn the real MySQL vs Postgres differences:

Remove RETURNING; vtgate rejects it (real error: (1105, "syntax error at position 63 near 'RETURNING'"); the position depends on your statement). Use result.lastrowid, then SELECT.
Remove set_config (Postgres only). Put AND tenant_id = :t back into every query. Without it there's no isolation, and every query becomes a scatter.
Create a tenants table in the lookup keyspace, or skip the tenant check.

Issues you will hit on Day 5

Symptom	Cause	Fix
table todos not found	Sharded table has no VSchema entry	`ALTER VSCHEMA ON ... ADD VINDEX`
VT09015: schema tracking required	`SELECT ` with ORDER BY across shards when vtgate has no schema information (schema tracking off). With this vttestserver version, `SELECT ... ORDER BY id` worked in our test.	List the columns by name
syntax error near 'RETURNING'	Postgres-only syntax	Use `lastrowid`, then SELECT
Duplicate ids across shards	MySQL AUTO_INCREMENT counts per shard	Use a Vitess sequence (as above)
Queries slow as tenants grow	Missing `tenant_id` in WHERE → scatter	`VEXPLAIN PLAN`; add the shard key
One transaction updates 2 tenants, and only half the changes stick	The rows sit on different shards; by default Vitess commits each shard separately, so it isn't atomic if one commit fails	Keep transactions to one tenant; Vitess also has an opt-in two-phase-commit mode
One shard busier than the others	A very large tenant (a "hot shard")	Watch per-shard load; move big tenants with resharding or custom vindexes
Vitess container very slow / exits	Intel-only image emulated, or out of memory	Stop other stacks; check `docker logs vitess`

Day 6 · Temporal

Bulk-assign todos from a CSV, and survive every failure

By the end: a tenant admin uploads a CSV of todos for their team. A Temporal workflow validates it, imports it in batches, assigns each todo, notifies each person, and reports progress. It keeps going through crashes, retries and cancellations without creating a single duplicate.

Idea 15

Schema 15

Workflow 45

Worker + API 30

Run it 15

Break it 30

Why this needs Temporal

15 min

Importing 50 rows could be one database transaction inside the request. Temporal earns its place when the job is long, touches outside services, and must not half-finish:

What goes wrong in real imports	How this design handles it	Temporal idea
Worker crashes at row 1,100 of 2,000	Restarts from the last finished batch	Durable workflow
A batch is retried after it already saved	Unique key per CSV line + `ON CONFLICT DO NOTHING`	Activities must be idempotent
Email service times out	Each notification retries on its own	Retry policy
Wrong file uploaded	Fails at once with a clear message; no pointless retries	Non-retryable error
"How far along is it?"	`GET /imports/{id}` asks the running workflow	Query
"Stop, that was the wrong file!"	`POST /imports/{id}/cancel`	Signal
20 MB CSV	Workflow receives only the import id; activities read the file from Postgres	Keep payloads small (~2 MB limit)
One step per row on 10,000 rows	Batches of 100	Workflow history has size limits

Workflow code (workflows.py) only decides what happens next. Temporal replays it from history after a crash, so it must do the same thing every time. No database, network, random or clock calls in it.
Activities (activities.py) do the real work: read the CSV, write to Postgres, send emails. Temporal retries them.
The worker is a separate process (a new Compose service) that runs both. The API only starts workflows and asks about them.

Users, imports, and the idempotency key

15 min

sql/03-users-imports.sql

Windows PowerShell: Get-Content sql/03-users-imports.sql | docker compose exec -T db psql -U postgres -d todo

The key line is the unique index on (import_id, import_line). Each CSV line can become a todo once and only once, however many times a batch is retried.

Activities and the workflow

Read the comments: each one marks a lesson from the table above.

45 min

app/activities.py: the real work, retried by Temporal

app/workflows.py: the plan, replayed after crashes

Two failures are faked on purpose: 20% of batches "lose the connection" right after committing, and 30% of notifications time out. Every run exercises the retry and duplicate protection. BATCH_PAUSE_SECONDS slows imports so you have time to break them.

Worker, API endpoints, Compose

30 min

app/worker.py

app/main.py: new imports + three endpoints

In list_todos, select assignee_id too: SELECT id, title, done, assignee_id FROM todos ORDER BY id.

The import endpoints are async because the Temporal client is async. Database calls inside them go through run_in_threadpool so they don't block other requests.
Each workflow id is import-{uuid}, so you can always find an import in the Temporal UI from its API id.

Also: add TEMPORAL_ADDRESS: temporal:7233 to the api's environment, and temporal-data: to the bottom volumes: list. Without --db-filename, the dev server keeps everything in memory, and a restart wipes every workflow.

Real output

{"time": "2026-10-12T12:54:56", "level": "info", "logger": "imports", "msg": "worker started on task queue 'imports'"}

Run your first import

15 min

Real output

{"detail":"only a tenant admin can do this"}                       ← member: 403
{"import_id":"47b4094e-9756-4a76-9d58-203f759d5270"}               ← admin: 202 Accepted
{"workflow_status":"RUNNING","state":"starting","total_rows":0,"processed":0,"created":0,...}
{"workflow_status":"COMPLETED","state":"done","total_rows":6,"processed":6,"created":0,"already_existed":3,"rejected":3,"notified":2}
{"detail":"import not found"}                                      ← tenant 2 can't see it

Result  {..."rejected_lines":[{"line":5,"reason":"empty title"},
                              {"line":6,"reason":"unknown user nobody@acme.test"},
                              {"line":7,"reason":"unknown user kim@globex.test"}]}

Why "created":0 and "already_existed":3? In our run the batch hit the simulated "lost connection after commit": the 3 rows were saved, Temporal retried the batch (attempt 2), and ON CONFLICT skipped them. Counts come from the successful attempt only. The database had exactly 3 rows. Most runs show "created":3.

Line 7 is rejected because row-level security only lets Acme's import see Acme's users, so Kim from Globex simply doesn't exist for it. Open localhost:8233 and click the workflow to see each step. The worker's "notified assignee" lines are in Loki too: {service="worker"} | json | msg="notified assignee".

Break it on purpose

Make a 2,000-row file first, with 40 deliberately bad emails.

30 min

1. Kill the worker halfway. Start a big import, then after 5 seconds run docker compose stop worker. Ask for progress, check the database, then docker compose start worker.

Real output

t+5s:     {"workflow_status":"RUNNING","processed":500,"created":392,"already_existed":98,"rejected":10,...}
worker stopped →  {"detail":"no import worker is running, so progress is unavailable"}  [503]
final:    {"workflow_status":"COMPLETED","processed":2000,"created":1470,"already_existed":490,"rejected":40,"notified":2}

SELECT count(*) AS rows, count(DISTINCT import_line) AS distinct_lines FROM todos WHERE import_id = '...';
 rows | distinct_lines
 1960 |           1960        ← 2,000 − 40 bad rows, and zero duplicates

The 490 "already existed" rows are retries and the batch re-run after the crash, all safely skipped. Your numbers will differ because the failures are random. Queries are answered by the worker, so progress is unavailable while it's down even though the workflow is fine.

2. Where are the retries? The history doesn't list failed attempts as separate events. Each ActivityTaskStarted has an attempt number instead. In the UI, check "Pending Activities" while it runs.

Real output (counted from temporal workflow show -o json)

activities: 23 | needed a retry: 6 | max attempt: 4 | history events: 143

Cancel it

Start a big import, then curl -X POST localhost:8000/imports/$ID/cancel -H 'X-Tenant-ID: 1' -H 'X-User-ID: 1'.
Real result: "state":"cancelled","processed":400. The finished batches stay, and the people already assigned are still notified.

Wrong file

Upload a CSV with header task,owner.
Real result: workflow FAILED after 1 attempt with CSV header must be title,assignee_email (got ['task', 'owner']).

Change the code mid-import

Stop the worker during an import, add await workflow.sleep(timedelta(seconds=1)) as the first line of run(), rebuild the worker. After 20–30 s the worker log shows [TMPRL1100] Nondeterminism error: Timer machine does not handle this event: HistoryEvent(id: 5, ActivityTaskScheduled). Revert and rebuild: the import finishes. The safe way to change code is workflow.patched().

Remove the safety net

Delete ON CONFLICT ... DO NOTHING. Retried batches now fail with duplicate key value violates unique constraint "todos_import_line_uq". Also drop that index, and retries quietly create duplicates: rows becomes bigger than distinct_lines. Put both back.

Temporal is down

docker compose stop temporal, then POST an import. You get a 500, and the stack in Loki ends with RuntimeError: Failed client connect: Server connection error: ... (real; ours continued with a "dns error", because the stopped container's name no longer resolves). Is the CSV row saved even though no workflow started? How would you fix that?

Stale list

Comment out cache.delete(...) in import_batch. Imported todos don't show in GET /todos for up to 60 s. Background jobs have to invalidate caches too.

Issues you will hit on Day 6

Symptom	Cause	Fix
POST /imports → 500; stack ends "Failed client connect: Server connection error"	Temporal not running, or `TEMPORAL_ADDRESS` is `localhost` inside a container (hit in testing)	`temporal:7233` in Compose; check `docker compose ps`
GET /imports/{id} → 503 "no import worker is running"	Queries are answered by a worker (tested)	Start the worker; the workflow itself is safe
Import stuck at "starting", nothing happens	No worker listening on task queue `imports`, often a spelling mismatch	Temporal UI → the workflow → Workers; match the queue names
unable to create SQLite admin DB: unable to open database file (14)	Dev server can't write to the new volume (hit in testing)	`user: "0"` locally
All workflows vanished after restarting Temporal	Dev server defaults to in-memory: its log says `Temporal Persistence: in-memory`	`--db-filename` on a volume
[TMPRL1100] Nondeterminism error	Workflow code changed while runs were in flight (tested)	Revert, or wrap the change in `workflow.patched()`
Workflow FAILED after one attempt	A non-retryable `ApplicationError`, such as a bad header	By design. Read the failure message.
duplicate key value violates unique constraint "todos_import_line_uq"	A retried batch without `ON CONFLICT`	Make every write in an activity safe to repeat
Imported todos missing from the list for a minute	Cache not deleted by the worker	`cache.delete(cache_key(tenant))` after each batch
"start_to_close_timeout or schedule_to_close_timeout required"	Activity started without a timeout	Always pass one
Workflow history keeps growing	One activity per row; history has size limits	Batch rows; for very long jobs, look up `continue_as_new`
A crash's log lines never reached Loki	The container lived only seconds, shorter than Alloy's discovery interval (hit in testing)	`docker compose logs` for short-lived containers; don't rely on one source

Exit check

1. Why does the workflow get an import id instead of the CSV? Payload size limits, and history would store the whole file.

2. A batch saved its rows, then the worker died before reporting success. What happens? Temporal retries the batch; ON CONFLICT skips the saved lines; "already_existed" goes up.

3. Why can't the workflow call random() or the database directly? It's replayed after crashes and must make the same decisions every time. Side effects belong in activities.

4. Progress returns 503 but the Temporal UI shows "Running". Is the import broken? No. No worker is running: queries need one, the workflow just waits.

Day 7 · Kafka

An activity feed that survives consumers going down

By the end: every todo change is published once to Kafka, two independent consumers react to it (an activity feed and live stats), and you've seen lag, rebalancing, duplicates, a poison message, a replay and a lost event, then fixed each one. Builds on the Day 6 Compose stack.

Idea 15

Broker 15

Producer 30

Consumers 40

Watch 20

Break it 50

Kafka vs Temporal, and the five words you need

15 min

Tool	Job	In this lab
Temporal (Day 6)	Runs one job's steps reliably, in order	"Import this CSV, then notify people"
Kafka (today)	Records a fact once; any number of services read it, each at its own pace	"Todo 1 was completed" → feed + stats

Topic: a named, append-only log of messages (todo-events).
Partition: a topic is split into partitions. Messages with the same key always go to the same partition, so they stay in order. We key by tenant_id, so each tenant's events stay in order.
Offset: a message's position in its partition. A consumer stores ("commits") the offset it has finished, so it can resume after a restart.
Consumer group: consumers sharing a group.id split the partitions between them. Two different groups each get every message. That's how feed and stats both see everything.
Lag: how far a group is behind the newest message. The number to watch.

Start Kafka and create the topics

One broker in KRaft mode (no ZooKeeper). Used about 360 MB in our test.

15 min

Add kafka-data: to the bottom volumes: list. ADVERTISED_LISTENERS is the address Kafka tells clients to use; kafka:9092 works from other containers, which is where all our clients run.

Real output

Created topic todo-events.
Created topic todo-events-dlq.
Topic: todo-events  PartitionCount: 3  ReplicationFactor: 1  Configs: min.insync.replicas=1
	Topic: todo-events	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: todo-events	Partition: 1	Leader: 1	Replicas: 1	Isr: 1
	Topic: todo-events	Partition: 2	Leader: 1	Replicas: 1	Isr: 1

If Kafka has only just started, wait 10–20 seconds before creating topics.

The producer: publish an event after every change

30 min

app/events.py (new file)

app/main.py: four small changes

In create_todo, before the return: publish(t, "todo.created", row.id, row.title)
In mark_done, after the 404 check: publish(t, "todo.completed", row.id, row.title)
In delete_todo, after the 404 check: publish(t, "todo.deleted", todo_id)

Publish after the database commit: never announce a change that might still roll back. In compose.yaml, add KAFKA_BOOTSTRAP: kafka:9092 to the api's environment.

Windows PowerShell: Get-Content sql/04-activity.sql | docker compose exec -T db psql -U postgres -d todo

Two consumers: the feed and the stats

Same topic, two consumer groups, so each gets every event.

40 min

app/consumers.py (new file)

The feed turns off auto-commit and commits each offset after the row is in Postgres. That gives at-least-once delivery: a crash can mean seeing a message twice, never losing it. The unique event_id makes the second time harmless.
Messages that can never be processed go to the dead-letter topic instead of blocking everything behind them.
On docker compose stop, the consumer finishes its message and calls close(), so its partitions move to other consumers at once.

Your todo ids depend on what's already in your table: use an id that belongs to tenant 1.

Real output (trimmed)

[{"type":"todo.completed","todo_id":1,"title":"Renew SSL","at":"2026-10-12T13:55:40.055194+00:00"},
 {"type":"todo.created","todo_id":1,"title":"Renew SSL","at":"2026-10-12T13:55:39.987908+00:00"}]   ← tenant 1 only (RLS)

Partition:1	2	{"event_id": "cae720ed-…", "type": "todo.created", "tenant_id": 2, … "title": "Ship v2", …}
Partition:2	1	{"event_id": "adacaf70-…", "type": "todo.created", "tenant_id": 1, … "title": "Renew SSL", …}
Partition:2	1	{"event_id": "c350edeb-…", "type": "todo.completed", "tenant_id": 1, … "title": "Renew SSL", …}

The key (1 or 2) decides the partition: both tenant 1 events are in partition 2, in order. The very first time, the feed can take 10–20 seconds to join its group before rows appear.

Watch the lag

20 min

Real output (columns trimmed)

GROUP           TOPIC        PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
activity-feed   todo-events  0          -               0               -
activity-feed   todo-events  1          1               1               0
activity-feed   todo-events  2          2               2               0

Break it on purpose

Six incidents, each one a real support ticket.

50 min

Windows: the loops below are bash. Run them in WSL Ubuntu, or in PowerShell 7 write the loop as 1..30 | ForEach-Object { ... } with curl.exe.

1. "The activity feed stopped updating." Stop the feed, keep using the app, then start it again.

Real output

PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
1          1               16              15
2          2               17              15        ← 30 events waiting; nothing lost
after start: caught up in about 11 seconds (container start included)

Kafka kept the events, so the consumer resumes from its committed offset. The lag panel in Grafana shows the same climb and fall.

2. "We added consumers but it isn't faster."

Real output (columns trimmed)

CONSUMER-ID                                   #PARTITIONS
rdkafka-08241046-ffce-48b7-ac1e-54dd76144bd6  1
rdkafka-0dbdd684-c5e1-448c-b3e9-3a872343a2df  1
rdkafka-7f7ab749-050c-47a8-810a-69ab091428b9  1
rdkafka-de65a4f2-d5c3-489f-97e8-cff82981ea88  0     ← 3 partitions = at most 3 busy consumers

3. "Some activity shows up twice." Make the consumer crash after writing a row but before committing its offset.

Real output

{"level": "error", "logger": "consumer", "msg": "simulated crash: row written, offset NOT committed"}
lag on that partition: 1         ← the event will be delivered again
 count
-------
     1                           ← redelivered, but ON CONFLICT skipped it

A consumer that crashes never says "I'm leaving", so the group waits for its session to time out (45 s by default) before handing its partitions to another consumer. That's why recovery can take a minute. Now remove the safety net: drop the unique constraint and the redelivered insert fails with there is no unique or exclusion constraint matching the ON CONFLICT specification. Remove ON CONFLICT too, and the same event is stored twice. Put both back.

4. "The feed is stuck." Publish a message that isn't JSON.

Real output

{"level": "error", "logger": "consumer", "msg": "bad event sent to DLQ", "partition": 1, "offset": 17,
 "error": "Expecting value: line 1 column 1 (char 0)"}
this is not json                 ← parked in the dead-letter topic; later events kept flowing

Without the except block, the consumer would crash before committing, restart, read the same message and crash again, forever. One bad message blocks its whole partition.

5. "Rebuild the feed from scratch." Replay: move the group back to the start of the topic.

Real output

GROUP           TOPIC        PARTITION  NEW-OFFSET
activity-feed   todo-events  0          0
activity-feed   todo-events  1          0
activity-feed   todo-events  2          0
activity rows: 49 before → 0 after TRUNCATE → 49 after replay

Run the reset while the consumer is still running and Kafka refuses: Error: Assignments can only be reset if the group 'activity-feed' is inactive, but the current state is Stable. Use --dry-run instead of --execute to preview.

6. "A todo exists but never appeared in the feed." Kafka is down while the API writes.

Real output

A: API 201 → event waited in the producer's memory → in the feed once Kafka returned
B: API 201 → todo saved in Postgres → API restarted → event gone (not in the topic, not in the feed)

The producer buffers and retries for a while (librdkafka's default message timeout is 5 minutes), but that buffer lives in process memory. The standard fix is the outbox pattern: write the event into an outbox table in the same transaction as the todo, and let a separate process publish from that table. The event is then as durable as the todo itself.

Issues you will hit on Day 7

Symptom	Cause	Fix
kafka-exporter exits: "client has run out of available brokers to talk to"	It started before Kafka was ready (hit in testing)	`restart: on-failure`
Subscribed topic not available: todo-events: Broker: Unknown topic or partition	Consumer started before the topic existed (hit in testing)	Create topics first; the consumer recovers once the topic exists
WARNING: topics with a period ('.') or underscore ('_') could collide	Kafka metric names treat `.` and `_` the same (hit with `todo-events.dlq`)	Use only one of them in topic names
Clients connect, then fail with "Connection refused" to another address	`ADVERTISED_LISTENERS` points somewhere the client can't reach	Advertise an address that works from where the client runs (`kafka:9092` inside Compose)
Lag panel shows −1	kafka-exporter reports −1 for partitions with no committed offset	`clamp_min(..., 0)`
Events of one tenant processed out of order	No key, or key isn't the tenant, so events spread across partitions	Key by the thing that needs ordering
Extra consumers sit idle	More consumers than partitions	Add partitions (plan ahead: you can add them but not remove them)
Duplicate rows after a crash	At-least-once delivery without an idempotent write	Unique event id + `ON CONFLICT DO NOTHING`
A partition stops moving and lag climbs	A poison message crashes the consumer every time	Catch it and send it to a dead-letter topic
Consumer takes a minute to resume after a crash	Its old session must time out (45 s default) before a rebalance	Expected after crashes; on normal shutdown call `close()`
Assignments can only be reset if the group is inactive	Offsets reset while consumers run (tested)	Stop the group's consumers first
A change saved but its event never arrived	Event lost in the producer's memory when the process stopped (tested)	Outbox pattern

Exit check

1. Why key by tenant_id? Same key → same partition → that tenant's events stay in order.

2. Feed and stats both read every event. Why don't they split them? Different consumer groups. Consumers split partitions only within one group.

3. Why commit the offset after the database write, not before? Before: a crash loses the event. After: a crash repeats it, and the unique event id makes repeats harmless.

4. 3 partitions, 5 consumers in one group. How many are busy? 3. The other 2 wait as spares.

5. A customer's todo exists but isn't in their feed, and Kafka had an outage. What happened, and what prevents it? The event was lost from producer memory; the outbox pattern.

After Day 7

From "it works" to "I can debug it in production"

Each item is about a weekend. Pick the ones that match the job you're aiming for.

Real auth. Replace the X-Tenant-ID header with a signed JWT, and read the tenant from the token. This is how Supabase does it.

Replication + backups. A primary and a replica, replication lag, reading stale data from a replica, point-in-time restore from WAL.

Postgres memory + disk. shared_buffers, cache hit ratio, work_mem spills to disk, WAL growth filling the disk.

Schema migrations. Alembic, plus safe production changes: CREATE INDEX CONCURRENTLY, adding columns without long locks.

Isolation levels. READ COMMITTED vs REPEATABLE READ vs SERIALIZABLE, and the bugs each one allows (lost updates, write skew).

Observability on k8s. kube-prometheus-stack with Helm, and the Day 4b Loki + Tempo setup moved into the cluster. Needs more Docker memory.

Vitess for real. The Vitess operator on kind, a resharding (2 → 4 shards), and the VReplication workflows PlanetScale support deals with.

More Temporal. Due-date reminders: one workflow per todo that sleeps until the day before, reminds the assignee, escalates if still open, and is cancelled by "mark done". Then trace across API → worker with Temporal's OpenTelemetry interceptor.