Storage Tracking & Retention — Org-Level Storage Metering
Overview
A heavy nightly S3 scan that could take down a node, replaced with a real-time counter that never throws away the data it measures.
The problem
There was no aggregation, alerting, or enforced limit on how much storage an organization could use, only a raw total shown to the user. That total came from a single nightly cron that scanned every org's S3 bucket from scratch to compute usage. The scan was heavy enough that it sometimes brought down the whole service node it ran on, it only ever reflected the previous night's numbers, so a user's usage never updated in real time, and whenever a user deleted something it was hard-deleted immediately, which meant that data was permanently gone and unusable for any later analysis or model training.
What I built
Replaced the nightly scan entirely with a `total_storage` field on the org document in MongoDB, updated incrementally in real time on every upload or delete instead of recomputed from scratch. For production, I ran a one-off backfill script that walked existing media and calculated each org's starting total before the field went live. On top of that I built a storage page where users can see how their usage breaks down by type, photos, videos, and map-assets, plus a monthly breakdown over time. I introduced a 50GB storage limit per organization, with an automated alert email sent once usage crosses 80% of that limit. Deletes no longer remove data outright: instead of a raw S3 delete, the object is moved into a separate archive bucket, so the underlying media is preserved for later analysis or model training even after a user deletes it from their view.
Impact
Removed a cron heavy enough to take down a service node and replaced it with an incremental counter that costs almost nothing to keep updated. Usage numbers are now live instead of a day stale, organizations get a real limit with an 80% warning before they hit it instead of finding out after the fact, and deleted media is archived rather than destroyed, turning what used to be permanently lost data into a retained dataset.