Indexing the Data Lake for Online Point Queries

1PB of EBS volumes on sc1-class HDD is roughly $15k/month. Let's say you need at least 2 copies of each byte for cross-AZ availability and durability, that's 30k per PB for JUST the drives.

These disks also need compute attached to them, coupling their scaling needs. Now you have a fleet of (expensive) VMs, regardless of how many queries end up hitting that database.

Of course, those VMs have to run pretty complex software such as PlanetScale, which has to take care of replication, backups, concurrency control, sharding, indexing, vaccuuming, and a whole bunch of other things that add operational complexity and require expertise. That's the fixed costs of an OLTP database. A lot of that cost is derived from the need to, well, process transactions.

Thing is, they already have a data lake. So the fixed cost to store a PB of data on S3 is already paid. They mention the index is roughly 3 orders of magnitude smaller than the data set (1PB of data is roughly 1TB of index). With a marginally less complex compute stack, I believe this becomes cost effective pretty quickly.