Vlad

30 pillars of YT Cluster

Articles · YT Cluster

From 30 years of personal IT experience into 30 pillars of the solution: what YT Cluster solves, on the PostgreSQL you already run. Each title links to its detail - how it works and where it is tested in public - in 30 pillars of YT Cluster on dbinvent.com.

  1. Performance growth without limits - Grow as far as the load goes, with no vendor change and no data transfer - still your favorite, proven PostgreSQL.
  2. No vendor lock-in - Your own solution: like a cloud database, but cheaper and better - your hardware, your rules, nobody else’s price list.
  3. Same tech stack, simple onboarding - No learning curve and no source code change: the same proven PostgreSQL code, SQL, drivers and tools. Your team is onboarded already - and simple onboarding is money saved.
  4. High availability, multi-master - Every node writes - strong on write and still fast. A usual PostgreSQL cluster is a complicated manual setup: a single primary with a standby, a limited number of replicas that are slow when strongly consistent, or eventual consistency.
  5. No cap on nodes - Other cloud databases with two-phase-commit transactions limit the number of nodes, and get slower on writes as they grow. Here a write waits only for the nodes its table asks for, and clusters link into a hierarchy.
  6. Survives network outages - A node cut off by the network keeps running and writing the part of the schema you allow, and catches up when the link is back.
  7. Self-healing, verified - Copies are re-checked in the background and repaired by majority vote. A row nobody can vouch for is blocked, not guessed at.
  8. Consistency per table - Each table sets how many nodes must confirm a commit - half, N, or none - and whether conflicting writes are refused. One cluster holds both the strict tables and the fast ones.
  9. Topology by rules, changed online - Zones, tiers, shards and data residency are declared as rules. When a rule changes, partitions move while being written to - no redeploy, no downtime.
  10. Serverless Python, built in - A serverless framework like AWS Lambda, inside the cluster: Python functions run on insert, update and delete, next to the data.
  11. Distributed cache and queue - A built-in distributed key-value cache with a message queue, behind a standard etcd API - no extra cluster to run.
  12. KV and point-to-point streams, any language - Key-value, change streams and point-to-point queues all speak the standard etcd API v3, so any language with an etcd client can use them - no vendor API to learn.
  13. Key to queue to Python - no SQL on the way - A message written to a queue reaches your Python function with no SQL and no database on the way, so none of their latency. When the function needs to save or load, the DB binding is right there.
  14. Secret key-value store - Built in: secrets kept encrypted in the cluster itself, readable only on the nodes you approve - no separate vault to deploy.
  15. LLM-ready - Natural language to SQL, available today. No GPU on every node - one model box on the side serves the whole cluster.
  16. Built-in MPP - Massively parallel analytics on Apache DataFusion - up to 4× faster than Spark - across the nodes you already run.
  17. ClickHouse in one click - One command sets up ClickHouse as a shadow cluster for your analytics data.
  18. Schema management, without Java - Regular schema management, as with Flyway: versioned, immutable and repeatable SQL scripts, tracked by checksum per environment, with full psql support and built-in ETL - in a single binary, with no JVM and no JDBC.
  19. Cluster-ready migrations - Unique to Schema Guard: each migration is ordered against ordinary writes on every node of the cluster - otherwise the nodes stop agreeing about what their data means. It ships in the package and is configured in the same table.
  20. Dynamic schema adaptation, open source - It reads the schema an existing database already has, and applies only the declarative changes needed to match the one declared in YAML. This part is open source, MIT licensed.
  21. Hierarchical and regional boundaries - Link clusters vertically and side by side. Up the hierarchy go the analytics, down it come the domain dictionaries - only the tables a rule names, as read-only copies. Regions are neighbours: each cluster keeps its own leader, quorum and data.
  22. Operated through standard SQL - Almost every operation is plain SQL: settings and operator actions are rows in one configuration table, jobs are rows too, and status is a select - no custom command set to learn.
  23. Secret management and security - A node joins only once an operator approves it, and nodes talk over mutual TLS with certificates from the cluster’s own CA. The master key is wrapped per approved node, rotated online, revoked with a retired node and recoverable with a key you keep.
  24. Audit ready - The source code of ytctl, ytserv and the PostgreSQL extension is open to audit at any time, under either license: a security review or a compliance check gets the code itself, not a promise.
  25. Console UI installer - ytctl walks you through installing a node in a console UI: it checks the dependencies and the PostgreSQL version, then installs the extension, the configuration table and the service, showing every command it runs. A second console UI controls the running cluster, and --batch installs unattended.
  26. Add a node with no data - A new node starts empty, copies the tables it holds from a member while the cluster keeps writing, and joins once it has caught up. Pausing and rejoining are requests the leader approves.
  27. Agent ready - agent.md puts the whole platform on one page, every subsystem with an honest status: an AI agent onboards as simply as an engineer does, and builds on nothing that is only configured.
  28. Non-blocking concurrent document operations - Two writers editing different paths of one jsonb document both land, on every node, and neither waits on the other. The changes commute - Base + A + B = Base + B + A - so they apply in any order; edits to the same path are refused, never silently overwritten. Same model as our open-source diff-doc-rs.
  29. Leave any time - a full rollback - Sign a node out by deleting its roster row, and every member retires it. To leave altogether, ytctl --purge stops the service and removes the extension’s files and the cluster WAL; DROP EXTENSION takes the extension’s tables and triggers with it. Your data never moves: each node is plain PostgreSQL again - no vendor lock, for real.
  30. A Rust server - no JVM - ytserv, the daemon on every node, is written in Rust: fast, memory safe and small, with no JVM to install, tune or pause for garbage collection. So are ytctl, the etcd server, rppd and Schema Guard; the PostgreSQL extension is a small C library.