Architecture Overview

This section explains how LatticeDB's storage engine works, from the ground up. Each chapter builds on the previous, showing how simple primitives combine to create a durable, transactional graph database with vector search, full-text search, and local event streams.

The Stack

The engine stack from top to bottom: the application calls the transaction manager, which drives the B+Tree, write-ahead log and checkpointer; those share the buffer pool, which sits on the page manager, the virtual file system, and finally the operating system

Chapters

Storage Engine

  1. Virtual File System - Abstracting file I/O for portability and testing
  2. Portable Databases - Serializing to bytes, and running with no files at all
  3. Page Manager - Fixed-size pages, allocation, and checksums
  4. Buffer Pool - Caching pages in memory with eviction
  5. B+Tree - Ordered key-value storage with efficient lookups

Durability & Transactions

  1. Write-Ahead Log - Durability through logging before data changes
  2. Transaction Manager - ACID transactions with begin/commit/abort
  3. Checkpointing - Bounding recovery time by flushing dirty pages
  4. Recovery - ARIES-style crash recovery with redo/undo

Data Model

  1. Graph Storage - Nodes, edges, labels, and properties
  2. Durable Streams - Transactional event records and semantic graph changes

Search Indexes

  1. Vector Search - HNSW approximate nearest neighbor search
  2. Full-Text Search - BM25-scored inverted index, per-property declarations, and search as an access path

Query System

  1. Query Execution - Volcano iterator model, operators, and planning

Design Principles

Direct Page Manipulation

We don't deserialize pages into objects. Instead, we read/write bytes directly in page buffers. This is "zero-copy" - no intermediate representations, no serialization overhead.

Traditional access deserialises page bytes into an in-memory object, modifies it, and serialises back. LatticeDB modifies the page bytes in place through calculated offsets, with no intermediate object

Everything is Pages

The entire database is built on fixed-size pages (4KB by default):

  • B+Tree nodes are pages
  • Large B+Tree values spill into linked overflow pages
  • WAL frames are pages
  • The file header is a page
  • Free list entries are pages

This uniformity simplifies the system - one caching layer, one I/O path, one checksum format.

Durability Through WAL

Changes are logged before being applied. This means:

  • Commit = log is on disk (fast, sequential write)
  • Actual data pages can be written lazily
  • Crash recovery replays the log

Simple Concurrency Model

Each page has a reader-writer latch. Multiple readers OR one writer. No complex lock hierarchies - simplicity over maximum concurrency.