Documents and files. One database, one binary.
Your data and your files, in one place you control. HexDB finds every photo, recording and document by what's inside it, lets the AI you choose tag it, and keeps it on your own servers or the cloud storage you already pay for. Encrypted, copied to every server, and back on its feet when one fails. One download, nothing else to run.
HexDB 1.5.0 for Windows, macOS, Linux and Docker, free and open source.
- Found by what's inside
- camera, place, artist, words, AI tags
- Your AI, your storage
- your keys, no lock-in
- For developers too
- REST, GraphQL and SQL
The one-slide version
HexDB, Explained at a High Level
No jargon. What it does for the business, what it replaces, and why it costs so little. Print it, put it on a screen, or press play and listen.
In one sentence: HexDB is a database that keeps your data and your files encrypted, copied and searchable, in a single program that runs anywhere, with no license fees.
- AES-256
- on every byte stored
- Auto
- failover when a server dies
- 52 MB
- for the whole install
- $0
- in license fees
Short on time? Listen instead. A 42-minute podcast explains, in plain English, how HexDB replaces the pile of cloud services most teams stitch together. Good for the commute before the meeting.
- 01
Stolen data is unreadable
Everything HexDB writes to disk is encrypted with AES-256, the standard banks and governments rely on. Without your key, a stolen disk or backup is noise.
Keys can be changed without stopping the service.
- 02
It keeps running when hardware fails
Run it on two or more servers and they keep full copies of each other. If the one in charge dies, another takes over in seconds, by itself.
In memory, each record is kept in six pieces with two spare. Any two can be corrupted and it is rebuilt.
- 03
It grows by adding servers
Need to serve more readers or survive more failures? Start another server with the same name and secret, in any cloud or your own building. It finds the others, copies the data and starts working.
No clustering software, no coordinator, no consultant.
- 04
It runs wherever your teams work
Linux, macOS, Windows and Docker. Official libraries for JavaScript, TypeScript, Python and .NET (including Entity Framework), an ODBC driver for reporting tools, and plain web requests for everything else.
Any language, any framework, any cloud or none.
- 05
It costs almost nothing to own
Open source, with no license fees. One small program (52 MB, admin console included) does the work of a database, a file store, a search engine, a message queue, a job scheduler and an admin console, so there is less to buy, run and secure.
Runs comfortably on a laptop: tens of thousands of writes a second, reads in a fifth of a millisecond.
- 06
It stores less and finds things faster
Records are compressed before they are encrypted, and a file stored twice is kept once, so disks and backups stay small. Indexes and full-text search find what you need without reading everything.
Reports and real-time alerts are built in, and BI tools can ask it questions in SQL.
New in 1.6
Your files live here too.
Photos, recordings, videos and documents are kept next to your data, with the same encryption, copies and permissions, on your servers or in the cloud storage you already pay for. HexDB reads what each file says about itself, and the AI service you choose can tag it too, so people find files by searching, not by remembering where they saved them.
How it works- Any kind of filePhotos, audio, video, design files, PDFs and office documents.
- Stored onceTen copies of the same file take the space of one.
- Found by what's insideSearch by camera, artist, place or author, or by tags an AI wrote.
- Kept where you chooseOn your own servers, or in S3, Azure, Google Drive or Dropbox.
What it replaces
- Database
- File storage
- Search engine
- Message queue
- Job scheduler
- Admin console
- Encryption layer
Fewer moving parts means fewer things to patch, monitor, license and explain to auditors.
Everything in the box
A whole data platform, not just a key-value store with ambitions.
Most databases make you bolt on search, a message bus, a scheduler, encryption, an admin console and somewhere to keep files. HexDB ships them in the same process, behind the same permissions, replicated the same way.
Hexes, tessellations, lattices
A lattice that elects its own leader and heals itself.
Start one hex and you have a database. Start more with the same lattice name and secret, on this machine or on hundreds of others, and they discover each other, elect an Overseer, replicate, and fail over on their own. No coordinator, no sidecar, no operator in the loop.
Four hexes, one Overseer. Replicas follow the change feed and serve reads.
Dashed hexes are ones you could add: on any cloud, in any region or on your own hardware, as many as you like.
HexDB is one self-contained binary for Windows, macOS and Linux on x86_64 or arm64, or a Docker image, with no cloud service behind it. A cloud VM, a server in your own building and a small arm64 box on a shelf can all be hexes in the same lattice, as long as they can reach each other.
Try it for real: node scripts/lattice-demo.mjs, then type stop 1.
Three roles
- Overseer. Takes writes, publishes the change feed, runs plugins, stream sources and schedules.
- Harvester. A read replica that follows the Overseer and can be elected if it fails.
- Replicant. A standby copy that follows but is never elected. For backups and rebuilding Harvesters.
The vocabulary
- Hex
- One running HexDB server.
- Tessellation
- A collection of documents, like a table. Roles are granted per tessellation.
- Lattice
- A group of hexes sharing a name and a secret that can reach each other. A single hex is a lattice of one.
- Vertex
- One of six separate regions of a hex's memory, each at its own addresses. Four data vertices each hold a quarter of every document; two parity vertices hold Reed-Solomon check data. A corrupt page in RAM only touches the one vertex that lives there: the document is rebuilt from the other five, the bad fragment is rewritten, and a whole vertex can be rebuilt and reassigned to fresh memory.
As many hexes as you have machines
The local demo runs three and the playground four, but nothing in HexDB caps the count. Put a hex on every server you own, in one rack or across regions: each needs only the lattice name, the secret and a few peers to start from, and joins as a replica with a full copy. More hexes means more read capacity and more machines you can lose. Writes still go through the one Overseer, so add hexes for reads and resilience rather than write throughput.
Durability is a dial
Replication is asynchronous by default. Two settings let you require that a write exists on replicas before it's acknowledged, and that an Overseer refuses writes when it can't see a majority.
# hexdb.toml: trade a little latency for a lot of safety
[replication]
min_acks = 1 # replicas that must hold a write before it's acknowledged
ack_timeout_ms = 5000 # on timeout the write stays committed; the client gets 503
quorum = 2 # hexes the Overseer must see to accept writes (no split brain)
forward_writes = true # a write sent to a replica is forwarded to the Overseernode scripts/lattice-demo.mjs # three hexes on 7800/7810/7820 with sample data
> stop 1 # stop the Overseer and watch another hex take over
> start 1 # it comes back as a replicaAnatomy of a hex
What one hex does with your data.
A hex is one running HexDB server. Step through with the arrows, or press play and it advances every few seconds, to follow a document from the moment a request arrives, through the write-ahead log, into six error-correcting vertices, out to the change feed, and down to disk.
A request arrives
REST, GraphQL, the admin UI and the drivers all come through the same door.
- One process serves the HTTP API, GraphQL and the admin UI, on one port.
- Every request needs credentials: a session cookie or an hxk_ API key. There is no unauthenticated mode, and unknown paths get 401 too.
- A write sent to a replica is forwarded to the Overseer, signed with the lattice key, and the response says where it went.
Step 1 of 7. Use the arrows to move at your own pace.
The parts, in one glance
- The server
- One Rust binary, hexdb_api, serving REST, GraphQL and the admin UI on a single port. The hexdb CLI starts, stops, inspects and backs it up.
- Tessellations
- Named collections of JSON documents, like tables. Each can have indexes, a text index with its own analyzer, and a versioned schema. Roles are granted per tessellation.
- Querying
- One filter language for equality, ranges, sets, prefixes and nested paths, with sorting, paging, counts and aggregations. Full-text search with stemming, n-grams and autocomplete. Spoken three ways: REST, GraphQL and read-only SQL. Projections pick fields in GraphQL (field paths) and SQL (select columns).
- Percolation
- Queries that wait for documents instead of the other way round. A stream source, a trigger or a plugin registers a filter, and every write that matches it is delivered as it commits: to a webhook, a script, Kafka, or a consumer group.
- Memory
- The newest unflushed documents plus an LRU cache of recent reads, within memory.ram_mb. Every document in memory is spread over six vertices, each a separate region of RAM at its own addresses: four data, two parity. A corrupt page damages one fragment, not the document.
- Disk
- A write-ahead log for durability, immutable SSTables for bulk storage, encrypted index snapshots, a sealed catalog, and a change-history archive. All AES-256-GCM.
- The catalog
- Tessellations, index definitions, schemas, the change-history ID and the lattice name, in one sealed file. Replicas copy it from the Overseer and re-read it within a poll when its fingerprint changes.
- Background work
- TTL sweeps, vertex integrity checks, flushes, compaction, metrics every 15 seconds, discovery probes, and, on the Overseer, plugins, stream deliveries, after triggers and schedules.
Playground
Try a live lattice.
Four hexes run on Fly.io as a lattice called Playground, loaded with about 85,000 documents (products, customers, orders, payments and shipments) and a files tessellation holding a whole album: Codex Animae by Ashes Gone, 15 WAV masters, three full-length music videos and the artwork. Sign in to the admin UI on the Overseer, query the lot in GraphQL or SQL, run the saved functions, edit the sandbox and watch its changes flow through a stream while the other three hexes keep up. The sandbox is reset every hour.
The lattice
Sign in to the admin UI
Everyone shares this login. It can read everything, write to the sandbox tessellation and watch the lattice; nothing else.
- User
- playground
- Password
- ········
Things to try
- Dashboard. The lattice card lists all four hexes with their roles and replication lag. Vertex health shows the six memory vertices.
- Queries. Switch between GraphQL and SQL, pick an example and press Ctrl+Enter: revenue by country, payments by method, or a SELECT with ? parameters. Aggregations run over the whole tessellation.
- Lookups. Find an order by order_no, then its payments and shipment by order_id. Those fields are indexed; a filter on status scans instead, so compare the timings.
- Functions. Run the saved ones: find_order by number, orders_by_status, revenue_by_country since a date, and place_sandbox_order, a transaction that writes to the sandbox. Each runs with your permissions.
- Streams. Open sandbox-events. Every write to the sandbox becomes a message in order: change a sandbox order on the Documents page and watch it arrive, or publish one yourself.
- Files. Open the Files page and browse media: play a WAV master from the list and seek through it, watch a music video, open the artwork, or search for "mermaid". In Queries, SELECT track, title, meta.duration_ms, meta.sample_rate FROM media WHERE artist = 'Ashes Gone' AND content_type = 'audio/x-wav' ORDER BY track.
- Configuration. Open Configuration at the bottom of the sidebar: file storage and AI settings for the whole lattice, write-only secrets, AI usage and token estimates, each hex's applied version, and the version history. It's read-only here.
- Documents. Pick sandbox, filter with JSON such as { "status": "paid" }, open one and change it. The write is forwarded to the Overseer if you are on a replica. The big tessellations are read-only.
The API
Small surface. Deep features.
If you can send JSON over HTTP you already know how to use HexDB. Here's what that buys you.
Plain HTTP. Every write is atomic and durable before it returns.
- PUT replaces, PATCH is a JSON merge patch, null removes a field.
- Bulk writes, update-by-filter and upserts are each a single atomic write.
- Every write takes an Idempotency-Key: a retry returns the original response, even after a crash.
- A write to a tessellation that doesn't exist creates it.
# Create: 201 with the document and a Location header. ?ttl=<seconds> sets an expiry.
curl -X POST http://localhost:7700/orders -H "Content-Type: application/json" \
-d '{ "order_no": "ORD-1001", "status": "pending", "total": 445.5, "customer": { "name": "Ada", "country": "GB" } }'
# Read, replace, merge-patch (null removes a field), delete
curl http://localhost:7700/orders/01JTY87RVJ9B5863KMB2YD896B # ETag = version
curl -X PUT http://localhost:7700/orders/01JTY... -d '{ "order_no": "ORD-1001", "status": "paid", "total": 445.5 }'
curl -X PATCH http://localhost:7700/orders/01JTY... -d '{ "status": "shipped", "notes": null }'
curl -X DELETE http://localhost:7700/orders/01JTY... # 204
# Bulk: one atomic write, up to 10,000 documents
curl -X POST http://localhost:7700/orders/_bulk -d '[{ "order_no": "ORD-1002" }, { "order_no": "ORD-1003" }]'
# Patch everything a filter matches
curl -X POST http://localhost:7700/orders/_update \
-d '{ "filter": { "status": "pending" }, "update": { "status": "paid" } }'
# => { "matched": 10, "modified": 10 }
# Upsert by key fields
curl -X POST http://localhost:7700/customers/_upsert \
-d '{ "key": ["email"], "documents": [{ "email": "ada@example.com", "name": "Ada", "tier": "gold" }] }'Drivers
Or skip HTTP and use a client.
Node.js, Python and .NET drivers cover documents, queries, aggregations, transactions, idempotency keys, the change feed, streams, functions and GraphQL. They retry 429 and 503 where it's safe to. On .NET there is also an Entity Framework Core provider: LINQ filters, sorting and paging run on the server, and each SaveChanges is one atomic transaction. Reporting tools connect through the ODBC driver, which speaks HexDB's read-only SQL.
import { HexDB } from "@dreaminhex/hexdb"
const db = new HexDB({ url: "http://127.0.0.1:7700", apiKey: process.env.HEXDB_API_KEY })
const orders = db.tessellation("orders")
await orders.insert({ customer: "ada", total: 12 })
const page = await orders.query({ filter: { total: { $gt: 10 } }, sort: "-total" })Storage engine
Built so you don't lose data. Or sleep.
A log-structured engine written from scratch in Rust: a write-ahead log with group commit, SSTables with compaction, encryption on every byte that touches disk, and error-correcting memory that repairs itself.
The write path
- 01Validate
Permissions, size limits, the schema, and expected versions for read-modify-write operations.
- 02Sequence and apply
Under one state lock the batch gets consecutive sequence numbers, is queued to the WAL and applied to memory and indexes. A query always sees its own writes.
- 03Group commit
A dedicated thread batches whatever is waiting into one write and one fsync. Zstandard first, then AES-256-GCM.
- 04Acknowledge and publish
Once durable, the write is acknowledged, published to the change feed, and, with min_acks, held until enough replicas confirm.
On disk and in memory
- Memory
- Newest unflushed documents plus an LRU cache, each spread over four data vertices and two parity vertices with Reed-Solomon coding, about 1.5 times the JSON size. Each vertex is a separate region of RAM, so a corrupt page costs one fragment, which is rebuilt from the others. On disk, authenticated encryption means a bad sector is detected rather than served, and every other hex holds a full copy. See the hex above.
- SSTables
- Immutable, per tessellation, with Bloom filters and a chunked encrypted index. Documents are stored as plain JSON, compressed with Zstandard, then encrypted: large structured documents shrink to about 16% of their size. About 2 bytes of RAM per document.
- Compaction
- Merges SSTables, drops tombstones and expired documents, and rewrites files on old formats or old keys.
- Recovery
- Replays the WAL in order. A torn final record is tolerated; anything that can't be decrypted stops startup rather than discarding data.
Encrypted at rest, rotated without downtime
WAL records, SSTable bodies and indexes, the catalog, settings, metrics and index snapshots are all AES-256-GCM with random 96-bit nonces, authenticated with their IDs so files and entries can't be swapped. Only SSTable headers are plaintext. A server whose data needs a key it doesn't have refuses to start rather than discard anything.
# Every developer and every deployment has its own key. Keep it.
printf '[storage]\nencryption_key = "%s"\n' "$(hexdb secret)" > hexdb_api/hexdb.local.toml
# Rotate without downtime: add the new key, move the old one to previous_encryption_keys,
# restart, and let compaction rewrite the files. Then drop the old key.
[storage]
encryption_key = "base64:NEW..."
previous_encryption_keys = ["base64:OLD..."]What's in the data directory
Everything a hex knows lives in one folder. Back it up online with one command, and restore by pointing a hex at the copy.
<storage.path>/
├── catalog.hxe tessellations, indexes, schemas, history ID, lattice name (sealed)
├── settings.hxe runtime settings changed from the UI or API (sealed)
├── metrics-history.hxe 6 hours of metrics samples (sealed)
├── query-stats.hxe the query advisor's statistics (sealed)
├── hexdb.pid PID, endpoint and shutdown token (owner-only)
├── wal/
│ ├── <first-seq>.wal active write-ahead log segments
│ └── archive/<first-seq>.wal the change history
├── indexes/<tessellation>/<index>.hxi index snapshots (sealed)
├── blobs/vertex-<n>/ file content: one pack series per vertex
│ ├── <number>.hxp a pack (sealed ones can also live on S3, Azure, Drive or Dropbox)
│ └── remote.json which packs are on the storage backend
└── <tessellation>/<ulid>.hxs SSTables, including system tessellationsFiles
Store any file. Find it by what's inside.
HexDB is more than a document database. It stores whole binary files, from JPEGs and WAVs to MP4s, PSDs and PDFs, chunked and named with BLAKE3, encrypted and spread across the vertices. Then it reads the metadata out of each file by its type, so a camera model, a GPS fix, an artist, a duration or a document author becomes a field you can query in GraphQL and SQL.
What happens to a file
- 01Stream it in
PUT a file to a folder and name, or send a large one in resumable 8 MB parts. Uploads stream to disk, never held whole in memory, and one sent to a replica streams on to the Overseer. The type comes from the file's first bytes, not its extension, and the file gets a SHA-256.
- 02Chunk with BLAKE3
Content-defined chunking (FastCDC) cuts the file into pieces of 512 KB to 8 MB, 2 MB on average, so an edit changes only the chunks around it. Each chunk is named by a keyed BLAKE3 hash of its content: identical content is stored once across every file, version and tessellation, and with the key secret, a chunk's name reveals nothing.
- 03Encrypt and spread
Each chunk is sealed with AES-256-GCM, then split with the same Reed-Solomon 4+2 code as documents: six fragments, one per vertex, each with its own BLAKE3 hash. Lose or corrupt any two and the chunk is rebuilt and repaired. Put the six vertices on six disks and a file survives losing two of them.
- 04Extract by file type
Once the content is fsynced and the document committed, the Overseer reads the file in the background with the extractors its type calls for: EXIF and XMP for a photo, ID3 for an MP3, QuickTime atoms for a MOV, document properties for a PDF. Images, PSDs and H.264 videos get a thumbnail: for a video, a keyframe about a third of the way in, decoded on the server. A file nothing can read is still found by name, folder, type, size, hash and dates.
Put files in
Over REST, from the hexdb command, from the Node.js, Python and .NET drivers, or by dragging them onto the admin UI's Files page. It browses by folder or by type, previews images, audio and video, and searches as you type, forgiving typos: "tatoo" still finds "tattooed".
# A files tessellation, then a photo into a folder
curl -X POST http://localhost:7700/tessellations \
-H "Content-Type: application/json" -d '{ "name": "media", "kind": "files" }'
curl -X PUT --data-binary @glacier-lagoon.jpg \
"http://localhost:7700/media/_files/2024/Iceland/glacier-lagoon.jpg"
# Or a whole folder from the command line, keeping its structure
hexdb put media ./Pictures -r --to /photos
hexdb ls media --search "glacier lagoon"
hexdb get media 2024/Iceland/glacier-lagoon.jpg -o ~/DownloadsSearch it in GraphQL and SQL
Every file is a document in a files tessellation, so GraphQL, SQL and the REST filters see its metadata like any other fields, with indexes, full-text search, aggregations, roles and row filters applied unchanged. A files tessellation starts with a text index over names, folders, tags and metadata; index a field such as meta.artist to make its filters fast. Queries only touch the documents, so searching a terabyte of video costs what searching its catalog costs.
# Photos taken in Iceland with a GPS position, newest first.
# Variables: { "filter": {
# "content_type": { "$startsWith": "image/" },
# "meta.location.country": "Iceland",
# "meta.gps": { "$exists": true } } }
query Photos($filter: JSON) {
documents(
tessellation: "media"
filter: $filter
sort: [{ field: "meta.taken_at", descending: true }]
limit: 50
) {
total
documents {
id
camera: field(path: "meta.camera.model")
gps: field(path: "meta.gps")
}
}
}-- The ten longest Miles Davis tracks
SELECT name, meta.title, meta.duration_ms FROM media
WHERE meta.artist = 'Miles Davis'
ORDER BY meta.duration_ms DESC LIMIT 10Further reading: FastCDC paper (USENIX ATC '16) · fastcdc for Rust · BLAKE3
What it reads
- ImagesJPEG, PNG, WebP, HEIC, AVIF, TIFF, camera RAW (CR3, NEF, ARW, DNG, ...), PSD
- EXIF with GPS, IPTC, XMP, PNG text, Photoshop size, layers and color mode, camera, lens and exposure, dimensions
- AudioMP3, AAC and M4A, FLAC, Ogg, Opus, WAV and Broadcast WAV, AIFF, WMA, APE
- ID3v1 and ID3v2, Vorbis comments, APE and MP4 tags, RIFF INFO, bext and iXML from field recorders, duration, bitrate, sample rate, channels
- VideoMP4, MOV, 3GP, MKV, WebM, WMV
- QuickTime metadata (camera, date, location), Matroska tags, MPEG-7 descriptions, duration, size and codec, and a thumbnail from an H.264 keyframe, upright even for portrait phone video
- DocumentsPDF, Illustrator, EPS, Word, Excel, PowerPoint, OpenDocument, EPUB, SVG, fonts
- Document properties and XMP, page and word counts. The start of the text of Office, OpenDocument, Markdown, SVG and plain-text files goes into full-text search
- Everything elseDICOM scans, ZIP, JAR, APK and TAR archives, any other binary
- DICOM tags (never patient names or IDs), archive listings, C2PA content credentials. Anything else is stored and found by name, folder, type, size, hash and tags
- Byte-exact
Images, audio and video, lossless or not, and already-compressed formats (ZIP, Office files, EPUB) are stored exactly as they arrived. Everything else gets Zstandard when it saves at least 5%, and that's lossless too.
- Replicated
Every hex copies the chunks it's missing from the Overseer, sealed in transit, and hexes repair each other: a chunk one hex can't rebuild is fetched from another.
- Versions
Uploading to the same path makes a new version and keeps the last ten. Versions share unchanged chunks, and restoring one makes a new version, so a rollback can itself be undone.
- Trash
Deleting moves a file to the trash for 30 days, where it can be restored, renamed if its old name is taken, or deleted for good.
- Downloads
Range requests, ETags and HEAD work as browsers and media players expect, so audio and video can seek. Stored HTML and SVG can't run scripts in a browser.
- Scrubbed
Every fragment is verified once a day and bad ones are rewritten. /status reports chunks, stored bytes, degraded chunks and repairs per vertex.
- Where it lives
On the hexes' own disks, or in S3 (and S3-compatible services), Azure Blob Storage, Google Drive or Dropbox. Content is encrypted and coded before it leaves, each hex keeps a local cache of what it reads, and switching providers moves everything across.
Performance
Measured on a laptop, with durability on.
Numbers from the benchmark that ships in the repository, run on an Intel Core i9-12900H laptop with a Samsung 980 PRO NVMe drive under Windows 11, server and client on the same machine. Every write is in the write-ahead log and fsynced before the client hears back. Run it on your own hardware and compare.
- 31,000documents/sBulk writes
1,000 documents per request, every write in the WAL and fsynced before it is acknowledged.
- 2.5msSingle write, median
One client, one document per request. 1,700 writes per second from 8 clients, thanks to group commit.
- 0.18msRead by ID, median
13,600 reads per second from 8 concurrent clients.
- 0.8msIndexed lookup, median
An equality query over 100,000 documents using a field index. SQL SELECT COUNT(*) answers in 0.4 ms from the maintained count.
- 52MBEverything installed
38 MB server with storage engine, REST, GraphQL, SQL and replication; 9 MB CLI; 3 MB ODBC driver; 2 MB admin UI.
- 1.5xJSON sizeMemory per document
Four data fragments plus two parity fragments across the six vertices, each with its hash.
- 16%of JSON on diskLarge structured documents
A 10 MB report compresses to 1.6 MB before encryption. Small documents average 81% after encryption and per-document overhead.
- 890MB/sAES-256-GCM on one core
Encrypting at 890 MB/s and decrypting at 1,150 MB/s, so encryption at rest is never the bottleneck.
- 0.4sRebuild a lost vertex
Repairing the fragments of 100,000 documents after a vertex is lost. Documents stay readable throughout, even with two of six vertices gone.
- 0.5sRestart after a crash
With data flushed to SSTables. Replaying 100,000 unflushed writes from the WAL takes 2.6 s.
- 2.6sFailover to a new Overseer
With discovery every second. With the default 10-second interval it is 31 s, because a hex is declared lost after three missed rounds.
- 600msWrite a 10 MB document
Structured JSON; reading it back takes 190 ms. Incompressible data writes in 94 ms and reads in 28 ms.
Not everything is fast. Aggregations group in memory after the filter, so a GROUP BY with SUM and AVG over 100,000 orders takes about 1.25 seconds. Documents over 1 MB need limits.max_document_kb raised, up to 64 MB.
The full report lists every measurement with how it was taken, including compression rates per document shape, Reed-Solomon encode and repair throughput, crash recovery with and without unflushed writes, and failover timing. The benchmark is part of the repository, so the same numbers can be produced on any machine:
cargo run --release -p hexdb_benchRecommended hardware
Starting points for each hex, from how HexDB uses memory, CPU, disk and network, and from what we learned running the playground. They aren't cloud benchmarks: run hexdb_bench on a candidate instance before you commit, and watch /status as real traffic arrives.
- Memory: budget a quarter of RAM
memory.ram_mb counts JSON bytes, but a document held in memory costs about 3.7 times that once it's parsed and spread over six vertices. Set ram_mb to roughly a quarter of the machine's RAM; whatever doesn't fit is read from the SSTables through the cache. On small Linux hosts, MALLOC_ARENA_MAX=2 keeps the allocator from growing past it.
- CPU: dedicated, never burstable
Burstable instances (AWS T, Azure B, GCP shared-core) are fine for one hex on a laptop budget, not for a lattice. When the credits run out, heartbeats miss, replicas think the Overseer is gone and elect another. Two vCPUs to start, four for production; aggregations and index builds use what you give them.
- Disk: low-latency SSD, room to compact
Every write is fsynced before it's acknowledged, so disk latency is write latency. Use SSD with at least 3,000 IOPS. Every hex holds a full copy, so size each disk for all the data after compression, the WAL and 24 hours of change history, with free space equal to your largest tessellation for compaction. Files take about 1.5 times their compressed content on disk, plus about 80 MB of RAM per TB for the chunk index.
- Network: keep the write path close
The Overseer streams every write to every hex, so replication traffic is your write volume times the number of replicas. Hexes in other regions or clouds are fine for reads; with replication.min_acks set, each acknowledged write waits for a round trip, so keep at least one replica near the Overseer. Open 7700 (API) and 7702 (discovery) between hexes.
- Overseer: give it the biggest machine
Elections rank hexes by RAM, then disk, so the largest hex leads, and it does the most work: it takes every write and feeds every replica. Size it first, or pin the role with identity.role. Lower network.discovery_interval_seconds for faster failover: 1 second takes about 2.6 s, the default 10 about 31 s.
- StartDevelopment, staging, a small app2 vCPU · 8 GB · 50 GB SSDram_mb = 2048
- AWS
- m7i.large x86 · m7g.large arm64gp3
- Azure
- D2s v5 x86 · D2ps v5 arm64Premium SSD v2
- GCP
- n2-standard-2 x86 · t2a-standard-2 arm64Hyperdisk Balanced
- ProductionEach hex of a production lattice4 vCPU · 16 GB · 200 GB SSDram_mb = 4096
- AWS
- m7i.xlarge x86 · m7g.xlarge arm64gp3, 3,000+ IOPS
- Azure
- D4s v5 x86 · D4ps v5 arm64Premium SSD v2
- GCP
- n2-standard-4 x86 · t2a-standard-4 arm64Hyperdisk Balanced
- LargeBig working sets, heavy aggregation, the Overseer8 vCPU · 64 GB · 500 GB+ SSDram_mb = 16384
- AWS
- r7i.2xlarge x86 · r7g.2xlarge arm64gp3, 6,000+ IOPS
- Azure
- E8s v5 x86 · E8ps v5 arm64Premium SSD v2
- GCP
- n2-highmem-8 x86 · c4a-highmem-8 arm64Hyperdisk Balanced
| Per hex | AWS | Azure | GCP |
|---|---|---|---|
Start Development, staging, a small app 2 vCPU · 8 GB · 50 GB SSD ram_mb = 2048 | m7i.large x86 m7g.large arm64 gp3 | D2s v5 x86 D2ps v5 arm64 Premium SSD v2 | n2-standard-2 x86 t2a-standard-2 arm64 Hyperdisk Balanced |
Production Each hex of a production lattice 4 vCPU · 16 GB · 200 GB SSD ram_mb = 4096 | m7i.xlarge x86 m7g.xlarge arm64 gp3, 3,000+ IOPS | D4s v5 x86 D4ps v5 arm64 Premium SSD v2 | n2-standard-4 x86 t2a-standard-4 arm64 Hyperdisk Balanced |
Large Big working sets, heavy aggregation, the Overseer 8 vCPU · 64 GB · 500 GB+ SSD ram_mb = 16384 | r7i.2xlarge x86 r7g.2xlarge arm64 gp3, 6,000+ IOPS | E8s v5 x86 E8ps v5 arm64 Premium SSD v2 | n2-highmem-8 x86 c4a-highmem-8 arm64 Hyperdisk Balanced |
The arm64 machines (AWS Graviton, Azure Ampere, Google Axion and Ampere) run the Linux arm64 build and usually cost less for the same shape. Any current general-purpose or memory-optimized family with the same shape works just as well.
Security
Secure by default, not by checklist.
There is no unauthenticated mode. Every request outside a short public list needs credentials, every refusal is audited, and everything on disk is encrypted with a key only you hold.
Thirteen permissions, any combination
Built-in roles cover the usual cases (reader, writer, owner, operator, auditor, ai, admin). Custom roles pick from the same thirteen permissions.
| Permission | Allows | Scope |
|---|---|---|
| read | Get, list, query, count, aggregate, the change feed; consume streams | per tessellation |
| write | Insert, replace, patch, delete; publish to streams | per tessellation |
| manage | Delete tessellations; manage indexes, schemas and advice; configure streams | per tessellation |
| upload | Upload files and new versions; create files tessellations (with write) | per tessellation |
| delete | Move files to the trash and delete them from it | per tessellation |
| restore | Restore files from the trash and earlier versions of files | per tessellation |
| status | Server status, metrics, storage and lattice details | global |
| logs | The server log | global |
| audit | The audit trail | global |
| plugins | Plugins and their delivery state | global |
| maintenance | Flush, compact and back up | global |
| ai | Run prompts through the lattice's AI provider, within its budget | global |
| admin | Everything: users, roles, settings, configuration, functions, schedules, joining hexes | global |
Grants, in one request
A grant names a role and the tessellations it applies to. Streams use the resource name stream:<name>. A grant on * covers everything.
# A grant names a role and the tessellations it applies to
curl -X POST http://localhost:7700/users -H "Content-Type: application/json" -d '{
"login": "ada", "password": "correct horse battery staple",
"roles": [{ "name": "writer", "tessellations": ["orders", "customers"] },
{ "name": "reader", "tessellations": ["*"] }] }'
# Custom roles from any set of permissions
curl -X POST http://localhost:7700/roles \
-d '{ "name": "analyst", "permissions": ["read", "status"] }'Row filters and hidden fields
A role can be restricted per tessellation: a filter that limits which documents it sees and writes, using the caller's login, email or attributes, and a list of fields it can never see or change.
# A role that only sees and writes its own region, and never sees cost or margin
curl -X POST http://localhost:7700/roles -H "Content-Type: application/json" -d '{
"name": "regional", "permissions": ["read", "write"],
"restrictions": { "orders": { "filter": { "region": { "$user": "attributes.region" } },
"hide": ["cost", "margin"] } } }'
curl -X PATCH http://localhost:7700/users/ada -d '{ "attributes": { "region": "EU" } }'
# Queries, counts, aggregations, GraphQL and the change feed all see only matching documents.
# Filtering or sorting on a hidden field is refused, so values can't be probed.- • Responses carry nosniff, DENY framing, a no-referrer policy and a Content Security Policy.
- • No CORS headers are sent, so other sites can't read responses. Cookie-authenticated changes must be same-origin (CSRF protection).
- • TLS is two settings away, with HSTS, Secure cookies and HTTPS replication between hexes.
Operations
A CLI and an API for everything the UI does.
Nothing in HexDB is only reachable by clicking. The hexdb command starts, stops, inspects, backs up and queries a server, and every dashboard action is a documented REST call you can script.
hexdb start [-s] start the server (-s: in the background)
hexdb stop [--force] stop it gracefully
hexdb health is it up?
hexdb status status and metrics
hexdb secret print a new random key
hexdb backup [--name N] [--list] back up the running server
hexdb sql "SELECT ..." [-p V] run a SQL query and print a table
hexdb put|get|ls <files> ... upload, download and list files
hexdb config get|set|history the lattice configuration, its versions and rollback
hexdb config connect dropbox sign in to Dropbox or Google Drive for file storage
hexdb plugins add|remove|list manage the plugin registry
hexdb lattice spawn --count 2 start more hexes that join this lattice
hexdb lattice list|stop list or stop themcurl http://localhost:7700/status # documents, memory, disk, vertices, lattice, lag
curl "http://localhost:7700/status/history?minutes=60" # a sample every 15 s, kept for 6 hours
curl "http://localhost:7700/logs?level=warn&limit=100" # tail, search and filter the log
curl -X POST http://localhost:7700/backup -d '{"name": "nightly-1"}' # consistent, while serving writes
curl http://localhost:7700/openapi.json # the REST API as OpenAPI 3.1- Backups
- Consistent and online. SSTables are hard-linked, the WAL is rotated, and writes keep flowing. Restore by pointing a hex at the copy with the same keys.
- Metrics and logs
- A sample every 15 seconds, kept for six hours across restarts. The last 5,000 log records are searchable live. Export both with the OpenTelemetry plugin.
- Settings and configuration
- A hex's limits, retention and session length change live from the UI or API. Lattice-wide settings such as file storage and AI are encrypted, versioned and checksummed on every hex, with history and one-step rollback.
- Upgrades
- Stop, replace the binaries and the UI, start. Older file formats are read and rewritten by compaction. In a lattice, upgrade replicas first and let failover move the Overseer.
- Packaging
- A Windows installer and a macOS installer, archives for every platform, Debian packages from an APT repository, a Docker image and compose file, a systemd unit, and Homebrew, Chocolatey and winget packages.
- Health checks
- /health is public and cheap. /status adds documents, memory, disk, vertices, lattice membership and replication lag for anything that wants detail.
Admin UI
A control room in every hex.
Every HexDB server hosts its own admin UI. Watch the lattice, browse and edit documents, run GraphQL, manage indexes, schemas, streams, functions, users and roles, read the audit trail and tail the log, all from one place and all through the same permissions as the API.
What you're looking at
- 1Only the pages your role allows
The sidebar shows what the signed-in user may use. Everything else is hidden here and refused by the server anyway.
- 2Live figures
Documents in memory and on disk, tessellations, memory and disk against their budgets, writes and reads per minute, uptime and version.
- 3Activity, 15 minutes to 6 hours
Documents per tessellation, operations per minute, or storage over time. The history survives restarts.
- 4Vertex health
The six memory vertices of this hex, each its own region of RAM, with its size and repairs. Any two can fail and documents are rebuilt from the rest; a failed vertex is rebuilt and reassigned.
- 5The lattice
Every hex with its role, address, replication state and lag. “Add a hex” shows the settings a new server needs to join.
- 6Flush and back up
Write unflushed data to SSTables or take a consistent online backup, right from the dashboard, if your role has the maintenance permission.
- Served at /ui/ by every hex, replicas included. No separate service to deploy or secure.
- It calls the same API as any client, with the signed-in user's permissions, so what you see is what the API allows.
- Light, dark and system themes. Keyboard shortcuts in the query console (Ctrl+Enter runs).
- Sessions use HttpOnly, SameSite cookies with CSRF protection; MFA and API keys are managed on the Account page.
Every page, and what it does
- Dashboard
- Live document, memory, disk and operation figures, activity charts, vertex health, the lattice and 'Add a hex'
- Queries
- A GraphQL and SQL console with completion, examples, history, and JSON or table results
- Tessellations
- Create and delete; manage indexes (with analyzer choice and suggestions) and schemas, with version history and rollback
- Documents
- Browse with filters and sorting; create, edit and delete documents, and filter files by their metadata
- Files
- Browse folders, upload by dragging, preview and play media, search metadata and AI tags, and manage versions and the trash
- Streams
- Create streams, publish, read messages, watch consumer groups and deliveries
- Functions
- Create and run functions; manage schedules and triggers
- Users, Roles
- Accounts, role grants, user attributes, custom roles with row filters and hidden fields, MFA resets
- Audit Trail
- Security events, filterable by user, action and time
- Plugins
- Loaded plugins, their state and deliveries
- Logs
- The server log, tailed live
- Configuration
- File storage and AI for the whole lattice: write-only secrets, AI usage and cost estimates, each hex's applied version, history and rollback
- Settings
- This hex's runtime settings and its effective configuration
- Account
- Your password, MFA and API keys
Users, Roles, Audit Trail, Plugins, Logs and Settings appear only for roles that hold the matching permission.
Ecosystem
Plugs into what you already run.
Plugins extend HexDB without changing it. Each is a folder with a manifest: a process in any language reading JSON lines on stdin, a webhook, or a built-in exporter. They run on the Overseer, so a lattice delivers each payload once, and they resume from a saved position after a restart.
| Plugin | Type | What it does |
|---|---|---|
| @streams/kafka | stream | Produces changes to Kafka, keyed by document ID |
| @streams/kinesis | stream | Puts changes on an Amazon Kinesis stream |
| @sinks/otlp | metrics | Metrics to an OpenTelemetry endpoint (built in) |
| @sinks/log-file | logs | Rotating JSON-lines log files |
| @sinks/http-logs | logs | Log records to Splunk HEC, Logstash and friends |
| @sources/postgres | source | Copies a PostgreSQL table and keeps it in sync |
| @sources/mssql | source | The same for SQL Server |
| @sources/s3 | source | Imports JSON and JSON-lines objects from S3 |
| @enrichers/ai-keywords | stream | Adds keywords to new documents with the lattice's AI provider |
| @examples/change-logger | stream | Logs each change; the minimal example |
Process plugins run in a clean environment with only the variables they're given. The server's own keys are never passed.
import { payloads, api, log } from "../../sdk/hexdb-plugin.mjs"
for await (const change of payloads()) {
if (change.op === "put") log("order", change.id, change.document.total)
}
await api().post("/orders", { total: 12 }) // plugins with [access]Get Started
From a clean machine to a running lattice in minutes.
Use the Windows or macOS installer, a package manager, Docker, or build it yourself. Every way gives you the server, the hexdb command, the admin UI and the ODBC driver.
HexDB for Windows
Run the installer, or use a package manager. Either way you get the server, the hexdb command, the admin UI and the ODBC driver.
64-bit installer (.msi)
- Installs into C:\Program Files\HexDB and adds it to the PATH. Windows 10 or later, 64-bit; nothing else is needed.
- The first start creates your configuration and encryption key in %LOCALAPPDATA%\HexDB and tells you where the generated admin password is.
- The ODBC driver is registered as "HexDB", ready for Excel, Power BI and anything else that speaks ODBC.
winget install DreamInHex.HexDB
# or
choco install hexdb
# Then, in a new terminal
hexdb startnode scripts/seed.mjs loads sample data.Implement
Get it wherever you already get software.
Every release is published to the package managers below on the same day, with SHA-256 checksums on GitHub. The ODBC driver comes with HexDB itself, on every platform.
HexDB
The server, the hexdb command, the admin UI and the ODBC driver.
Windows installer, macOS installer, archives for every platform, .deb packages and checksums
github.com/dreaminhex/hexdb/releases- winget (opens in a new tab)In review
The Windows Package Manager, built into Windows 10 and 11
winget install DreamInHex.HexDBSubmitted; this works once the first version is approved.
- Chocolatey (opens in a new tab)In review
Runs the Windows installer silently
choco install hexdbSubmitted; this works once the first version is approved.
macOS and Linux, with a background service
brew install dreaminhex/hexdb/hexdbGitHub Container Registry, amd64 and arm64
docker pull ghcr.io/dreaminhex/hexdb
Client libraries
For applications that talk to HexDB. Each covers documents, queries, aggregations, transactions, streams, functions and files.
JavaScript and TypeScript, for Node.js, Deno, Bun and browsers
npm install @dreaminhex/hexdbHexDB.EntityFrameworkCore, an EF Core 8 provider
dotnet add package HexDB.EntityFrameworkCore
Your first request
Once it's running, every feature is an HTTP call away. Insert a document and query it back; the answer says which index it used.
# Insert a document
curl -X POST http://localhost:7700/orders \
-H "Authorization: Bearer $HEXDB_TOKEN" \
-H "Content-Type: application/json" \
-d '{ "customer": "ada", "total": 42, "status": "paid" }'
# Query it back, largest first
curl -G http://localhost:7700/orders \
-H "Authorization: Bearer $HEXDB_TOKEN" \
--data-urlencode 'filter={"status":"paid","total":{"$gte":10}}' \
--data-urlencode 'sort=-total'{
"documents": [
{ "id": "01JTY87RVJ9B5863KMB2YD896B",
"customer": "ada", "total": 42, "status": "paid" }
],
"total": 1,
"plan": { "indexes": ["status"], "scanned": 1 }
}FAQ
Questions people ask first.
Contact
Talk to the person who built it.
Questions about a deployment, a feature you need, a bug you hit, or a project you'd like help with. Messages go straight to the maintainer.
HexDB is open source and built in the open. Stars, issues and pull requests are all welcome.