> For the complete documentation index, see [llms.txt](https://bucketdb.sullux.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://bucketdb.sullux.com/readme.md). # Overview BucketDB is a lightweight Node.js library that uses S3-compatible storage as its backing store to implement a document database with schema-driven binary encoding. ## Mission Our primary mission is to provide rock-bottom pricing by leveraging S3's cost structure, without sacrificing consistency and reliability. We aim to hit the "boring web app" sweet spot: applications with high consistency and reliability requirements but low performance demands (e.g., 100ms response times are acceptable, and volume is in the hundreds of writes per day rather than thousands per second). We believe in complete freedom from vendor lock-in. BucketDB works seamlessly with any S3-compatible provider including AWS S3, Cloudflare R2, Tigris, MinIO, and Backblaze B2. This gives you the flexibility to choose the most cost-effective or geographically optimal storage layer for your needs. ## Vision and Philosophy Most websites—even fairly complex ones—have high requirements for consistency, redundancy, and reliability, but very low requirements for performance. A 100ms read or write is perfectly acceptable. Total write volume might be in the hundreds of writes per day, not millions. Unfortunately, most distributed database solutions fall into one of three categories: 1. **Proprietary and expensive** (DynamoDB, Firebase) 2. **Cheap but requiring heavy self-management** (PostgreSQL, MySQL) 3. **Open-source but vendor-locked** (MongoDB Atlas, Supabase) `bucket-db` targets a different sweet spot: **reliable, consistent, cheap, and vendor-agnostic**. Our philosophy is built on the following core tenets: * **Simplicity Over Features:** We intentionally exclude data integrity constraints from our schemas. A schema in BucketDB dictates *how* to store your data efficiently via custom binary serialization, not whether your data is valid. We believe that schema validation and business rules (e.g., a maximum size for an integer column) belong firmly in your business logic layer, not in the storage engine. * **Immutable Data:** Updates do not mutate state; they append state. We rely on block-based storage with fixed-width rows and variable-length heaps, ensuring complete audit trails and simplifying replication. * **Vanilla JavaScript Only:** We value auditable, readable code. You will find no TypeScript, no heavy transpilation steps, and minimal external dependencies in our codebase. * **Dual Support:** First-class support for both CommonJS (CJS) and ECMAScript Modules (MJS) without code duplication. ## Target Audience BucketDB is designed specifically for: * **Backend Developers** building low-cost applications. * **Teams** utilizing event sourcing patterns (e.g., storing projections). * **Projects** deployed on modern edge platforms (like Fly.io) that offer predictable, synchronized system clocks. * **Small-to-Medium SaaS Applications** with low write volumes but critical reliability needs. ## What This Is *Not* Good For BucketDB is intentionally constrained. It is not suitable for: * Real-time, high-throughput transactional processing (OLTP) on single records. * Environments where eventual consistency cannot be bridged by a local cache. * Client-side execution (Node.js only; no browser or Electron support). * Environments with heavily skewed system clocks (nodes must stay within \~1 second of each other). ## Architectural Overview At its core, BucketDB operates on a **Write-Forward** pattern utilizing **immutable blocks**. Modifying files directly in S3 suffers from eventual consistency issues and slow locking. Instead, BucketDB appends commits to a distributed Write-Ahead Log (WAL). ### Timesharing and Distributed Authority Because multiple server instances (nodes) might want to write to the database simultaneously, BucketDB uses a "taking turns" mechanism known as a **timesharing model**. Nodes coordinate via wall clock time to share write authority. Only the node currently with write authority has the permission to commit writes. This single-write-point model bridges S3's eventual consistency gap. The node with write authority groups incoming writes into a single batch and commits them. Other nodes periodically pull these commits into their local caches. ### Immutable Blocks When a node commits data, it generates a new immutable data block (configurable, default 64KB) containing fixed-width rows and a variable-length heap. This ensures that readers never encounter a partially updated file. When the primary metadata block (Block 0) is updated, it atomically swaps pointers to the newly written data blocks. ```mermaid flowchart TD App[Application Node] --> API[BucketDB API] API --> Cache[Local Committed Cache] API --> WF[Write-Forward Queue] WF --> Timesharing[Timesharing WAL Authority] Timesharing --> S3[S3 / S3-Compatible Storage] ``` For a deeper dive into Block 0, B+Tree indexes, and the Write-Forward log, please consult our comprehensive [Architecture reference](/core-concepts/architecture.md). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://bucketdb.sullux.com/readme.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.