12 posts tagged with "Company"

Cloud native databases are designed from the ground up to embrace core cloud principles: distributed architecture, automatic scalability, high availability, and elasticity. A prominent example is Amazon Aurora, which established the prevailing paradigm for online databases by championing the decoupling of compute and storage. This architecture allows the compute layer (responsible for query and transaction processing) and the storage layer (handling data persistence) to scale independently. As a result, database users benefit from granular resource allocation, cost efficiency through pay-per-use pricing, flexibility in hardware choices, and improved resilience by isolating persistent data from ephemeral compute instances.

In this blog post, we re-examine this decoupled architecture through the lens of cloud storage mediums. We argue that this prevailing model is at a turning point, poised to be reshaped by the emerging synergy between instance-level, volatile NVMe and highly durable object storage.

On October 20, 2025, AWS experienced a major disruption across multiple services in the us-east-1 region. According to AWS Health Status, various compute, storage, and networking services were impacted simultaneously. For many teams running OLTP databases on instances backed by local NVMe, this was not just a downtime problem-it was a data durability nightmare.

Cloud databases must constantly balance durability, performance, and cost. In modern cloud environments, there are three main types of storage available:

Let’s break down the trade-offs and why recent events place a spotlight on risky architectural choices.

Option 1: Block-Level Storage (EBS) - Durable but Expensive and Slow​

EBS is the default choice for reliability:

• It survives instance failures.
• It supports cross-AZ replication via multi-replica setups.
• It enables quick reattachment to replacement nodes.

But the downside?

• GP2/GP3 disks deliver modest IOPS and high latency.
• High-performance variants like IO2 are extremely expensive when provisioned for hundreds of thousands of IOPS.
• Scaling performance often means scaling cost linearly.

EBS gives you durability-but performance per dollar is disappointing.

Option 2: Local NVMe - Fast but Ephemeral (and Now Proven Risky)​

Instance families like i4i provide 400K+ to 1M+ IOPS from local NVMe, making them a natural fit for databases chasing performance.

So many database vendors recommend:

• Use local NVMe for primary storage
• Add cross-AZ replicas for durability

But here’s the problem: Local NVMe is tied to the node lifecycle. If the node restarts, fails, gets terminated due to spot interruption, or is impacted by a region-level failure such as the recent us-east-1 outage-you lose ALL the data.

During routine failures, cross-AZ replicas often protect you. But during region-wide degradation or cascading incidents, with local NVMe, there is nothing to recover. The storage is simply gone. What you can do is to recovery from recent backups - often lagging days. Write loss is guaranteed between last backup and crash.

In contrast, EBS volumes can always be reattached to a new node.

The AWS us-east-1 outage just validated that “local NVMe + async replication” is a high-risk strategy for mission-critical databases.

Option 3: Object Storage (S3) - Durable & Cheap, But Latency Is a Challenge​

Object storage is:

• 3x cheaper than block storage
• Regionally and cross-region durable
• Built to survive region-level failures
• Practically infinite
• A first-class citizen for modern cloud-native platforms

But the challenge remains: S3 latency is too high for OLTP if accessed synchronously.

This is why traditional OLTP engines avoid it.

So the question becomes: How do we get the cost & durability benefits of S3 without paying the latency penalty?

The Data Substrate Approach: Object Storage First, NVMe as Cache, EBS for Logs​

EloqData treats object storage (e.g., S3) as the primary data store, and architect the system to avoid the usual latency pitfalls:

Through Data Substrate, we decouple storage from compute and split durability between:

• Log: persists immediately to EBS
• Data store: periodically checkpointed to S3 (async + batched)
• NVMe: purely a cache layer, safe to lose at any time

This allows us to:

• Withstand node crashes seamlessly
• Recover fully even if local NVMe is wiped
• Handle region-level disruption by replaying logs and checkpoints
• Enjoy millions of IOPS from NVMe without durability risk
• Cut storage cost by 3x+ compared to full EBS-based systems

Check out more on our products powered by Data Substrate:

• EloqKV
• EloqDoc
• EloqSQL

The Larger Industry Trend​

We are not alone in this shift. The broader ecosystem is moving object-storage-first:

EloqData brings this model to OLTP with a transactional, low-latency engine powered by Data Substrate.

After the Outage: A Hard Question Every Architect Should Ask​

If my database node died right now, would I lose all my data?

If you're running a primary database on local NVMe, and relying solely on async replicas, the answer might be yes.

It’s time to rethink durability assumptions in the cloud era.

Summary​

AWS us-east-1 just reminded the industry: Performance is replaceable. Lost data is not.

With the right architecture, you don’t have to choose.

• Build fast.
• Stay durable.
• Be outage-proof.

That’s the future we’re building at EloqData.

Check out more on our open source databases:

• EloqKV
• EloqDoc
• EloqSQL

Online databases are the backbone of interactive applications. Despite coming in many different types, online databases are all engineered for low-latency, high-throughput CRUD operations. At EloqData, we use the universal Data Substrate to build online databases for any model—from key-value and tables to JSON documents and vectors. In this post, we explore one of our core engineering practices for future online databases.

In the previous article, we discussed the details of some of the architecture design of Data Substrate. In this article, we continue the discussion and elaborate on why we made these design choices and how these choices affect the resulting database solutions we built.

In this article, we dive deeper into the technical foundations of Data Substrate—highlighting the key design decisions, abstractions, and architectural choices that set it apart from both classical and modern distributed databases.

At EloqData, we've developed Data Substrate—a database architecture designed to meet the unprecedented demands of modern applications in the AI age. Unlike traditional database systems that struggle with the scale and complexity of AI workloads, Data Substrate reimagines the database as a unified, distributed computer where memory, compute, logging, and storage are fully decoupled yet globally addressable.

This series of articles explore the motivations, technical foundations, and benefits of Data Substrate, providing a comprehensive understanding of how this architecture addresses the critical challenges facing modern data infrastructure in the AI age.

Some of the topics covered are rather heavy in technical jargons, and require a good understanding of database internal mechanisms to appreciate. We apologize in advance.

1. Data Substrate: Motivation and Philosophy​

This article introduces the core philosophy behind Data Substrate. We explore why traditional database architectures fall short in the AI era and present our vision for a new approach that treats the entire distributed system as a single, unified computer.

2. A Deeper Dive Into Data Substrate Architecture​

This technical deep-dive explores the architectural foundations of Data Substrate. We examine the key design decisions, abstractions, and technical choices that set Data Substrate apart from both classical and modern distributed databases.

3. The Benefits of Data Substrate Architecture​

This article examines the practical benefits and real-world implications of Data Substrate. We discuss how our design choices translate into concrete advantages for modern applications, particularly in cloud environments.

Why Data Substrate Matters​

Traditional database architectures were designed for a different era—one where data volumes were smaller, workloads were more predictable, and the demands of AI applications were unimaginable. Data Substrate represents a fundamental rethinking of database design, built from the ground up for the challenges and opportunities of the AI age.

By treating the distributed system as a single, unified computer, Data Substrate eliminates many of the complexities that have traditionally made distributed databases difficult to build, operate, and reason about. This approach enables:

• Modular architecture enables community collaboration and avoid reinventing the (many) wheels
• True scalability without sacrificing consistency
• Independent resource scaling for compute, memory, logging, and storage
• Better performance through optimized hardware utilization and innovative algorithm design
• Cloud-native features like auto-scaling and scale-to-zero
• Simplified development through familiar single-node programming models

Get Started with Data Substrate​

Ready to explore Data Substrate in action? Our open-source implementations are available on GitHub:

• EloqKV: A high-performance key-value store built on Data Substrate
• EloqSQL: A MySQL-compatible distributed SQL database
• EloqDoc: A document database for modern applications

Join our Discord community to connect with other developers and stay updated on the latest developments in Data Substrate technology.

At the recent Data Stream Summit 2025, Hubert Zhang, CTO of EloqData, delivered a talk on building elastic, agentic AI data pipelines using Apache Pulsar and EloqDoc.

In the previous blog, we discussed the future database foundation for Agentic AI Applications. In this blog we will simplify the agentic application and use EloqKV as data store to explore EloqKV's decoupled architecture.

We have recently open sourced our three products: EloqKV, EloqSQL, and EloqDoc. These offerings reflect our commitment to addressing the evolving demands of modern data infrastructure, particularly as we enter an era dominated by powerful, autonomous AI systems.

LLM-powered Artificial Intelligence (AI) applications are driving transformative changes across industries, from healthcare to finance and beyond. We are rapidly entering the Agentic Application Age, an era where autonomous, AI-driven agents not only assist but actively make decisions, manage tasks, and optimize outcomes independently.

However, the backbone of these applications—the data infrastructure—faces immense challenges in scalability, consistency, and performance. In this post, we explore the critical limitations of current solutions and introduce EloqData’s innovative approach specifically designed to address these challenges. We also share our vision for an AI-native database, purpose-built to empower the Agentic Application Age, paving the way for smarter, more autonomous, and responsive AI applications in the future.

We have recently introduced EloqKV, our distributed database product built on a cutting-edge architecture known as Data Substrate. Over the past several years, the EloqData team has worked tirelessly to develop this software, ensuring it meets the highest standards of performance and scalability. One key detail we’d like to share is that the majority of EloqKV’s codebase was written in C++.

In the previous blog, we discussed the durable feature of EloqKV and benchmarked the write performance of EloqKV with the Write-Ahead-Log enabled. In this blog, we will continue to explore the transaction capabilities of EloqKV and benchmark the performance of distributed atomic operations using the Redis MULTI EXEC commands.

In this blog post, we introduce our transformative concept Data Substrate. Data Substrate abstracts core functionality in online transactional databases (OLTP) by providing a unified layer for CRUD operations. A database built on this unified layer is modular: a database module is optional, can be replaced and can scale up/out independently of other modules.