We are excited to announce the release of our most recent benchmark study on NoSQL databases, comparing throughput and scalability using Apache Cassandra 2.0.9, Couchbase Server 2.5.1, and MongoDB 2.6.4.
We shared the preliminary results from part one of the study in June in which we measured how much load we could put through the databases while keeping most of the working set in RAM. We found Couchbase to be the clear leader as far as sheer throughput when 90% of data is in-memory, followed by MongoDB and Cassandra. All databases served over 60,000 transactions per second with Couchbase far exceeding that with over 900,000 TPS.
In the second part of this study, we measure and analyze the linearity of scaling, with the predominant question being, does the throughput of these databases increase linearly as clusters are added, and do they do so with consistent levels of latency? We ran this second test on different hardware and with the data fitting entirely within memory.
To conduct these tests we measured throughput and latency as we added clusters for both heavy-read and balanced workloads:
Again we found that for the kind of dataset being measured, where the amount of data fits comfortably in RAM, Couchbase had much higher throughput than the other databases.
We measured latency at a “near maximum” point which we defined somewhat subjectively to be a point which was close to a database’s maximum possible throughput but before the point latency would start to increase substantially to serve limited additional traffic. Our logic was that this is how most organizations would provision their database environment.
As you can see from the graphs, all the databases were able to scale traffic without substantial effect on latency.
For heavy-read workloads (95% read, 5% write), both Couchbase and MongoDB scale close to linearly. For Cassandra, the answer is less clear, as the last two nodes had more impact than the first 2. A cluster size of 4 is not common for Cassandra installations.
For balanced workloads (50% read, 50% write), MongoDB showed the best scaling behavior. Couchbase and Cassandra both show scaling at about half of linear capacity throughout. Cassandra actually scales slightly better for the balanced workload.
*Couchbase sponsored this study. Sponsorship in this case means that the vendor offered to offset a portion of the benchmarking costs if we were to do a study measuring performance and scalability across popular NoSQL databases while keeping most of the working set in RAM. We believed this to be a useful and interesting aspect to study for ourselves and the community. It does not mean Couchbase participated in the creation of the actual test design, nor in the testing, nor in compiling the results.