Benchmarks don’t lie, let’s put the systems to the ultimate test.
ReductStore vs. MinIO & InfluxDB on Edge Device HX401
For anyone deeply immersed in the engineering world of Edge Computing, Computer Vision, or IoT, you’ll want to read further to understand why a time series database for blob data is needed and where it stands out.
Enter our contest: First, we have ReductStore—a time series database for blob data—specifically designed for edge devices.
Its counterpart? The duo of MinIO and InfluxDB, each optimized for their niche in blob storage and time-series data respectively.
When directly compared, which system takes the lead in performance?
Let’s roll up our sleeves and deep-dive into this benchmarking analysis to separate fact from fiction.
We are pleased to announce the release of the latest minor version of ReductStore, 1.6.0. ReductStore is a time series database designed for storing and managing large amounts of blob data.
To download the latest released version, please visit our Download Page.
We have updated the ReductStore license to the Business Source License (BUSL-1.1). This license permits free usage of the database for development, research, and testing purposes. Furthermore, it can be used in a production environment for free, provided that the Aggregate Financial Capacity of the company is less than $2,000,000 for the previous year. For additional information, please refer to here.
We believe that the new license strikes a good balance between freedom and revenue generation. This balance is necessary to maintain and improve our technology, and to bring benefits to its users.
Choosing the right database can be overwhelming–trust me, I know.
Photo by Jan Antonin Kolar on Unsplash
Since joining ReductStore’s project, I’ve been exploring alternative solutions to get a better understanding about how the project fits into current echosystem. I found all kind of databases, from the most popular ones to the most obscure ones.
To give you some context, we will look at solutions to store data from IoT devices (e.g. sensors, cameras, etc.) that commonly use MQTT to communicate with each other.
MQTT stands for “Message Queuing Telemetry Transport” and is a lightweight messaging protocol designed to be efficient, reliable, and scalable, making it ideal for collecting and transmitting data from sensors in real time.
Why is this important when choosing a database?
Well, MQTT is format-agnostic, but it works in a specific way. We should therefore be aware of its architecture, how it works, and its limitations to make the right choice. This is what this article is about, we will try to cut through the fog and explore some key factors to consider when selecting the right option.
Let’s get started!
Hello everyone,
I’m happy to announce that the next minor version of ReductStore has been released. For the last month, we worked hard to improve the user experience when querying data from the database. And in this release, we deliver two important features:
Let me show you how it works in detail and how you can use it.
I am happy to announce that we have completed the migration from C++ to Rust, and have released a stable version (v1.4.0) that is entirely written in Rust. 🤩
It was not an easy journey. After six weeks of coding, we encountered numerous regressions and changes in behavior. I needed to release two alpha and two beta versions with production testing to clean up the database. Now, it is finally ready!
This article provides an introduction to ReductStore and explains how to use the Reduct C++ SDK to subscribe to data from the database.
To subscribe to new records, we should use a continuous query, which has been supported by ReductStore since version v1.4. We can use the following Docker command to run it:
docker pull reduct/store:latest
docker run -p 8383:8383 reduct/store:latest
Now, we need to install the Reduct Client SDK for C++. Please refer to these instructions.
I previously wrote about my plan to rewrite ReductStore in Rust. I am happy to announce that the migration is now complete. It was a very interesting experience, and I would like to share it here.
I wrote ReductStore in C++20, utilizing coroutines and ranges.
For the HTTP frontend, I used uWebSockets as an HTTP server and its event loop for coroutines.
The storage engine was implemented from scratch.
I used Protobuf as a JSON and binary serializer in both the HTTP frontend and the storage engine. Many of the structures were shared between the two.
I managed dependencies with Conan and used CMake as a build system.
Codebase about 20k lines with unit tests.
I have been developing in C++ and Python for about five years. Mostly, I write services for data acquisition, processing, and storage. I like OOP, design patterns, and my C++ smells Java. However, I avoid using exceptions and follow the RAII approach.
In Rust, I have only written the “Hello, World” example.
I work on the project in my spare time 10-20 hours a week.
Initially, I planned to use the cxx.rs library and rewrite the project in small steps by wrapping Rust code and integrating it into C++.
[SPOILER] I wasn’t able to handle this…
Before we dive into the importance of data reduction for edge computing, it is important to define both terms. Data reduction refers to the process of reducing the amount of data that needs to be transmitted or stored, while still maintaining the necessary information and level of accuracy. This can be achieved through techniques such as compression, aggregation, and filtering.
Initially, I chose to use C++ for the early editions of ReductStore because of my experience with the language. This allowed me to quickly create a functional time series database for binary data. However, as our platform expanded to include Windows and MacOS, I found myself struggling to manage the C++ infrastructure as the codebase grew. This made it difficult for me to focus on enhancing the product’s functionality and unique features, as I had to ensure compatibility across multiple platforms while managing numerous dependencies.
Hey, I’ve released version 0.8.0 of Reduct CLI, the Python package for managing data stored in ReductStore. This release includes two new features that will be particularly helpful for our public datasets hosted on ReductStore, where metadata can be used to provide important context for the data.
The rcli export folder
command now has a new option --with-metadata
, which exports meta information about a
record and its labels into a JSON file along with the content of the record.
The rcli export
commands now accept integers for --start
and --stop
options, which means you can specify a time
interval in Unix time or use it as a range of record IDs if you don’t care about time:
# Export records with ID from 0 to 5
rcli export folder instance/bucket ./export_path --start 0 --stop 5
I hope these new features make it even easier to work with ReductStore and manage your data. As always, if you have any questions or feedback, please don’t hesitate to reach out!
Thanks for using ReductStore!
The ReductStore Project hosts the free “cats” dataset, which contains about 10K photos of cats in JPEG format with eye coordinates, a month, and ears as labels. In this example, we can learn how to download the dataset from the ReductStore instance and draw the features using the OpenCV library.
First, we need to install the ReductStore Client SDK for Python to download the photos and labels. We also need the OpenCV Python library for drawing features and Pillow to display images in the Jupyter environment:
pip install reduct-py opencv-python Pillow
To retrieve data we need the URL of the ReductStore instance, the bucket name, where we store our datasets and an API token with read access, so that we can connect to the database by using the Client class:
from reduct import Client, Bucket
HOST = "https://play.reduct.store"
API_TOKEN = "dataset-read-eab13e4f5f2df1e64363806443eea7ba83406ce701d49378d2f54cfbf02850f5"
BUCKET = "datasets"
client = Client(HOST, api_token=API_TOKEN)
bucket: Bucket = await client.get_bucket(BUCKET)
Using snap, users can now easily install and manage ReductStore on various Linux distributions.
Snap is a universal package manager developed by Canonical, the company behind Ubuntu Linux. It allows developers to package their applications and dependencies into a single package that can be installed on any Linux distribution that supports snap, without worrying about different packaging formats and dependency conflicts.
We are excited to announce the release of ReductStore Client SDK for C++ v1.3.0! This release includes support for the ReductStore API v1.3.0 with labels and content type.
Hey everyone,
I’m happy to announce that we have released Reduct CLI client v0.7.0 with some minor improvements and bug fixes. We started using the tool in real applications and faced some problems exporting data from a ReductStore instance when the connection is slow and we have many entries to download asynchronously.
First of all, it wasn’t very convenient to count all needed entries in the rcli export
command. Now we can use
wildcards:
rcli export folder instance/bucket ./export_path --entries=sensor-*
Hey everyone,
we are pleased to announce the release of version 1.3.0 of the ReductStore SDK for JavaScript. This version supports the new features of ReductStore v1.3, including labels and content type.
Now you can write a record with MIME type and labels to ReductStore:
const client = new Client("https://play.reduct.store");
const bucket = await client.getOrCreateBucket("bucket");
const record = await bucket.beginWrite("entry-1", {
contentType:"text/plain",
labels: {type:"example"}
})
await record.write("Some text");
Hello everyone,
We’re excited to announce the release of version 0.6.0 of the Reduct CLI! This release brings a number of new features for working with the new ReductStore API v1.3.
We are happy to announce the release of version 1.3.0 of the ReductStore SDK for Python! This release introduces several new features to help users better organize and filter their data.
One of the most notable new features is the ability to attach labels to data when writing and querying. Labels are key-value pairs that can be used to classify and categorize data. For example, you might use labels to store metadata about a record, such as its md5 sum or class. To start using labels, you need the version of the ReductStore database higher than 1.3.0.
Hello everyone,
We are excited to announce the release of version 1.3.0 of the ReductStore database! This update brings a number of new features and improvements that we believe will enhance your experience with the database.
First and foremost, we’ve changed the project license, switching from AGPLv3 to MPL-2.0 which allows usage of the program as a service over the network in proprietary software. This change is made to avoid any misunderstanding in the future, and to align with our goal of encouraging contributions back to the project while allowing everyone to use it for free.
Hello everyone,
We’re excited to announce the release of version 0.5.0 of the Reduct CLI! This release brings a number of new features to improve data export and mirroring. Here are the highlights:
We added a global option --parallel
to limit the number of entries to be exported or mirrored in parallel. This can be
useful when a bucket has a large number of entries and the network connection is slow. Usage example:
rcli export --parallel 5 folder server_1/bucket_1 .
This quick start guide will walk you through the process of installing and using the ReductStore JavaScript client SDK to interact with a ReductStore instance.
To install the ReductStore SDK, you will need to have Node.js 16 or higher installed on your machine. Once Node.js is
installed, you can use the npm
package manager to install the reduct-js
package:
npm install reduct-js
This quick start guide will walk you through the process of installing and using the ReductStore Python client SDK to interact with a ReductStore instance.
To install the ReductStore SDK, you will need to have Python 3.7 or higher installed on your machine. Once Python is
installed, you can use the pip
package manager to install the reduct-py
package:
pip install reduct-py
We often use blob storage like S3, if we need to store data of different formats and sizes somewhere in the cloud or in our internal storage. Minio is an S3 compatible storage which you can run on your private cloud, bare-metal server or even on an edge device. You can also adapt it to keep historical data as a time series of blobs. The most straightforward solution would be to create a folder for each data source and save objects with timestamps in their names:
bucket
|
|---cv_camera
|---1666225094312397.bin
|---1666225094412397.bin
|---1666225094512397.bin
I think, NGINX doesn’t need any introductions. It is one of the most widely used HTTP servers and reverse proxies. You can route your microservices or monolith application through it and make it responsible for:
The MQTT protocol is very popular in IoT applications. It is a simple way to connect different data sources with your application by using a publish/subscribe model. Sometimes you may want to keep a history of your MQTT data to use it for model training, diagnostics or metrics. If your data sources provide different formats of data that can not be interpreted as time series of floats, ReductStore is what you need.
Let’s make a simple MQTT application to see how it works.
Hello everyone,
We’re excited to announce the release
of version 0.4.0 of the Reduct CLI!
This release includes a new rcli export folder
command that allows you to export data from a bucket in your storage
engine to a local folder on your computer.
This can be useful if you want to make a copy of your data for backup or offline access. For more information,
see the documentation.
Hello, everyone!
ReductStore has released v1.2.0 of its
JavaScript SDK. This update includes support
for ReductStore API version 1.2 with the
new Client.me
method, which allows you to retrieve information about your current API token and its permissions.
Hello, everyone! We are excited to announce the release of the Python client SDK v1.2.0! This release includes support for the ReductStore HTTP API v1.2 and several other improvements.
We are excited to announce the release
of ReductStore Client SDK for C++ v1.2.0! This release
includes updated
documentation after we renamed the project from “Reduct Storage” to “ReductStore”, and supports ReductStore HTTP API
1.2.0 with the endpoint GET /api/v1/me
.
Hello, everyone!
We’re excited to announce the release of ReductStore v1.2.0. In this update, we’ve made some important changes to improve the way tokens and buckets are managed. Specifically, when you create a new token, the database will now check to ensure that the buckets specified in the token’s permissions actually exist. This will help prevent errors and ensure that users have the proper access to the appropriate buckets.
This guide will help you get started with the ReductStore C++ SDK.
The ReductStore C++ SDK is written in C++20 and uses CMake as a build system. To install it, you will need:
Currently, we have only tested the SDK on Linux AMD64, but if you need to port it to another operating system or platform, you can create an issue for assistance.
To install the ReductStore C++ SDK, follow these steps:
When it comes to computer vision, data storage is a critical component. You need to be able to store images for model training, as well as the results of the processing for model validation. There are a few ways to go about this, each with its own advantages and disadvantages. In this post, we’ll take a look at three different ways to store data in computer vision applications: a file system, an S3-like object storage and ReductStore. We’ll also discuss some of the pros and cons of each option.
For demonstration purposes, we’ll use a simple computer vision application which is connected to a CV camera and runs on an edge device:
The camera driver captures images from the CV camera every second and sends them to the model. The model detects something and shows the results in the user interface.
So far it isn’t too complicated, let’s see how we can deal with the data here.
ReductStore vs. MinIO & InfluxDB on LTE Network: Who Really Wins the Speed Race?
#Comparison #IotReductStore 1.6.0 has been released with new license and client SDK for Rust
#NewsHow to Choose the Right MQTT Data Storage for Your Next Project
#Advice #DatabaseReductStore v1.5.0 has been released
#NewsTutorials (15) Computer-vision (3) Sdks (11) News (19) Cli (5) Iot (3) Comparisons (1) Edge-computing (1) Advice (1) Database (1) Comparison (1)