ReductStore's Blog

Sep 13, 2023. | By: Anthony Cavin

Benchmarks don’t lie, let’s put the systems to the ultimate test.

ReductStore vs. MinIO & InfluxDB on Edge Device HX401

For anyone deeply immersed in the engineering world of Edge Computing, Computer Vision, or IoT, you’ll want to read further to understand why a time series database for blob data is needed and where it stands out.

Enter our contest: First, we have ReductStore—a time series database for blob data—specifically designed for edge devices.

Its counterpart? The duo of MinIO and InfluxDB, each optimized for their niche in blob storage and time-series data respectively.

When directly compared, which system takes the lead in performance?

Let’s roll up our sleeves and deep-dive into this benchmarking analysis to separate fact from fiction.

[Read More]

Aug 15, 2023. | By: Alexey Timin

We are pleased to announce the release of the latest minor version of ReductStore, 1.6.0. ReductStore is a time series database designed for storing and managing large amounts of blob data.

To download the latest released version, please visit our Download Page.

What is new in 1.6.0?

Business Source License (BUSL-1.1)

We have updated the ReductStore license to the Business Source License (BUSL-1.1). This license permits free usage of the database for development, research, and testing purposes. Furthermore, it can be used in a production environment for free, provided that the Aggregate Financial Capacity of the company is less than $2,000,000 for the previous year. For additional information, please refer to here.

We believe that the new license strikes a good balance between freedom and revenue generation. This balance is necessary to maintain and improve our technology, and to bring benefits to its users.

[Read More]

Jul 7, 2023. | By: Anthony Cavin

Choosing the right database can be overwhelming–trust me, I know.

Photo by Jan Antonin Kolar on Unsplash

Since joining ReductStore’s project, I’ve been exploring alternative solutions to get a better understanding about how the project fits into current echosystem. I found all kind of databases, from the most popular ones to the most obscure ones.

To give you some context, we will look at solutions to store data from IoT devices (e.g. sensors, cameras, etc.) that commonly use MQTT to communicate with each other.

MQTT stands for “Message Queuing Telemetry Transport” and is a lightweight messaging protocol designed to be efficient, reliable, and scalable, making it ideal for collecting and transmitting data from sensors in real time.

Why is this important when choosing a database?

Well, MQTT is format-agnostic, but it works in a specific way. We should therefore be aware of its architecture, how it works, and its limitations to make the right choice. This is what this article is about, we will try to cut through the fog and explore some key factors to consider when selecting the right option.

Let’s get started!

[Read More]

Jul 2, 2023. | By: Alexey Timin

Hello everyone,

I’m happy to announce that the next minor version of ReductStore has been released. For the last month, we worked hard to improve the user experience when querying data from the database. And in this release, we deliver two important features:

• Batching multiple records into an HTTP response for read operations
• Reading only meta information about a record without its body.

Let me show you how it works in detail and how you can use it.

[Read More]

Jun 9, 2023. | By: Alexey Timin

I am happy to announce that we have completed the migration from C++ to Rust, and have released a stable version (v1.4.0) that is entirely written in Rust. 🤩

It was not an easy journey. After six weeks of coding, we encountered numerous regressions and changes in behavior. I needed to release two alpha and two beta versions with production testing to clean up the database. Now, it is finally ready!

[Read More]

Jun 4, 2023. | By: Alexey Timin

This article provides an introduction to ReductStore and explains how to use the Reduct C++ SDK to subscribe to data from the database.

Prerequisites

To subscribe to new records, we should use a continuous query, which has been supported by ReductStore since version v1.4. We can use the following Docker command to run it:

docker pull reduct/store:latest
docker run -p 8383:8383 reduct/store:latest 

Now, we need to install the Reduct Client SDK for C++. Please refer to these instructions.

[Read More]

May 9, 2023. | By: Alexey Timin

I previously wrote about my plan to rewrite ReductStore in Rust. I am happy to announce that the migration is now complete. It was a very interesting experience, and I would like to share it here.

Project Design before Migration

I wrote ReductStore in C++20, utilizing coroutines and ranges.

For the HTTP frontend, I used uWebSockets as an HTTP server and its event loop for coroutines.

The storage engine was implemented from scratch.

I used Protobuf as a JSON and binary serializer in both the HTTP frontend and the storage engine. Many of the structures were shared between the two.

I managed dependencies with Conan and used CMake as a build system.

Codebase about 20k lines with unit tests.

About Me

I have been developing in C++ and Python for about five years. Mostly, I write services for data acquisition, processing, and storage. I like OOP, design patterns, and my C++ smells Java. However, I avoid using exceptions and follow the RAII approach.

In Rust, I have only written the “Hello, World” example.

I work on the project in my spare time 10-20 hours a week.

Migration

Initially, I planned to use the cxx.rs library and rewrite the project in small steps by wrapping Rust code and integrating it into C++.

[SPOILER] I wasn’t able to handle this…

[Read More]

Apr 5, 2023. | By: Alexey Timin

Before we dive into the importance of data reduction for edge computing, it is important to define both terms. Data reduction refers to the process of reducing the amount of data that needs to be transmitted or stored, while still maintaining the necessary information and level of accuracy. This can be achieved through techniques such as compression, aggregation, and filtering.

[Read More]

Apr 1, 2023. | By: Alexey Timin

Initially, I chose to use C++ for the early editions of ReductStore because of my experience with the language. This allowed me to quickly create a functional time series database for binary data. However, as our platform expanded to include Windows and MacOS, I found myself struggling to manage the C++ infrastructure as the codebase grew. This made it difficult for me to focus on enhancing the product’s functionality and unique features, as I had to ensure compatibility across multiple platforms while managing numerous dependencies.

[Read More]

Mar 9, 2023. | By: Alexey Timin

Hey, I’ve released version 0.8.0 of Reduct CLI, the Python package for managing data stored in ReductStore. This release includes two new features that will be particularly helpful for our public datasets hosted on ReductStore, where metadata can be used to provide important context for the data.

1. The rcli export folder command now has a new option --with-metadata, which exports meta information about a record and its labels into a JSON file along with the content of the record.

2. The rcli export commands now accept integers for --start and --stop options, which means you can specify a time interval in Unix time or use it as a range of record IDs if you don’t care about time:

    # Export records with ID from 0 to 5 
    rcli export folder instance/bucket ./export_path --start 0 --stop 5
   

I hope these new features make it even easier to work with ReductStore and manage your data. As always, if you have any questions or feedback, please don’t hesitate to reach out!

Thanks for using ReductStore!

[Read More]

Mar 7, 2023. | By: Alexey Timin

The ReductStore Project hosts the free “cats” dataset, which contains about 10K photos of cats in JPEG format with eye coordinates, a month, and ears as labels. In this example, we can learn how to download the dataset from the ReductStore instance and draw the features using the OpenCV library.

Installing Dependencies

First, we need to install the ReductStore Client SDK for Python to download the photos and labels. We also need the OpenCV Python library for drawing features and Pillow to display images in the Jupyter environment:

pip install reduct-py opencv-python Pillow

Getting Data

To retrieve data we need the URL of the ReductStore instance, the bucket name, where we store our datasets and an API token with read access, so that we can connect to the database by using the Client class:

from reduct import Client, Bucket

HOST = "https://play.reduct.store"
API_TOKEN = "dataset-read-eab13e4f5f2df1e64363806443eea7ba83406ce701d49378d2f54cfbf02850f5"
BUCKET = "datasets"

client = Client(HOST, api_token=API_TOKEN)

bucket: Bucket = await client.get_bucket(BUCKET)

[Read More]

Mar 5, 2023. | By: Alexey Timin

Using snap, users can now easily install and manage ReductStore on various Linux distributions.

Snap is a universal package manager developed by Canonical, the company behind Ubuntu Linux. It allows developers to package their applications and dependencies into a single package that can be installed on any Linux distribution that supports snap, without worrying about different packaging formats and dependency conflicts.

[Read More]

Feb 22, 2023. | By: Alexey Timin

We are excited to announce the release of ReductStore Client SDK for C++ v1.3.0! This release includes support for the ReductStore API v1.3.0 with labels and content type.

[Read More]

Feb 19, 2023. | By: Alexey Timin

Hey everyone,

I’m happy to announce that we have released Reduct CLI client v0.7.0 with some minor improvements and bug fixes. We started using the tool in real applications and faced some problems exporting data from a ReductStore instance when the connection is slow and we have many entries to download asynchronously.

First of all, it wasn’t very convenient to count all needed entries in the rcli export command. Now we can use wildcards:

rcli export folder instance/bucket ./export_path  --entries=sensor-*

[Read More]

Feb 10, 2023. | By: Alexey Timin

Hey everyone,

we are pleased to announce the release of version 1.3.0 of the ReductStore SDK for JavaScript. This version supports the new features of ReductStore v1.3, including labels and content type.

Now you can write a record with MIME type and labels to ReductStore:

const client = new Client("https://play.reduct.store");

const bucket = await client.getOrCreateBucket("bucket");
    
const record = await bucket.beginWrite("entry-1", {
    contentType:"text/plain", 
    labels: {type:"example"}
})

await record.write("Some text");

[Read More]

Feb 7, 2023. | By: Alexey Timin

Hello everyone,

We’re excited to announce the release of version 0.6.0 of the Reduct CLI! This release brings a number of new features for working with the new ReductStore API v1.3.

[Read More]

Jan 26, 2023. | By: Alexey Timin

We are happy to announce the release of version 1.3.0 of the ReductStore SDK for Python! This release introduces several new features to help users better organize and filter their data.

One of the most notable new features is the ability to attach labels to data when writing and querying. Labels are key-value pairs that can be used to classify and categorize data. For example, you might use labels to store metadata about a record, such as its md5 sum or class. To start using labels, you need the version of the ReductStore database higher than 1.3.0.

[Read More]

Jan 26, 2023. | By: Alexey Timin

Hello everyone,

We are excited to announce the release of version 1.3.0 of the ReductStore database! This update brings a number of new features and improvements that we believe will enhance your experience with the database.

License Change

First and foremost, we’ve changed the project license, switching from AGPLv3 to MPL-2.0 which allows usage of the program as a service over the network in proprietary software. This change is made to avoid any misunderstanding in the future, and to align with our goal of encouraging contributions back to the project while allowing everyone to use it for free.

[Read More]

Jan 20, 2023. | By: Alexey Timin

Hello everyone,

We’re excited to announce the release of version 0.5.0 of the Reduct CLI! This release brings a number of new features to improve data export and mirroring. Here are the highlights:

Limit parallel tasks

We added a global option --parallel to limit the number of entries to be exported or mirrored in parallel. This can be useful when a bucket has a large number of entries and the network connection is slow. Usage example:

rcli export --parallel 5 folder server_1/bucket_1 . 

[Read More]

Jan 15, 2023. | By: Alexey Timn

This quick start guide will walk you through the process of installing and using the ReductStore JavaScript client SDK to interact with a ReductStore instance.

Installing the SDK

To install the ReductStore SDK, you will need to have Node.js 16 or higher installed on your machine. Once Node.js is installed, you can use the npm package manager to install the reduct-js package:

npm install reduct-js

[Read More]

Jan 8, 2023. | By: Alexey Timn

This quick start guide will walk you through the process of installing and using the ReductStore Python client SDK to interact with a ReductStore instance.

Installing the SDK

To install the ReductStore SDK, you will need to have Python 3.7 or higher installed on your machine. Once Python is installed, you can use the pip package manager to install the reduct-py package:

pip install reduct-py

[Read More]

Jan 7, 2023. | By: Alexey Timin

We often use blob storage like S3, if we need to store data of different formats and sizes somewhere in the cloud or in our internal storage. Minio is an S3 compatible storage which you can run on your private cloud, bare-metal server or even on an edge device. You can also adapt it to keep historical data as a time series of blobs. The most straightforward solution would be to create a folder for each data source and save objects with timestamps in their names:

bucket
 |
 |---cv_camera
        |---1666225094312397.bin
        |---1666225094412397.bin
        |---1666225094512397.bin

[Read More]

Jan 4, 2023. | By: Alexey Timin

I think, NGINX doesn’t need any introductions. It is one of the most widely used HTTP servers and reverse proxies. You can route your microservices or monolith application through it and make it responsible for:

• TSL encryption
• Basic HTTP authorization
• Map public URL to services or applications
• Load balancing

[Read More]

Dec 29, 2022. | By: Alexey Timin

The MQTT protocol is very popular in IoT applications. It is a simple way to connect different data sources with your application by using a publish/subscribe model. Sometimes you may want to keep a history of your MQTT data to use it for model training, diagnostics or metrics. If your data sources provide different formats of data that can not be interpreted as time series of floats, ReductStore is what you need.

Let’s make a simple MQTT application to see how it works.

[Read More]

Dec 26, 2022. | By: Alexey Timin

Hello everyone,

We’re excited to announce the release of version 0.4.0 of the Reduct CLI! This release includes a new rcli export folder command that allows you to export data from a bucket in your storage engine to a local folder on your computer. This can be useful if you want to make a copy of your data for backup or offline access. For more information, see the documentation.

[Read More]

Dec 23, 2022. | By: Alexey Timin

Hello, everyone! ReductStore has released v1.2.0 of its JavaScript SDK. This update includes support for ReductStore API version 1.2 with the new Client.me method, which allows you to retrieve information about your current API token and its permissions.

[Read More]

Dec 22, 2022. | By: Alexey Timin

Hello, everyone! We are excited to announce the release of the Python client SDK v1.2.0! This release includes support for the ReductStore HTTP API v1.2 and several other improvements.

[Read More]

Dec 20, 2022. | By: Alexey Timin

We are excited to announce the release of ReductStore Client SDK for C++ v1.2.0! This release includes updated documentation after we renamed the project from “Reduct Storage” to “ReductStore”, and supports ReductStore HTTP API 1.2.0 with the endpoint GET /api/v1/me.

[Read More]

Dec 19, 2022. | By: Alexey Timin

Hello, everyone!

We’re excited to announce the release of ReductStore v1.2.0. In this update, we’ve made some important changes to improve the way tokens and buckets are managed. Specifically, when you create a new token, the database will now check to ensure that the buckets specified in the token’s permissions actually exist. This will help prevent errors and ensure that users have the proper access to the appropriate buckets.

[Read More]

Dec 17, 2022. | By: Alexey Timin

This guide will help you get started with the ReductStore C++ SDK.

Requirements

The ReductStore C++ SDK is written in C++20 and uses CMake as a build system. To install it, you will need:

• C++ compiler with C++20 support (we use GCC-11.2)
• cmake 3.18 or higher
• conan 1.40 or higher (it is optional, but conan is a convenient package manager)

Currently, we have only tested the SDK on Linux AMD64, but if you need to port it to another operating system or platform, you can create an issue for assistance.

Installing

To install the ReductStore C++ SDK, follow these steps:

[Read More]

Dec 10, 2022. | By: Alexey Timin

When it comes to computer vision, data storage is a critical component. You need to be able to store images for model training, as well as the results of the processing for model validation. There are a few ways to go about this, each with its own advantages and disadvantages. In this post, we’ll take a look at three different ways to store data in computer vision applications: a file system, an S3-like object storage and ReductStore. We’ll also discuss some of the pros and cons of each option.

A Simple Computer Vision Application

For demonstration purposes, we’ll use a simple computer vision application which is connected to a CV camera and runs on an edge device:

The camera driver captures images from the CV camera every second and sends them to the model. The model detects something and shows the results in the user interface.

So far it isn’t too complicated, let’s see how we can deal with the data here.

[Read More]

Recent Posts

• ReductStore vs. MinIO & InfluxDB on LTE Network: Who Really Wins the Speed Race?

  #Comparison #Iot

• ReductStore 1.6.0 has been released with new license and client SDK for Rust

  #News

• How to Choose the Right MQTT Data Storage for Your Next Project

  #Advice #Database

• ReductStore v1.5.0 has been released

  #News

Popular Tags

Tutorials (15) Computer-vision (3) Sdks (11) News (19) Cli (5) Iot (3) Comparisons (1) Edge-computing (1) Advice (1) Database (1) Comparison (1)