A year ago, we migrated ReductStore, time series database for unstructured data, from C++ to Rust. Now, reflecting on that decision after a year, it's clear that it was the right move, yielding the following benefits:

• Fewer bugs related to memory management, move semantics, and threading
• Easier codebase porting to MacOS and Windows
• Better dependency management with Cargo compared to what we had with C++

Today, I'm pleased to announce that we've rewritten the CLI client from Python to Rust. Our primary motivation was distribution. Although Python is one of the most widely used programming languages, we don't want to require users to install an interpreter to use our tools, and standalone installers are too bulky for us. With Rust, we can build compact and blazingly fast executable binaries for most popular platforms.

wget https://github.com/reductstore/reduct-cli/releases/latest/download/reduct-cli.linux-amd64.tar.gz
tar -xvf reduct-cli.linux-amd64.tar.gz
chmod +x reduct-cli
sudo mv reduct-cli /usr/local/bin

In edge computing, managing time series blob data efficiently is critical for performance-sensitive applications. This blog post will compare ReductStore, a specialized time series database for unstructured data, and MongoDB, a widely-used NoSQL database.

Using Docker containers for straightforward setup, we'll examine the speed of each system. We'll go through setting up ReductStore buckets and preparing MongoDB collections, focusing on how to effectively store and access blob data for time series scenarios.

By conducting performance tests on binary data insertion and retrieval, we aim to provide insights into which system might best serve your application's needs.

For those interested in replicating our benchmarks or conducting their own evaluations, we've made our methods easily accessible through this repository.

TimescaleDB is an open-source time-series database optimized for fast ingest and complex queries. It is engineered up from PostgreSQL and offers the power, reliability, and ease-of-use of a relational database, combined with the scalability typically seen in NoSQL systems. It is particularly suited for storing and analyzing things that happen over time, such as metrics, events, and real-time analytics.

Since TimescaleDB is based on PostgreSQL, it supports blob data and can be used to store a history of unstructured data such as images, binary sensor data, or large text documents. In this article, we will use the database as a time-series blob storage and compare its performance with ReductStore, which is designed specifically for this use case.

TimescaleDB and ReductStore both have Python Client SDKs. We'll create simple Python functions to read and write data, then compare performance with different blob sizes. To repeat these benchmarks on your own machine, use this repository.

The Robot Operating System (ROS) stands as a versatile framework for developing sophisticated robotic applications with various sensors, including cameras. These cameras are relatively inexpensive and widely used as they can provide a wealth of information about the robot's environment.

Processing camera output with computer vision requires efficient solutions to handle massive amounts of data in real time. ROS 2 is designed with this in mind, but it is a communication middleware and does not provide a built-in solution for storing and managing large volumes of image data.

Addressing this challenge, this blog post will guide you through setting up ROS 2 with ReductStore—a time-series database for unstructured data optimized for edge computing, ensuring your robotic applications can process and store camera outputs effectively.

We are pleased to announce the release of the latest minor version of ReductStore, 1.9.0. ReductStore is a time series database designed for storing and managing large amounts of blob data.

To download the latest released version, please visit our Download Page.

What’s New in 1.9.0?​

This release, version 1.9.0, introduces several key improvements and features to enhance the overall performance and user experience. These updates include optimizations for disk space management, the inclusion of replication support in the Web Console, and the provision of license information in the HTTP API.

The MQTT protocol is a simple way to connect different sources of data to applications, which makes it very popular for IoT (Internet of Things) applications. Some MQTT brokers can save messages for a while, even when the MQTT client is offline. However, sometimes, you need to keep this data for a longer time. In those cases, it's a good idea to use a time series database.

There are many time series databases available, but if you need to store things like images, sensor data, or Protobuf messages, you might want to use ReductStore. This database is designed to store a lot of blob data and works well with IoT and edge computing.

ReductStore has client SDKs (Software Development Kits) for many programming languages. This means you can easily use it in your existing system. In this example, we'll be using the Python SDK from ReductStore.

Now, let's make a simple MQTT application to see how this all works.

When it comes to computer vision, data storage is a critical component. You need to be able to store images for model training, as well as the results of the processing for model validation. There are a few ways to go about this, each with its own advantages and disadvantages. In this post, we’ll take a look at three different ways to store data in computer vision applications: a file system, an S3-like object storage and ReductStore. We’ll also discuss some of the pros and cons of each option.

A Simple Computer Vision Application​

For demonstration purposes, we’ll use a simple computer vision application which is connected to a CV camera and runs on an edge device:

The camera driver captures images from the CV camera every second and forwards them to the model, which then detects objects and displays the results in the user interface.

Your images and results need to be stored for training and validation purposes. The customer may also wish to view images featuring anomalous objects. These requirements present the challenge of maintaining a history of blob or unstructured data.

In this article, we will compare two data storage solutions: ReductStore and Minio. Both offer on-premise blob storage, but they approach it differently. Minio provides traditional S3-like blob storage, while ReductStore is a time series database designed to store a history of blob data. We will focus on their application in scenarios that require storage and access to a history of unstructured data. This includes images from a computer vision camera, vibration sensor data, or binary packages common in industrial data.

Handling Historical Data​

S3-like blob storage is commonly used to store data of different formats and sizes in the cloud or internal storage. It can also accommodate historical data as a series of blobs. A simple approach is to create a folder for each data source and save objects with timestamps in their names:

bucket
 |
 |---cv_camera
        |---1666225094312397.jpeg
        |---1666225094412397.jpeg
        |---1666225094512397.jpeg

Kafka stream saved in ReductStore database

In this guide, we will explore the process of storing Kafka messages that contain unstructured data into a time series database.

Apache Kafka is a distributed streaming platform capable of handling high throughput of data, while ReductStore is a databases for unstructured data optimized for storing and querying along time.

ReductStore allows to easily setup a data sink to store blob data for applications that need precise time-based querying or a robust system optimized for edge computing that can handle quotas and retention policies.

This guide builds upon an existing tutorial which provides detailed steps for integrating a simple architecture with these systems. To get started, revisit "Easy Guide to Integrating Kafka: Practical Solutions for Managing Blob Data" if you need help setting up the initial infrastructure.

You can also find the code for this tutorial in the kafka_to_reduct demo on GitHub.

We are pleased to announce the release of the latest minor version of ReductStore, 1.8.0. ReductStore is a time series database designed for storing and managing large amounts of blob data.

To download the latest released version, please visit our Download Page.

What's New in ReductStore v1.8.0​

In this release, we've introduced a crucial feature for any database - data replication. Now, you can create a server-side task that "subscribes" to new records written to a bucket and forwards them to another bucket. This bucket can be located on the same instance or a remote one. Since all databases implement replication differently based on their specializations, let's examine how ReductStore tackles this.