6 weeks with Rust
I previously wrote about my plan to rewrite ReductStore in Rust. I am happy to announce that the migration is now complete. It was a very interesting experience, and I would like to share it here.
Project Design before Migration
I wrote ReductStore in C++20, utilizing coroutines and ranges.
For the HTTP frontend, I used uWebSockets as an HTTP server and its event loop for coroutines.
The storage engine was implemented from scratch.
I used Protobuf as a JSON and binary serializer in both the HTTP frontend and the storage engine. Many of the structures were shared between the two.
I managed dependencies with Conan and used CMake as a build system.
Codebase about 20k lines with unit tests.
About Me
I have been developing in C++ and Python for about five years. Mostly, I write services for data acquisition, processing, and storage. I like OOP, design patterns, and my C++ smells Java. However, I avoid using exceptions and follow the RAII approach.
In Rust, I have only written the “Hello, World” example.
I work on the project in my spare time 10-20 hours a week.
Migration
Initially, I planned to use the cxx.rs library and rewrite the project in small steps by wrapping Rust code and integrating it into C++.
[SPOILER] I wasn’t able to handle this…
Data Reduction and Why It Is Important For Edge Computing
Before we dive into the importance of data reduction for edge computing, it is important to define both terms. Data reduction refers to the process of reducing the amount of data that needs to be transmitted or stored, while still maintaining the necessary information and level of accuracy. This can be achieved through techniques such as compression, aggregation, and filtering.
We're Moving to Rust
Initially, I chose to use C++ for the early editions of ReductStore because of my experience with the language. This allowed me to quickly create a functional time series database for binary data. However, as our platform expanded to include Windows and MacOS, I found myself struggling to manage the C++ infrastructure as the codebase grew. This made it difficult for me to focus on enhancing the product’s functionality and unique features, as I had to ensure compatibility across multiple platforms while managing numerous dependencies.
CLI Client for ReductStore v0.8.0 has been released
Hey, I’ve released version 0.8.0 of Reduct CLI, the Python package for managing data stored in ReductStore. This release includes two new features that will be particularly helpful for our public datasets hosted on ReductStore, where metadata can be used to provide important context for the data.
-
The
rcli export foldercommand now has a new option--with-metadata, which exports meta information about a record and its labels into a JSON file along with the content of the record. -
The
rcli exportcommands now accept integers for--startand--stopoptions, which means you can specify a time interval in Unix time or use it as a range of record IDs if you don’t care about time:# Export records with ID from 0 to 5 rcli export folder instance/bucket ./export_path --start 0 --stop 5
I hope these new features make it even easier to work with ReductStore and manage your data. As always, if you have any questions or feedback, please don’t hesitate to reach out!
Thanks for using ReductStore!
How to Use 'Cats' dataset with Python ReductStore SDK
The ReductStore Project hosts the free “cats” dataset, which contains about 10K photos of cats in JPEG format with eye coordinates, a month, and ears as labels. In this example, we can learn how to download the dataset from the ReductStore instance and draw the features using the OpenCV library.
Installing Dependencies
First, we need to install the ReductStore Client SDK for Python to download the photos and labels. We also need the OpenCV Python library for drawing features and Pillow to display images in the Jupyter environment:
pip install reduct-py opencv-python Pillow
Getting Data
To retrieve data we need the URL of the ReductStore instance, the bucket name, where we store our datasets and an API token with read access, so that we can connect to the database by using the Client class:
from reduct import Client, Bucket
HOST = "https://play.reduct.store"
API_TOKEN = "dataset-read-eab13e4f5f2df1e64363806443eea7ba83406ce701d49378d2f54cfbf02850f5"
BUCKET = "datasets"
client = Client(HOST, api_token=API_TOKEN)
bucket: Bucket = await client.get_bucket(BUCKET)
Streamline your edge computing workflows with ReductStore, now available on Snap
Using snap, users can now easily install and manage ReductStore on various Linux distributions.
Snap is a universal package manager developed by Canonical, the company behind Ubuntu Linux. It allows developers to package their applications and dependencies into a single package that can be installed on any Linux distribution that supports snap, without worrying about different packaging formats and dependency conflicts.
ReductStore Client SDK for C++ v1.3.0 with Labels Support
We are excited to announce the release of ReductStore Client SDK for C++ v1.3.0! This release includes support for the ReductStore API v1.3.0 with labels and content type.
CLI Client for ReductStore v0.7.0 has been released
Hey everyone,
I’m happy to announce that we have released Reduct CLI client v0.7.0 with some minor improvements and bug fixes. We started using the tool in real applications and faced some problems exporting data from a ReductStore instance when the connection is slow and we have many entries to download asynchronously.
First of all, it wasn’t very convenient to count all needed entries in the rcli export command. Now we can use
wildcards:
rcli export folder instance/bucket ./export_path --entries=sensor-*
New Release of ReductStore JavaScript SDK v1.3.0
Hey everyone,
we are pleased to announce the release of version 1.3.0 of the ReductStore SDK for JavaScript. This version supports the new features of ReductStore v1.3, including labels and content type.
Now you can write a record with MIME type and labels to ReductStore:
const client = new Client("https://play.reduct.store");
const bucket = await client.getOrCreateBucket("bucket");
const record = await bucket.beginWrite("entry-1", {
contentType:"text/plain",
labels: {type:"example"}
})
await record.write("Some text");
CLI Client for ReductStore v0.6.0 has been released
Hello everyone,
We’re excited to announce the release of version 0.6.0 of the Reduct CLI! This release brings a number of new features for working with the new ReductStore API v1.3.
Introducing the New Release of ReductStore Python SDK: v1.3.0: Labels Support and More
We are happy to announce the release of version 1.3.0 of the ReductStore SDK for Python! This release introduces several new features to help users better organize and filter their data.
One of the most notable new features is the ability to attach labels to data when writing and querying. Labels are key-value pairs that can be used to classify and categorize data. For example, you might use labels to store metadata about a record, such as its md5 sum or class. To start using labels, you need the version of the ReductStore database higher than 1.3.0.
ReductStore v1.3.0 Released
Hello everyone,
We are excited to announce the release of version 1.3.0 of the ReductStore database! This update brings a number of new features and improvements that we believe will enhance your experience with the database.
License Change
First and foremost, we’ve changed the project license, switching from AGPLv3 to MPL-2.0 which allows usage of the program as a service over the network in proprietary software. This change is made to avoid any misunderstanding in the future, and to align with our goal of encouraging contributions back to the project while allowing everyone to use it for free.
CLI Client for ReductStore v0.5.0 has been released
Hello everyone,
We’re excited to announce the release of version 0.5.0 of the Reduct CLI! This release brings a number of new features to improve data export and mirroring. Here are the highlights:
Limit parallel tasks
We added a global option --parallel to limit the number of entries to be exported or mirrored in parallel. This can be
useful when a bucket has a large number of entries and the network connection is slow. Usage example:
rcli export --parallel 5 folder server_1/bucket_1 .
Quick Start with the ReductStore JavaScript SDK
This quick start guide will walk you through the process of installing and using the ReductStore JavaScript client SDK to interact with a ReductStore instance.
Installing the SDK
To install the ReductStore SDK, you will need to have Node.js 16 or higher installed on your machine. Once Node.js is
installed, you can use the npm package manager to install the reduct-js package:
npm install reduct-js
Quick Start with the ReductStore Python SDK
This quick start guide will walk you through the process of installing and using the ReductStore Python client SDK to interact with a ReductStore instance.
Installing the SDK
To install the ReductStore SDK, you will need to have Python 3.7 or higher installed on your machine. Once Python is
installed, you can use the pip package manager to install the reduct-py package:
pip install reduct-py
Performance comparison: ReductStore vs. Minio
We often use blob storage like S3, if we need to store data of different formats and sizes somewhere in the cloud or in our internal storage. Minio is an S3 compatible storage which you can run on your private cloud, bare-metal server or even on an edge device. You can also adapt it to keep historical data as a time series of blobs. The most straightforward solution would be to create a folder for each data source and save objects with timestamps in their names:
bucket
|
|---cv_camera
|---1666225094312397.bin
|---1666225094412397.bin
|---1666225094512397.bin
ReductStore behind NGINX
I think, NGINX doesn’t need any introductions. It is one of the most widely used HTTP servers and reverse proxies. You can route your microservices or monolith application through it and make it responsible for:
- TSL encryption
- Basic HTTP authorization
- Map public URL to services or applications
- Load balancing
How to keep a history of MQTT data with Node.js
The MQTT protocol is very popular in IoT applications. It is a simple way to connect different data sources with your application by using a publish/subscribe model. Sometimes you may want to keep a history of your MQTT data to use it for model training, diagnostics or metrics. If your data sources provide different formats of data that can not be interpreted as time series of floats, ReductStore is what you need.
Let’s make a simple MQTT application to see how it works.
CLI Client for ReductStore v0.4.0 has been released
Hello everyone,
We’re excited to announce the release
of version 0.4.0 of the Reduct CLI!
This release includes a new rcli export folder command that allows you to export data from a bucket in your storage
engine to a local folder on your computer.
This can be useful if you want to make a copy of your data for backup or offline access. For more information,
see the documentation.
ReductStore Client SDK for JavaScript v1.2.0: New Features and Example Use
Hello, everyone!
ReductStore has released v1.2.0 of its
JavaScript SDK. This update includes support
for ReductStore API version 1.2 with the
new Client.me method, which allows you to retrieve information about your current API token and its permissions.
ReductStore Client SDK for Python v1.2.0: New Features and Example Use
Hello, everyone! We are excited to announce the release of the Python client SDK v1.2.0! This release includes support for the ReductStore HTTP API v1.2 and several other improvements.
ReductStore Client SDK for C++ v1.2.0: New Features and Example Use
We are excited to announce the release
of ReductStore Client SDK for C++ v1.2.0! This release
includes updated
documentation after we renamed the project from “Reduct Storage” to “ReductStore”, and supports ReductStore HTTP API
1.2.0 with the endpoint GET /api/v1/me.
ReductStore v1.2.0 Released
Hello, everyone!
We’re excited to announce the release of ReductStore v1.2.0. In this update, we’ve made some important changes to improve the way tokens and buckets are managed. Specifically, when you create a new token, the database will now check to ensure that the buckets specified in the token’s permissions actually exist. This will help prevent errors and ensure that users have the proper access to the appropriate buckets.
How to integrate ReductStore into your C++ application
This guide will help you get started with the ReductStore C++ SDK.
Requirements
The ReductStore C++ SDK is written in C++20 and uses CMake as a build system. To install it, you will need:
- C++ compiler with C++20 support (we use GCC-11.2)
- cmake 3.18 or higher
- conan 1.40 or higher (it is optional, but conan is a convenient package manager)
Currently, we have only tested the SDK on Linux AMD64, but if you need to port it to another operating system or platform, you can create an issue for assistance.
Installing
To install the ReductStore C++ SDK, follow these steps:
3 Ways To Store Data in Computer Vision Applications
When it comes to computer vision, data storage is a critical component. You need to be able to store images for model training, as well as the results of the processing for model validation. There are a few ways to go about this, each with its own advantages and disadvantages. In this post, we’ll take a look at three different ways to store data in computer vision applications: a file system, an S3-like object storage and ReductStore. We’ll also discuss some of the pros and cons of each option.
A Simple Computer Vision Application
For demonstration purposes, we’ll use a simple computer vision application which is connected to a CV camera and runs on an edge device:
The camera driver captures images from the CV camera every second and sends them to the model. The model detects something and shows the results in the user interface.
So far it isn’t too complicated, let’s see how we can deal with the data here.