Skip to main content
Version: 1.10.x

ReductStore Bucket Guide

Buckets are the primary storage unit in ReductStore. They are used to group data entries and define the storage settings for the data. This guide will cover the concepts of buckets, their settings, and operations like creating, browsing, changing settings, and removing buckets.

Concepts

A bucket is a container used for data storage and serves as a logical grouping of data. Each bucket can contain multiple entries, with each entry representing a time-series dataset composed of time-stamped records. If you're familiar with databases, think of a bucket as a database, and entries as tables.

Given the nature of time-series data, ReductStore partitions data into blocks. Each block, stored in a separate file, contains a set of records. There are limits to the block size and the number of records it can hold. When a block reaches its limit, a new block is created to store additional data. This method allows ReductStore to store and access data efficiently. You can read more about blocks and records in the How Does It Work? document.

Unlike other blob storage or file systems, ReductStore has a flat hierarchy. It doesn't use folders or directories. Instead, all data is stored in entries, which are grouped by buckets.

Bucket Settings

Each bucket has settings that determine how data is stored and accessed. These settings include:

  • Quota Type: You can choose between NONE, which means there is no quota, and FIFO (First In, First Out), where the oldest data is removed to make space for new data. The default setting is NONE.
  • Quota Size: This represents the quota size in bytes. Only enforced with FIFO Quota.
  • Max Block Size: This is the maximum allowable size of a block, in bytes. The default size is 64MB.
  • Maximal Number of Records: This is the maximum number of records that can be contained in a block. The default number is 256.

Quota Type

The quota type determines the method of quota enforcement and can be set to either NONE or FIFO.

NONE implies no quota, allowing data to be stored without any restrictions.

FIFO, short for first-in-first-out, enforces the quota by deleting the oldest block to accommodate new data once the quota is reached. ReductStore ensures that the data does not surpass the quota size, refusing to store new data if there's insufficient room.

Quota Size

The quota size defines the maximum size of the bucket in bytes. It's ignored if the quota type is set to NONE.

Max Block Size

The max block size determines the maximum size of a block in bytes. Once a block reaches this limit, a new block is created for storing additional data. A record's size isn't restricted by the block size. If a record exceeds the block size, it will be stored in the current block, and a new block will be created for the next record.

The default value is set at 64MB. Typically, there is no need to alter this unless your records exceed the default size. ReductStore pre-allocates the block size during the creation of a new block. As such, it improves performance when multiple records can be stored in a single block. We recommend to keep the block size enough to store at least 256 records.

Maximal Number of Records

The maximal number of records refers to the maximum number of records that can be stored in a block. When a block reaches this capacity, a new block is created for additional data. The default value is 256. Generally, you don't need to adjust this unless you have numerous small records and an excess of blocks, which could impact search performance.

To optimize search performance, bear in mind that record searching consists of two steps:

  • Locating the block that contains the record. This search time is O(log(n)), where n represents the number of blocks.
  • Finding the record within the block. This search time is O(m), with m representing the number of records in the block.

If there are too many blocks, the time spent searching for a block could exceed the time spent searching for the record within the block. In such cases, consider increasing the maximal number of records to reduce the total number of blocks.

Bucket Operations

Here you will find examples of how to create, browse, change settings, and remove buckets using the ReductStore SDKs, CLI client, Web Console, and REST API. Pay attention that all the examples are written for a local ReductStore instance available at http://127.0.0.1:8383 with API token my-token.

For more information on how to set up a local ReductStore instance, refer to the Getting Started guide.

Creating a Bucket

A bucket can be created using the SDKs, CLI client, Web Console, or REST API. The bucket name must be unique within the store, and a client must have full access permission if the authentication is enabled. Provisioning a bucket with environment variables is also possible. Refer to the example below:

reduct-cli alias add local -L http://localhost:8383 -t "my-token"
reduct-cli bucket create local/my-bucket --quota-type FIFO --quota-size 1GB

Browse Buckets

You might want to view the list of buckets in your store or see the details of a specific bucket. This can be done by using the SDKs, CLI client, Web Console, or REST API. For listing all buckets, a client must have a valid access token if the authorization is enabled.

reduct-cli alias add local -L http://localhost:8383 -t "my-token"
reduct-cli bucket ls local --full

# Output:
#| Name | Entries | Size | Oldest record (UTC) | Latest record (UTC) |
#|-------------------|---------|----------|--------------------------|--------------------------|
#| example-bucket | 3 | 30.7 GB | 1970-01-01T00:00:00.000Z | 1970-01-01T00:00:00.060Z |


reduct-cli bucket show local/example-bucket --full

# Output:
#Name: datasets Quota Type: NONE
#Entries: 3 Quota Size: 0 B
#Size: 30.7 GB Max. Block Size: 64.0 MB
#Oldest Record (UTC): 1970-01-01T00:00:00.000Z Max. Block Records: 256
#Latest Record (UTC): 1970-01-01T00:00:00.060Z
#
#| Name | Records | Blocks | Size | Oldest Record (UTC) | Latest Record (UTC) |
#|----------------|---------|--------|---------|--------------------------|--------------------------|
#| cats | 9993 | 40 | 2.0 GB | 1970-01-01T00:00:00.000Z | 1970-01-01T00:00:00.016Z |
#| imdb | 46641 | 459 | 28.7 GB | 1970-01-01T00:00:00.000Z | 1970-01-01T00:00:00.046Z |
#| mnist_training | 60000 | 236 | 17.5 MB | 1970-01-01T00:00:00.000Z | 1970-01-01T00:00:00.060Z |

Changing Bucket Settings

Bucket settings can be modified using the SDKs, CLI client, Web Console, or REST API. A client must have full access permission to change the settings if the authorization is enabled. The settings that can be modified include the quota type, quota size, max block size, and maximal number of records. You cannot change the bucket name.

reduct-cli alias add local -L http://localhost:8383 -t "my-token"
# Change the quota size of the bucket
reduct-cli bucket update local/example-bucket --quota-size 10GB
note

You can't change settings for provisioned buckets. You should unprovision it first.

Removing a Bucket

A bucket can be deleted using the SDKs, CLI client, Web Console, or REST API. A client must have full access permission to delete a bucket if the authorization is enabled.

reduct-cli alias add local -L http://localhost:8383 -t "my-token"
# Remove the bucket without confirmation
reduct-cli bucket rm local/bucket-to-remove -y
warning

Removing a bucket will also remove all the data stored in it. This action is irreversible.

note

You can't remove a provisioned bucket. You should unprovision it first.