ReductStore Settings
ReductStore can be configured using environment variables. The following sections describe the available settings and how to use them.
Remote Backend Settings
ReductStore supports using a remote object storage as a backend to store data. You can configure the remote backend using the following environment variables:
| Name | Default | Description |
|---|---|---|
RS_REMOTE_BACKEND_TYPE | File system | If it is set to S3, the storage uses an S3-compatible object storage as a backend |
RS_REMOTE_ENDPOINT | URL of the S3-compatible object storage endpoint. It's ignored for the AWS S3 service. | |
RS_REMOTE_REGION | Region of the S3-compatible object storage. It's ignored for on-premises storage like MinIO. | |
RS_REMOTE_ACCESS_KEY | Access key for the S3-compatible object storage. | |
RS_REMOTE_SECRET_KEY | Secret key for the S3-compatible object storage. | |
RS_REMOTE_BUCKET | Name of the bucket to store data. The bucket must be created before starting ReductStore. | |
RS_REMOTE_CACHE_PATH | Path to a folder where the storage caches data before uploading it to the remote backend. | |
RS_REMOTE_CACHE_SIZE | 1GB | Maximum size of the cache folder. When the limit is reached, the oldest data is removed. |
RS_REMOTE_SYNC_INTERVAL | 0.1, 60 for S3 | Synchronization interval with the remote backend in seconds. |
RS_REMOTE_DEFAULT_STORAGE_CLASS | STANDARD | Default storage class for data blocks. See the list of supported S3 storage classes. |
By default, ReductStore uses the local file system to store data. To use a remote backend, set the RS_REMOTE_BACKEND_TYPE variable to S3 and configure the other variables accordingly.
When the remote backend is enabled, ReductStore caches data in the folder specified by the RS_REMOTE_CACHE_PATH variable, thereby reducing the number of requests made to the remote backend.
The RS_DATA_PATH variable is ignored when the remote backend is enabled.
For end-to-end deployment examples (standalone, active-passive, and read-only replicas) using S3, see the S3 integration guide.
Supported S3 Storage Classes
A user can specify default storage class for data blocks stored in the S3-compatible object storage using the RS_REMOTE_DEFAULT_STORAGE_CLASS environment variable.
The following storage classes are supported:
| Name | Description |
|---|---|
STANDARD | Standard storage class. Suitable for frequently accessed data. |
STANDARD_IA | Standard-Infrequent Access storage class. Suitable for data that is infrequently accessed but requires rapid access when needed. |
INTELLIGENT_TIERING | Intelligent-Tiering storage class. Automatically moves data to the most cost-effective access tier based on usage patterns. |
ONEZONE_IA | One Zone-Infrequent Access storage class. Suitable for data that is infrequently accessed and stored in a single availability zone. |
EXPRESS_ONEZONE | Express-One Zone storage class. Suitable for data that requires low latency access and is stored in a single availability zone. |
GLACIER_IR | Glacier Instant Retrieval storage class. Suitable for data that is rarely accessed and requires immediate access when needed. |
GLACIER | Glacier storage class. Suitable for data that is rarely accessed and can be retrieved within a few hours. |
DEEP_ARCHIVE | Deep Archive storage class. Suitable for data that is rarely accessed and can be retrieved within 12 hours. |
OUTPOSTS | Outposts storage class. Suitable for data that needs to be stored on AWS Outposts. |
For more information about S3 storage classes, see the Amazon S3 Storage Classes documentation.
I/O Settings
In addition to general settings, you can configure I/O settings to optimize communication over HTTP with the storage engine. The following table describes the available environment variables:
| Name | Default | Description |
|---|---|---|
RS_IO_BATCH_MAX_SIZE | 8MB | Maximum size of a batch of records sent to the client. |
RS_IO_BATCH_MAX_RECORDS | 85 | Maximum number of records in a batch sent and received from the client. |
RS_IO_BATCH_MAX_METADATA_SIZE | 8KB | Maximum size of metadata in a batch of records sent and received from the client. |
RS_IO_BATCH_TIMEOUT | 5s | Maximum time for a batch of records to be prepared and sent to the client. If the batch is not full, it will be sent after the timeout. |
RS_IO_BATCH_RECORD_TIMEOUT | 1s | Maximum time to wait for a record to be added to a batch. If the record is not added, the unfinished batch will be sent to the client. |
RS_IO_OPERATION_TIMEOUT | 60s | Maximum time for an I/O operation (e.g., read or write). If the operation takes longer, it will be aborted. |
Most of the I/O settings are related to batching and specify the size of the batch, the number of records in the batch, and the timeout for preparing and sending the batch on the server side.
However, if a client is using the HTTP/1.1 protocol, the RS_IO_BATCH_MAX_METADATA_SIZE and RS_IO_BATCH_MAX_RECORDS settings are used to limit the size of the headers in the HTTP request from the client.
This means that the client can't send more records in a single batch than specified by the RS_IO_BATCH_MAX_RECORDS and the size of the headers in the request can't exceed the value of the RS_IO_BATCH_MAX_METADATA_SIZE setting.
Read more about batching in the HTTP API Reference.
Lock File Settings
ReductStore uses a lock file to prevent multiple instances from accessing the same data folder. The lock file is created in the data folder specified by the RS_DATA_PATH variable, using the same storage backend as the data. You can configure the lock file settings using the following environment variables:
| Name | Default | Description |
|---|---|---|
RS_LOCK_FILE_POLLING_INTERVAL | 10s | Interval between attempts to acquire the lock file or rewrite the acquired lock file to update its timestamp. |
RS_LOCK_FILE_ROLE | primary | Role of the instance. It can be primary or secondary. The primary instance acquires the lock file immediately. The secondary instance waits for the primary to acquire the file first. |
RS_LOCK_FILE_TTL | 30s | Time-to-live for the lock file. If the lock file is not updated within the TTL, it is considered stale and can be overwritten by another instance. |
RS_LOCK_FILE_TIMEOUT | 60s | Timeout for acquiring the lock file. If the lock file is not acquired within the timeout, the storage will exit. When set to 0, the storage will wait indefinitely. |
RS_LOCK_FILE_FAILURE_ACTION | abort | Action to take if the lock file can't be acquired. It can be abort or proceed. If set to abort, the storage will exit. If set to proceed, the storage will overwrite the lock file and continue. |
The lock file is enabled when the RS_INSTANCE_ROLE variable is set to primary or secondary. If the variable is not set, the lock file mechanism is disabled.
Storage Engine Settings
A user can configure storage engine settings to change behavior of data storage and retrieval. The following table describes the available environment variables:
| Name | Default | Description |
|---|---|---|
RS_ENGINE_COMPACTION_INTERVAL | 60s | Interval between compaction WALs into blocks and synchronization of the storage state to the backend. |
RS_ENGINE_REPLICA_UPDATE_INTERVAL | 60s | Interval between updates of bucket, entry lists and indexes from the backend in the read-only mode. |
RS_ENGINE_ENABLE_INTEGRITY_CHECKS | true | If set to false, the storage engine skips integrity checks at startup to improve performance. |
Replication Settings
ReductStore supports data replication between different instances. You can configure replication settings using the following environment variables:
| Name | Default | Description |
|---|---|---|
RS_REPLICATION_TIMEOUT | 5s | Timeout for attempts to reconnect to the target server in seconds. |
RS_REPLICATION_LOG_SIZE | 1000000 | Maximum number of pending records in the replication log. The oldest records are overwritten when the limit is reached. |