Skip to content

README.md

Latest commit

 

History

History
542 lines (399 loc) · 24.9 KB

README.md

File metadata and controls

542 lines (399 loc) · 24.9 KB
Raw

SPT Engine

Overview

The SPT Engine is the high-performance Java core of the Dell Storage Performance Tool. It executes storage workloads, collects metrics, and supports distributed testing across multiple nodes. The engine is typically managed by the SPT CLI, but can also be run standalone for advanced use cases.

This directory contains the engine source, extensions (storage drivers, load patterns), and Docker bundle configuration.

Standalone and CLI-managed runs expose immutable engine provenance; see Engine Build Information for the version endpoint, local build records, combined manifests, and compatibility policy.

Key Features

  • High Performance: Leverages Virtual Threads to sustain millions of concurrent operations
  • Distributed Testing: P2P architecture enables horizontal scaling across multiple nodes
  • Extensible Architecture: Plugin-based design supports custom storage drivers and load patterns
  • Comprehensive Metrics: Detailed performance metrics including latency distributions, throughput, and concurrency
  • Multiple Storage Protocols: Supports S3, Swift, Atmos, filesystem, and custom protocols
  • Flexible Load Patterns: Create, read, update, delete operations with partial updates and composite operations
  • Scenario Support: Complex test scenarios using JavaScript, Python, or Groovy

Prerequisites

  • Java 21 or higher
  • Docker (optional, for containerized deployment)
  • Git

Quick Start

Building from Source

Standard Build Method (Recommended)

The project uses Gradle with the Gradle Wrapper for building. This ensures consistent builds across all platforms:

# Clone the repository
git clone https://github.com/dell/storage-performance-tool.git
cd storage-performance-tool/engine

# Build all components using Gradle Wrapper
./gradlew clean build -x test    # Unix/Linux/macOS
# OR
gradlew.bat clean build -x test   # Windows

# The distribution will be available at:
# bundle/build/dist/
#   ├── spt.jar       # Main application JAR
#   ├── ext/              # Extension JARs directory
#   ├── run.sh            # Unix/Linux/macOS launcher
#   └── run.bat           # Windows launcher

# For distribution, a ZIP file is also created:
# bundle/build/distributions/spt-bundle-<version>.zip

Alternative: Using Make (Optional)

For convenience, a Makefile is provided that wraps common Gradle commands:

# Build the bundle
make build

# Other useful commands
make clean      # Clean build artifacts  
make test       # Run tests
make dist       # Create ZIP distribution
make help       # Show all available targets

Note: The Makefile is a convenience wrapper and requires a Unix-like environment. For official builds, CI/CD, and cross-platform compatibility, use the Gradle Wrapper (./gradlew).

Build identity and incremental builds

Development builds capture the current UTC time when generateEngineBuildInfo executes. Unchanged builds and artifacts restored from the Gradle cache retain that timestamp. Changes to engine sources or build inputs regenerate the metadata. JAR manifests and runtime build information use the same generated snapshot.

An explicit -PsptBuildTime or SPT_BUILD_TIME takes precedence over SOURCE_DATE_EPOCH; changing an explicit timestamp invalidates the metadata. Release workflows supply the release revision and timestamp, and discover source cleanliness from Git. Tracked changes and untracked, non-ignored files prevent release packaging.

Running Your First Test

# Navigate to the distribution directory
cd bundle/build/dist

# Check version and available extensions
./run.sh --version    # Unix/Linux/macOS
# OR
run.bat --version     # Windows

# Basic S3 write test
./run.sh \
  --storage-driver-type=s3 \
  --storage-net-node-addrs=your-s3-endpoint.com \
  --storage-auth-uid=your-access-key \
  --storage-auth-secret=your-secret-key \
  --item-data-size=1MB \
  --load-op-limit-count=1000

Selecting a Storage Driver

SPT supports multiple S3 storage driver backends. Use --storage-driver-type to select one:

Value Description
s3 Default Netty/REST-based S3 driver
s3-aws AWS SDK v2 synchronous client
s3-rdma S3 driver with optional RDMA acceleration for PUT/GET; other operations use HTTP 
# Write test using the AWS SDK driver
./run.sh \
  --storage-driver-type=s3-aws \
  --storage-net-node-addrs=your-s3-endpoint.com:9020 \
  --storage-auth-uid=your-access-key \
  --storage-auth-secret=your-secret-key \
  --item-data-size=1MB \
  --load-op-limit-count=1000

# Read test using the AWS SDK driver
./run.sh \
  --storage-driver-type=s3-aws \
  --storage-net-node-addrs=your-s3-endpoint.com:9020 \
  --storage-auth-uid=your-access-key \
  --storage-auth-secret=your-secret-key \
  --load-op-type=read \
  --item-input-file=items.csv \
  --load-op-limit-count=500

AWS SDK driver workers

Every s3-aws request blocks the worker that makes it until the response arrives. When storage.driver.threads is unset (the default, 0), the driver therefore runs one worker per concurrency permit (storage.driver.limit.concurrency, the CLI's --threads), so that the configured number of requests is in flight. Before 5.16.2 it ran at most one worker per hardware thread, which capped the requests in flight at the CPU count of the node whatever the thread count asked for, and queued the rest behind busy workers. Set storage.driver.threads to pin the worker count instead; the workers are virtual threads, so a large count costs little.

AWS SDK driver timeouts

The s3-aws driver applies two timeouts to every S3 client it builds (the transfer client, the client for exact-version requests and the client for standalone DELETE), and two bounds of its own to its wait for each request:

Option Default Meaning
--storage-crt-connectionTimeoutMs 5000 Milliseconds allowed to establish a connection
--storage-crt-socketTimeoutMs 30000 Milliseconds a connection with a request in flight may move no data. After that the connection is closed and the request fails as a timeout, which is counted as a failed operation and written to the errors log. 0 turns the detection off; any other value must be at least 2000, and the monitor counts whole seconds, rounding down (2500 is two seconds)
--storage-crt-requestTimeoutMs 300000 Milliseconds a request may take from the moment the driver makes it until its response has been read in full, a GET's body included. A request that outlives this is cancelled and fails as a timeout, counted and logged like any other. It covers a request the client has lost with no connection for socketTimeoutMs to watch. Must exceed the longest request the workload makes; 0 waits without bound
--storage-crt-stopTimeoutMs 5000 Milliseconds the requests still in flight get once the step stops, before the driver cancels them; they then end as INTERRUPTED. 0 leaves them to the step's wait and the stop of the workers

socketTimeoutMs is an idle timeout, not a request deadline: a transfer that keeps moving data is never cut off, and no timer runs per request. The AWS clients retry a failed request on a new connection before they give up (the transfer client up to five times, the others up to three), so a request that stalls on every attempt fails only after all of them: with the defaults, after more than three minutes. Lower socketTimeoutMs for a tighter bound; raise it for operations that stay silent for longer than 30 seconds. requestTimeoutMs is the deadline of the whole request, retries and the reading of a GET's body included; it is the only bound that ends a request the client has lost without a connection, or a response that keeps coming but never ends. One watchdog thread per driver checks the requests in flight ten times a second; no timer is armed per request. A missing or invalid value stops the driver from starting and names the option. The first stalled connection of a run and the first timeout of any other kind are each logged once as a warning, and only the stall warning names socketTimeoutMs; further timeouts are logged at DEBUG.

When a step stops, it closes admission and then waits up to load.op.wait.limit seconds for the requests in flight. Once admission has closed, the driver gives the requests in flight stopTimeoutMs to finish and then cancels them, a GET still reading its body included, so that a lost request cannot hold the stop for the whole of that wait; a request made after that is cancelled at once, and the stop of the workers cancels whatever is left. The default is half of the engine's ten-second budget for stopping a run, which leaves the other half for the stop itself. A cancelled request ends as INTERRUPTED; a cancelled DELETE gets no result, because the server may or may not have carried it out. The first cancelled request of a run is logged as a warning, and the others at DEBUG. Set stopTimeoutMs no higher than load.op.wait.limit, and raise both for in-flight transfers that need longer than five seconds to finish at stop.

Note: Extensions are loaded from the ext/ directory next to spt.jar.

Common Command-Line Options

# Display version and available extensions
./run.sh --version
./run.sh -v

# Show help and all available options
./run.sh --help

# Run with custom configuration file
./run.sh --run-scenario=mytest.js

--version reports the same immutable Engine Build Information snapshot used by startup reporting, the /version endpoint, effective run.version, and local engine.build.json records. It does not derive the engine version from mutable defaults or command-line configuration.

Distribution Structure

After building, the distribution contains:

  • spt.jar - The main application JAR containing core SPT functionality
  • ext/ - Directory containing all extension JARs (storage drivers, load patterns)
  • run.sh/run.bat - Platform-specific launcher scripts that properly configure the classpath

The launcher scripts handle:

  • Setting up the correct classpath configuration
  • Launching SPT with extensions loaded from the adjacent ext/ directory

IDE Integration

For development in VS Code:

# Create VS Code launch configurations
./gradlew :bundle:createVSCodeLaunch
# OR
make vscode

This creates debug configurations for:

  • Running SPT with --help
  • Running S3 performance tests
  • Checking version with --version

Using Docker

# Build the Docker image locally
make docker

# Run a simple test
docker run --rm ghcr.io/dell/storage-performance-tool:latest \
  --storage-driver-type=s3 \
  --storage-net-node-addrs=your-s3-endpoint.com \
  --storage-auth-uid=your-access-key \
  --storage-auth-secret=your-secret-key

# Run using the AWS SDK driver
docker run --rm ghcr.io/dell/storage-performance-tool:latest \
  --storage-driver-type=s3-aws \
  --storage-net-node-addrs=your-s3-endpoint.com \
  --storage-auth-uid=your-access-key \
  --storage-auth-secret=your-secret-key

# Run with a built-in scenario
docker run --rm ghcr.io/dell/storage-performance-tool:latest \
  --run-scenario=/opt/spt/scenarios/js/types/weighted.js \
  --storage-driver-type=s3 \
  --storage-net-node-addrs=localhost:9000

# Run with your custom scenario
docker run --rm \
  -v $(pwd):/workspace \
  -v $(pwd)/logs:/opt/spt/log \
  ghcr.io/dell/storage-performance-tool:latest \
  --run-scenario=/workspace/my-scenario.js

# List available built-in scenarios
make docker-scenarios

# Run with a scenario using make
make docker-scenario SCENARIO=example-scenario.js

Docker Image Layout

The SPT Docker image uses /opt/spt as its working directory:

/opt/spt/
├── spt.jar          # Main application JAR
├── ext/             # Extension JARs (storage drivers, load patterns)
├── scenarios/       # Built-in example scenarios
├── entrypoint.sh    # Container entrypoint
└── log/             # Log output (created at runtime)

Docker Volumes

The SPT Docker image supports several volume mount points:

  • /opt/spt/log - Log files output directory
  • /workspace - Custom scenarios and data files

Example with volumes:

docker run --rm \
  -v $(pwd)/scenarios:/workspace \
  -v $(pwd)/logs:/opt/spt/log \
  ghcr.io/dell/storage-performance-tool:latest \
  --run-scenario=/workspace/my-test.js

Environment Variables

Variable Default Description
SPT_HOME (ephemeral temp dir) Override the engine's app-home directory. Set this to a persistent path if you need to preserve extracted config/resources across runs.
SPT_JAVA_OPTS -Xshare:off -XX:UseAVX=2 Additional JVM options appended to the default JAVA_OPTS.
JAVA_OPTS (see entrypoint.sh) Full JVM options override. Default includes ZGC, 4GB heap, NUMA, and NFS compatibility flags.

Docker Compose

A docker-compose.yml example is provided in the bundle/ directory that includes:

  • Multiple SPT service configurations
  • Volume mounts for scenarios and logs
  • Environment variable support for storage configuration
  • Example .env.example file for easy setup
# Quick setup using the provided example
cd bundle
cp .env.example .env
# Edit .env with your S3 credentials
docker-compose up spt-basic

# Or set environment variables directly
export SPT_S3_ENDPOINT=your-s3-endpoint:9000
export SPT_S3_ACCESS_KEY=your-access-key
export SPT_S3_SECRET_KEY=your-secret-key
export SPT_S3_BUCKET=your-bucket-name
make docker-compose-run

# Run custom scenario
make docker-compose-custom

# Run weighted load test
make docker-compose-weighted

S3 Configuration

To run SPT against an S3-compatible storage system, you need to configure the following environment variables:

Required Environment Variables

# S3-compatible endpoint (without http:// prefix)
export SPT_S3_ENDPOINT=<your-endpoint>:9000

# S3 access credentials
export SPT_S3_ACCESS_KEY=<your-access-key>
export SPT_S3_SECRET_KEY=<your-secret-key>

# Target bucket name
export SPT_S3_BUCKET=<your-bucket-name>

Example Configuration

For AWS S3:

export SPT_S3_ENDPOINT=s3.us-east-1.amazonaws.com
export SPT_S3_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE
export SPT_S3_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export SPT_S3_BUCKET=my-test-bucket

For MinIO or other S3-compatible storage:

export SPT_S3_ENDPOINT=minio.example.com:9000
export SPT_S3_ACCESS_KEY=minioadmin
export SPT_S3_SECRET_KEY=minioadmin
export SPT_S3_BUCKET=test-bucket

For Dell ObjectScale (ECS) with multiple S3 endpoints:

# Comma-separated list of data node addresses
export SPT_S3_ENDPOINT=http://ecs-node1:9020,http://ecs-node2:9020,http://ecs-node3:9020
export SPT_S3_ACCESS_KEY=your-objectscale-access-key
export SPT_S3_SECRET_KEY=your-objectscale-secret-key
export SPT_S3_BUCKET=perf-test

Security Note

Never commit credentials to version control. Consider using:

  • Environment variable files (add to .gitignore)
  • Secret management tools
  • CI/CD secret variables

Project Structure

engine/
├── core/
│   └── spt-base/              # Core functionality and APIs
├── extensions/
│   ├── load-steps/            # Load pattern extensions
│   │   ├── pipeline/          # Sequential operations
│   │   └── weighted/          # Weighted random operations
│   └── storage-drivers/       # Storage protocol implementations
│       ├── primitives/        # Base driver implementations
│       ├── protocols/         # Protocol-specific drivers
│       └── implementations/   # Storage-specific drivers (S3, S3-RDMA, etc.)
├── bundle/                    # Docker bundle and distribution packaging
└── docs/                      # Additional documentation

Documentation

Health & Readiness Endpoints

  • GET /health — liveness probe; returns 200 with {status:"ok", scope:"node", role, node_id[, cluster_id]}.
  • GET /ready — readiness probe; returns 503 while starting, 200 once services are ready. Body includes {ready:bool, status:"starting|ready", ...}.

These endpoints are useful for orchestrators and for spt to simplify startup logic. See full details in the API docs above.

Supported Storage Types

  • S3: Amazon S3 and S3-compatible storage (MinIO, Ceph, etc.)
  • Swift: OpenStack Swift
  • Atmos: Dell Atmos
  • Filesystem: Local or network filesystems
  • Custom: Extensible framework for custom storage drivers

S3-RDMA Acceleration

The S3-RDMA storage driver provides an optional RDMA (Remote Direct Memory Access) data path for S3 PUT and GET operations. When enabled, object data bypasses the kernel networking stack entirely — the storage server reads from and writes to the client's memory directly over the RDMA fabric.

How It Works

SPT acts as an RDMA target: it registers a memory buffer, creates a DC (Dynamically Connected) Target endpoint, and sends a standard S3 HTTP request with an additional x-amz-rdma-token header. The storage server uses the token to perform the actual RDMA data transfer.

S3 Operation RDMA Action
PUT (write) Server performs RDMA READ from client memory
GET (read) Server performs RDMA WRITE to client memory
DELETE None; uses the inherited Netty S3 HTTP DeleteObject/DeleteObjects path

Because the server initiates the data transfer, the HTTP request body is empty and Content-Length is set to 0. The actual object size is encoded in the RDMA token. This is critical for compatibility with HTTP frameworks (e.g. Jetty) that block until the declared body bytes arrive.

Standalone DELETE requests are indivisible non-data operations. They keep their complete target list and use the inherited HTTP implementation even when the RDMA transport is available: one target issues DeleteObject, while multiple same-bucket targets issue one non-quiet DeleteObjects request. Current-key and exact-version behavior therefore matches the Netty S3 driver. DELETE qualification uses deterministic unavailable-transport and fake-available-transport contract tests; RDMA-capable hardware is not required because DELETE transfers no payload over RDMA.

For SPT-owned seeded benchmarks, optional residual cleanup is a later ordinary DELETE step, not part of the standalone measured request assembler. It starts only after the measured step has finished post-verification and committed its immutable items.csv, and consumes those canonical current-key or exact-version identities without an item output. The scenario finalizer reconciles terminal success and failure counts to that residual, reports one bounded cleanup result, suppresses cleanup failure from the benchmark verdict, and rethrows the original measured failure when one exists. This preserves the legacy single-object extension path and keeps cleanup time and errors out of standalone request metrics.

RDMA Token Format

The x-amz-rdma-token header carries all information the server needs to perform the transfer:

addr:size:rkey:lid:dctn:g:gid
Field Hex Digits Description
addr 16 Virtual address of the registered buffer
size 8 Buffer size in bytes
rkey 8 Remote key for RDMA memory access
lid 4 Local ID (0 for RoCE, used in InfiniBand)
dctn 6 DC Target Number (connection endpoint)
g 1 Global addressing flag (1 for RoCE)
gid 32 Global ID for RoCE routing

Example token for a 1 MiB buffer:

00007fffc3200000:00100000:00004d16:0000:00133f:1:fe80000000000000000000000012ab34

Architecture

The driver is implemented in three layers:

  • S3RdmaStorageDriver (Java) — extends S3StorageDriver, overrides submit() to route operations above a configurable size threshold to the RDMA path. Falls back to super.submit() for small objects or when RDMA is unavailable.
  • RdmaTransport (Java) — JNI bridge that manages buffer registration, token generation, and native lifecycle.
  • libspt_rdma.so (rdma_native.c, ~675 lines of C) — native implementation using libibverbs, libmlx5, and librdmacm. Handles device initialization, DC Target creation, memory registration, and token formatting.

PUT/GET RDMA Requirements

  • Linux only (RDMA stack depends on libibverbs / kernel verbs)
  • RDMA-capable NIC (NVIDIA/Mellanox ConnectX-4 or newer)
  • An S3-compatible storage target with RDMA support (e.g. Dell ObjectScale / ECS)
  • rdma-core system packages (libibverbs, librdmacm, libmlx5)
  • Docker device passthrough (--device /dev/infiniband) for containerized deployments

On systems without RDMA hardware, the driver detects this at initialization and fails by default. Set storage.rdma.fallback to true to fall back to HTTP instead.

Engine Configuration

RDMA settings are passed through the scenario YAML under the storage.rdma namespace:

Key Default Description
storage.rdma.thresholdBytes 1048576 Minimum object size (bytes) for RDMA transfer. Set to 0 to use RDMA for all sizes.
storage.rdma.timeoutMs 30000 RDMA operation timeout in milliseconds
storage.rdma.device auto RDMA device name or auto for auto-detection
storage.rdma.fallback false Fall back to HTTP if RDMA initialization fails

Data Generation Controls

The engine supports two knobs for shaping generated object data, useful for benchmarking storage compression and deduplication efficiency.

Configuration

Settings live under item.data in the config schema (config-schema.yaml) and defaults (defaults.yaml):

Key Type Default Description
item.data.input.compressibility double 0.0 Target compressibility percentage (0-100). Each 4KB chunk is split into a pseudo-random portion and a zero-filled portion according to this percentage. 0 = fully random, 100 = fully compressible.
item.data.dedupable boolean true When true, the engine uses its standard ring-buffer data which may deduplicate across objects. When false, the engine stamps every 4KB of the output stream with a 16-byte header (8-byte deterministic object ID + 8-byte absolute stream offset) to defeat inline deduplication.

Compressibility Model

Data generation uses a micro-compressibility model: within each 4KB chunk, the first portion is filled with XOR-shift pseudo-random bytes (incompressible) and the remainder is zero-filled (compressible). At 75% compressibility, each 4KB chunk contains ~1KB random + ~3KB zeros. This forces storage-side compression to evaluate every block rather than only detecting large all-zero regions.

Anti-Dedupe Stamping

When dedupable=false, a 16-byte stamp is written at every 4KB boundary in the output stream:

  • Bytes 0-7: deterministic 64-bit identifier derived from the object name
  • Bytes 8-15: absolute byte offset in the object stream

Stamps are applied in-transit without mutating the shared cached ring-buffer layers. All transfer paths (Netty non-SSL, Netty SSL/TLS, AWS SDK, RDMA buffer fill) apply identical stamping for consistency.

The stamp overhead is ~0.39% (16 bytes per 4096 bytes), which slightly reduces the effective compressibility of the data.

Constraints

  • dedupable=false is incompatible with file-based data input (item.data.input.file). If both are configured, the engine rejects the configuration at startup.
  • When dedupable=false, data integrity verification is disabled in incompatible paths and a warning is logged. Full stamp-aware verification is planned for a future release.

Contributing

We welcome contributions! Please see our Contributing Guide for details on how to get started.

License

See the LICENSE file for license rights and limitations.

Support

Direct dispatch

Completion-driven direct dispatch is enabled by default for drivers that declare support for it. Drivers without that capability retain dispatcher scheduling. To disable it when launching an engine container or the SPT CLI, set:

export SPT_JAVA_OPTS="${SPT_JAVA_OPTS:+$SPT_JAVA_OPTS }-Dspt.dispatch.direct=false"

For a direct Java launch, place -Dspt.dispatch.direct=false before -jar. The JVM system property is read when each driver is constructed; restart the engine to change it. An absent property enables direct dispatch; explicit true enables it and false disables it. This is a JVM setting, not a scenario configuration flag. Avoid duplicate definitions in other JVM option sources.