ConduitIO/benchi: Benchmark any tool from the CLI

Benchi is a minimal benchmarking framework designed to help you measure the
performance of your applications and infrastructure. It leverages Docker to
create isolated environments for running benchmarks and collecting metrics.

It was developed to simplify the process of setting up and running benchmarks
for Conduit.

• Docker Integration: Define and manage your benchmarking environments using
  Docker Compose.
• Metrics Collection: Collect and export metrics in CSV format for further
  analysis.
• Custom Hooks: Define custom hooks to run commands at various stages of the
  benchmark.
• Progress Monitoring: Real-time monitoring of container statuses and
  metrics during the benchmark run.

To install Benchi, download
the latest release or
install it using Go:

go install github.com/conduitio/benchi/cmd/benchi@latest

Alternatively, you can install just the Benchi binary on a Linux OS with:

curl https://raw.githubusercontent.com/ConduitIO/benchi/main/install.sh | sh

Run benchi and point -config to a benchmark configuration file.
The repository includes an example benchmark, which can be run
using the following command:

benchi -config ./example/bench-kafka-kafka/bench.yml

Running the benchmark will store the results in a folder named after the current
date and time inside of results (e.g. results/20060102_150405). You can
adjust the output folder using the -out flag.

The output folder will contain two files:

• benchi.log: Log file containing the full output of benchi.
• aggregated-results.csv: Aggregated metric results from all collectors and
  all tests. The results are aggregated using a
  trimmed mean, where the top
  and bottom 5% of the results are removed. Benchi also disregards any 0 values
  from the start and end of the test, to accomodate for warm-up and cool-down
  periods.

The output folder will also contain one folder per benchmark run (i.e. per test
and tool combination). Each benchmark run folder will contain:

• infra_NAME.log: Log file containing the output of the infrastructure docker
  containers, split per infra service.
• tool_NAME.log: Log file containing the output of the tool docker containers.
• COLLECTOR.csv: Raw metrics collected using the corresponding
  metrics collector.

• -config: Path to the benchmark config file (required).
• -out: Path to the output folder (default: ./results/${now}*).
• -tool: Filter tool to be tested (can be provided multiple times).
• -tests: Filter test to run (can be provided multiple times).

* ${now} is replaced with the current time formatted as YYYYMMDD_HHMMSS.

Benchi creates a Docker network named benchi to connect the infrastructure
services and tools. This network is created automatically and removed after the
benchmark run. Please make sure to connect your services to this network to
ensure they can communicate with each other.

Example Docker Compose configuration:

services:
  my-service:
    networks:
      - benchi

networks:
  benchi:
    external: true

Benchi runs all Docker Compose commands using the same environment variables as
the current shell. This means that you can use environment variables to pass
values to your services.

For instance, having the following Docker Compose configuration:

services:
  my-service:
    environment:
      - MY_ENV_VAR=${MY_ENV_VAR}

You can inject the environment variable by running Benchi as follows:

MY_ENV_VAR=my-value benchi -config ./my-benchmark.yml

Benchi uses a YAML configuration file to define the benchmark in combination
with Docker Compose configurations.

Below is an example configuration:

infrastructure:
  database:
    compose: "./compose-database.yml"
  cache:
    compose: "./compose-cache.yml"

tools:
  my-app:
    compose: "./compose-my-app.yml"

metrics:
  prometheus:
    collector: "prometheus"
    settings:
      url: "http://localhost:9090/metrics"
      queries:
        - name: "http_requests_rate"
          query: "rate(request_count{endpoint=hello}[2s])"
          unit: "req/s"
          interval: "1s"

tests:
  - name: Endpoint Load
    duration: 2m
    steps:
      pre-infrastructure:
      post-infrastructure:
        - name: Setup Database
          container: database
          run: /scripts/setup-database.sh
      pre-tools:
      post-tools:
      pre-test:
      during:
        - name: Run Load Test
          container: my-app
          run: /scripts/run-load-test.sh
      post-test:
      pre-cleanup:
        - name: Cleanup
          container: my-app
          run: /scripts/cleanup.sh
      post-cleanup:

The infrastructure section defines the Docker Compose configurations for the
infrastructure services required for the benchmark. Each service is identified
by a custom name, used in logging and to correlate overridden configurations
specified in a test (see tests). The path to the docker compose file
is relative to the location of the benchmark configuration file.

Example:

infrastructure:
  name-of-infrastructure-service:
    compose: "./path/to/compose-file.yml"

The tools section defines the Docker Compose configurations for the tools
being benchmarked. Each tool is identified by a custom name, used in logging and
to correlate overridden configurations specified in a test (see tests.
The path to the docker compose file is relative to the location of the benchmark
configuration file.

Example:

tools:
  name-of-tool:
    compose: "./path/to/compose-file.yml"

The metrics section defines the metric collectors running during the
benchmark. Each metric collector has a custom name used for logging. The
collector field specifies the type of metric collector to use. The settings
field contains the configuration for the chosen collector.

Example:

metrics:
  name-of-metric-collector:
    collector: "conduit"
    settings:
      url: "http://localhost:8080/metrics"

See collectors for available collectors and their configurations.

The tests section defines the benchmarks to run. Each test has a custom name
used for logging. The duration field specifies the duration of the test. The
steps field contains the commands to run at various stages of the benchmark.

The steps field contains the following stages:

• pre-infrastructure: Commands to run before starting the infrastructure
  services.
• post-infrastructure: Commands to run after starting the infrastructure
  services.
• pre-tools: Commands to run before starting the tools.
• post-tools: Commands to run after starting the tools.
• pre-test: Commands to run before starting the test.
• during: Commands to run during the test.
• post-test: Commands to run after the test.
• pre-cleanup: Commands to run before cleaning up the test.
• post-cleanup: Commands to run after cleaning up the test.

Each hook can run its commands either in an existing container or in a temporary
container created from a specified image. The container field specifies the
name of the container to run the commands in. The image field specifies the
image to use for the temporary container. If neither container nor image is
specified, the commands will run in a temporary container using the
alpine:latest image. Note that running a custom script in a container requires
the container to contain the necessary script (hint: mount the script as a
volume).

You can optionally configure the tools field to run a hook only for certain
tools. If the field is not present or empty, the hook is applied for all tools.

Example:

tests:
  - name: My Test
    duration: 2m
    steps:
      pre-infrastructure:
      post-infrastructure:
        # This script will run after the infrastructure services have started.
        # It will run in the `database` container (presumably started by the
        # infrastructure service).
        - name: Setup Database
          container: database
          run: /scripts/setup-database.sh
      pre-tools:
      post-tools:
      pre-test:
      during:
        # This script will run for the duration of the test (2 minutes). It will
        # run in the `my-app` container (presumably started by the tool). The
        # script will receive a SIGINT signal when the test duration is over.
        - name: Run Load Test
          container: my-app
          tools:
            - my-app # Only run this hook when benchmarking my-app
          run: /scripts/run-load-test.sh
      post-test:
      pre-cleanup:
        # This script will run before infrastructure and tools containers are
        # stopped and removed. It will run in a temporary container created
        # from the `busybox:latest` image and connected to the `benchi` network.
        - name: Cleanup
          image: "busybox:latest"
          run: |
            echo "Cleaning up..."
            sleep 5
      post-cleanup:

You can also include custom infrastructure and tools configurations to
override the default configurations specified in the infrastructure and
tools sections. Note that the global configurations will still be applied, the
additional configurations are merged with the global configurations (see
merging compose files).
This can be useful to inject custom configurations for a specific test.

Example:

tests:
  - name: My Test
    duration: 2m
    infrastructure:
      name-of-infrastructure-service:
        compose: "./compose-file-infra.override.yml"
    tools:
      name-of-tool:
        compose: "./compose-file-tool.override.yml"

Collectors are used to collect metrics from various sources during the benchmark
run. The collected metrics are exported in CSV format for further analysis.

Collectors are configured in the metrics section of the benchmark
configuration file.

Supported collectors:

The Conduit metrics collector tracks the throughput for each configured
pipeline in Conduit.

Settings:

• url: URL of the Conduit metrics endpoint (needs to be reachable from the
  benchi process).
• pipelines: Array of pipelines to track.

metrics:
  my-conduit-collector:
    collector: "conduit"
    settings:
      url: "http://localhost:8080/metrics"
      pipelines:
        - "pipeline1"
        - "pipeline2"

Metrics:

• msg-rate-per-second[PIPELINE]: Messages per second per pipeline.
• msg-megabytes-in-per-second[PIPELINE]: Incoming megabytes per second per
  pipeline (measured as records read by the source connector).
• msg-megabytes-out-per-second[PIPELINE]: Outgoing megabytes per second per
  pipeline (measured as records written by the destination connector).

The Docker metrics collector tracks the container resource usage during the
benchmark run.

Settings:

• containers: Array of containers to track.

metrics:
  my-docker-collector:
    collector: "docker"
    settings:
      containers:
        - "my-app"
        - "database"
        - "cache"

Metrics:

• cpu-percentage[CONTAINER]: CPU usage in percent at a specific point in time
  per container.
• memory-usage[CONTAINER]: Memory usage in megabytes at a specific point in
  time per container.

The Kafka metrics collector tracks the throughput for each configured topic in
Apache Kafka.

The collector expects Kafka to expose a Prometheus metrics endpoint via the
Prometheus JMX exporter. To see
how to configure it, check out the
example configuration.

Settings:

• url: URL of the Kafka metrics endpoint (needs to be reachable from the
  benchi process).
• topics: Array of topics to track.

metrics:
  my-kafka-collector:
    collector: "kafka"
    settings:
      url: "http://localhost:7071/metrics"
      topics:
        - "topic1"
        - "topic2"

Metrics:

• msg-rate-in-per-second[TOPIC]: Incoming messages per second per topic.
• msg-megabytes-in-per-second[TOPIC]: Incoming megabytes per second per topic.
• msg-megabytes-out-per-second[TOPIC]: Outgoing megabytes per second per topic.

The Prometheus metrics collector continuously scrapes a metrics endpoint, stores
the metrics in memory and queries them using
PromQL.

It is expected that the query returns a matrix with a single series. The query
should be a
ranged query,
which will be evaluated between the start of the test and the end of the test.

Settings:

• url: URL of the Prometheus metrics endpoint.
• queries: Array of queries to run.
  • name: Name of the query.
  • query: PromQL query.
  • unit: Unit of the query (optional, only for displaying in the CLI).
  • interval: Resolution of the ranged query.

metrics:
  my-prometheus-collector:
    collector: "prometheus"
    settings:
      url: "http://localhost:8080/metrics"
      queries:
        - name: "http_request_success_rate"
          query: "rate(request_count{endpoint=hello,status=200}[2s])"
          unit: "req/s"
          interval: "1s"
        - name: "http_request_fail_rate"
          query: "rate(request_count{endpoint=hello,status!=200}[2s])"
          unit: "req/s"
          interval: "1s"

Metrics are user defined using the queries field. The name field is used as
the metric name in the CSV output.

• Benchi fails with the error Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.

Benchi is communicating with Docker using the default Docker socket. If you are
using Docker Desktop, you can enable it under Settings -> Advanced and check the
box for “Allow the default Docker socket to be used”.

Benchi is licensed under the Apache License, Version 2.0. See the
LICENSE file for more details.

Keep your files stored safely and securely with the SanDisk 2TB Extreme Portable SSD. With over 69,505 ratings and an impressive 4.6 out of 5 stars, this product has been purchased over 8K+ times in the past month. At only $129.99, this Amazon’s Choice product is a must-have for secure file storage.

Help keep private content private with the included password protection featuring 256-bit AES hardware encryption. Order now for just $129.99 on Amazon!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!