• About TC
  • Affiliate Disclaimer
  • Privacy Policy
  • TOS
  • Contact
Thursday, June 5, 2025
Techcratic
  • TC
  • AI
    Artificial Intelligence

    Top 5 Alternative Data Career Paths and How to Learn Them for Free

    Artificial Intelligence

    Implementing Machine Learning Pipelines with Apache Spark

    Artificial Intelligence

    Learn Power BI for Free This Week

    Artificial Intelligence

    Build GraphRAG applications using Amazon Bedrock Knowledge Bases

    Artificial Intelligence

    How to Use Deep Research Like a Pro

    Artificial Intelligence

    World-Consistent Video Diffusion With Explicit 3D Modeling

    Artificial Intelligence

    Deploy Amazon SageMaker Projects with Terraform Cloud

    Artificial Intelligence

    Data Science ETL Pipelines with DuckDB

    Artificial Intelligence

    New Amazon Bedrock Data Automation capabilities streamline video and audio analysis

  • Crypto
    Bitcoin Craters to $100K as $837M in Crypto Positions Vanish

    Bitcoin Craters to $100K as $837M in Crypto Positions Vanish

    SHA256 Strikes Back: Bitcoin’s Mining Algorithm Reclaims the Throne in 2025

    SHA256 Strikes Back: Bitcoin’s Mining Algorithm Reclaims the Throne in 2025

    NYSE Welcomes Circle in Major Crypto Public Listing

    NYSE Welcomes Circle in Major Crypto Public Listing

    ECB Slashes Rates Again—While the Fed Stands Frozen

    ECB Slashes Rates Again—While the Fed Stands Frozen

    Circle’s IPO Surpasses Expectations, Raising $1.05 Billion at $31 a Share

    Circle’s IPO Surpasses Expectations, Raising $1.05 Billion at $31 a Share

    Warning: Europe Might Irrevocably Make Bitcoin Illegal

    Warning: Europe Might Irrevocably Make Bitcoin Illegal

    Crypto to “Become Part of All Sectors” Under Trump: Kevin O’Leary

    “Bitcoin Might go to Infinite Against Dollar,” Says Tim Draper

    Elon Musk Inches Closer to Bitcoin Maximalism Amid US Debt Alarm

    Elon Musk Inches Closer to Bitcoin Maximalism Amid US Debt Alarm

    Grayscale: Bitcoin Demand Explodes as $5T Deficit Projection Fuels Fiat Doubt

    Grayscale: Bitcoin Demand Explodes as $5T Deficit Projection Fuels Fiat Doubt

  • Cybersecurity
    Cybersecurity

    Popular Chrome Extensions Leak API Keys, User Data via HTTP and Hardcoded Credentials

    Cybersecurity

    Critical Cisco ISE Auth Bypass Flaw Impacts Cloud Deployments on AWS, Azure, and OCI

    Cybersecurity

    Why Traditional DLP Solutions Fail in the Browser Era

    Cybersecurity

    HPE Issues Security Patch for StoreOnce Bug Allowing Remote Authentication Bypass

    Cybersecurity

    Critical 10-Year-Old Roundcube Webmail Bug Allows Authenticated Users Run Malicious Code

    Cybersecurity

    Android Trojan Crocodilus Now Active in 8 Countries, Targeting Banks and Crypto Wallets

    Cybersecurity

    Microsoft and CrowdStrike Launch Shared Threat Actor Glossary to Cut Attribution Confusion

    Cybersecurity

    New Chrome Zero-Day Actively Exploited; Google Issues Emergency Out-of-Band Patch

    Cybersecurity

    Qualcomm Fixes 3 Zero-Days Used in Targeted Android Attacks via Adreno GPU

  • Deals
    WD_BLACK 1TB SN7100 NVMe Internal Gaming SSD Solid State Drive – Gen4 PCIe, M.2 2280, Up…

    WD_BLACK 1TB SN7100 NVMe Internal Gaming SSD Solid State Drive – Gen4 PCIe, M.2 2280, Up…

    MagicFiber Microfiber Cleaning Cloth, 6 Pack – Premium Cloth for Glasses, Lens, Screens…

    MagicFiber Microfiber Cleaning Cloth, 6 Pack – Premium Cloth for Glasses, Lens, Screens…

    HyperX Alloy Origins 60 – Mechanical Gaming Keyboard, Ultra Compact 60% Form Factor,…

    HyperX Alloy Origins 60 – Mechanical Gaming Keyboard, Ultra Compact 60% Form Factor,…

    Apple iPad Pro 9.7in 256GB Gold WiFi + 4G Cellular ( )(Renewed)

    Apple iPad Pro 9.7in 256GB Gold WiFi + 4G Cellular ( )(Renewed)

    SEDETA 55″ White Corner Computer Desk with Storage, L Shaped Desk with Fabric Drawers…

    SEDETA 55″ White Corner Computer Desk with Storage, L Shaped Desk with Fabric Drawers…

    HUANUO Single Monitor Mount, 13 to 32 Inch Gas Monitor Arm, Adjustable Stand, Vesa Mount…

    HUANUO Single Monitor Mount, 13 to 32 Inch Gas Monitor Arm, Adjustable Stand, Vesa Mount…

    GMKtec Mini PC Intel Core i9 13900HK(Turbo 5.4GHz) 32GB DDR5 2TB SSD NVMe Mini Computers…

    GMKtec Mini PC Intel Core i9 13900HK(Turbo 5.4GHz) 32GB DDR5 2TB SSD NVMe Mini Computers…

    Fantasy Flight Games Arkham Horror The Card Game The Innsmouth Conspiracy Investigator…

    Fantasy Flight Games Arkham Horror The Card Game The Innsmouth Conspiracy Investigator…

    Goliath Go Go UFO Game | Alien Adventure Game with Non-Stop Slinky Action | 2-6 Players,…

    Goliath Go Go UFO Game | Alien Adventure Game with Non-Stop Slinky Action | 2-6 Players,…

  • Gaming
    Gungrave G.O.R.E. 03 – Mutant – Stage 3 – Infected Sewers | Easy | No Commentary | All Trophies

    Gungrave G.O.R.E. 03 – Mutant – Stage 3 – Infected Sewers | Easy | No Commentary | All Trophies

    EA Play: EA Sports FC 25 Is Coming to Game Pass Ultimate with EA Play

    EA Play: EA Sports FC 25 Is Coming to Game Pass Ultimate with EA Play

    Assassin's Creed Odyssey – Before You Buy

    Assassin's Creed Odyssey – Before You Buy

    SUPER MARIO MOVIE FINAL TRAILER REACTION | THE SUPER MARIO BROS. MOVIE DIRECT – 3.9.2023

    SUPER MARIO MOVIE FINAL TRAILER REACTION | THE SUPER MARIO BROS. MOVIE DIRECT – 3.9.2023

    Hogwarts Legacy Review – What Makes It SO GOOD?

    Hogwarts Legacy Review – What Makes It SO GOOD?

    Why Epic Games give Games for Free? – Explained in Tamil (& How?)

    Why Epic Games give Games for Free? – Explained in Tamil (& How?)

    The Dark Truth About Ubisoft

    The Dark Truth About Ubisoft

    RTX Blackwell GPUs Accelerate Video Editing

    RTX Blackwell GPUs Accelerate Video Editing

    Mr Meat – Full Episode | Gameplay Walkthrough Part 1 | Android Gameplay HD

    Mr Meat – Full Episode | Gameplay Walkthrough Part 1 | Android Gameplay HD

  • Tesla
    YONZEE Foldable Desk Food Tray (Carbon Edition) for Tesla Model Y Model 3 – Upgraded…

    YONZEE Foldable Desk Food Tray (Carbon Edition) for Tesla Model Y Model 3 – Upgraded…

    3PCS Car Micro Squeegee Curves Slot Tint Tool Set, Auto Vinyl Wrap Tool Kit, 3 in 1…

    3PCS Car Micro Squeegee Curves Slot Tint Tool Set, Auto Vinyl Wrap Tool Kit, 3 in 1…

    for Tesla Model Y 20 Inch Wheel Rim Protector Hubcaps Induction Wheels Rim Cover Tire…

    for Tesla Model Y 20 Inch Wheel Rim Protector Hubcaps Induction Wheels Rim Cover Tire…

    SIM&NAT RC Lipo Battery Charger Adapter Connector Splitter Cable, 8 in 1 Octopus Convert…

    SIM&NAT RC Lipo Battery Charger Adapter Connector Splitter Cable, 8 in 1 Octopus Convert…

    Tesla admits it would ‘suffer financial harm’ if its self-driving crash data becomes public

    Spigen Cybercase Adapter Case Designed for Tesla SAE J1772 Charging Adapter with Magnet…

    Spigen Cybercase Adapter Case Designed for Tesla SAE J1772 Charging Adapter with Magnet…

    Motrobe Tesla Cybertruck Center Console Organizer Tray [TPE+Anti-Scratch] Armrest…

    Motrobe Tesla Cybertruck Center Console Organizer Tray [TPE+Anti-Scratch] Armrest…

    Waterproof Car Covers Fit for 2024 Tesla Cybertruck, 6 Layers Upgrade 210T Windproof…

    Waterproof Car Covers Fit for 2024 Tesla Cybertruck, 6 Layers Upgrade 210T Windproof…

    Center Console Armrest Cover Compatible with Tesla Cybertruck 2024 2025, Upgrade…

    Center Console Armrest Cover Compatible with Tesla Cybertruck 2024 2025, Upgrade…

  • UFO
    The UFO 'cookbook': How the American government investigates the unexplainable

    The UFO 'cookbook': How the American government investigates the unexplainable

    Alien Research

    Alien Research

    Ancient Aliens: Ancient Alien DNA (Season 10) | History

    Ancient Aliens: Ancient Alien DNA (Season 10) | History

    SOJOS Retro Trendy Polarized Aviator Sunglasses for Women Men, Large Square Bold Pilot Shades SJ2385

    SOJOS Retro Trendy Polarized Aviator Sunglasses for Women Men, Large Square Bold Pilot Shades SJ2385

    Lily Nova | Astrophotographer, Contactee, & CE5 practitioner

    Lily Nova | Astrophotographer, Contactee, & CE5 practitioner

    I Hate Models – Intergalactic Emotional Breakdown [ARTS040]

    I Hate Models – Intergalactic Emotional Breakdown [ARTS040]

    Journey to End of Universe! #space #spaceexploration

    Journey to End of Universe! #space #spaceexploration

    Craziest Conspiracy Theories

    Craziest Conspiracy Theories

    UFO, Unidentified Flying Object, Round Sphere, or Weather Balloon

    UFO, Unidentified Flying Object, Round Sphere, or Weather Balloon

No Result
View All Result
  • TC
  • AI
    Artificial Intelligence

    Top 5 Alternative Data Career Paths and How to Learn Them for Free

    Artificial Intelligence

    Implementing Machine Learning Pipelines with Apache Spark

    Artificial Intelligence

    Learn Power BI for Free This Week

    Artificial Intelligence

    Build GraphRAG applications using Amazon Bedrock Knowledge Bases

    Artificial Intelligence

    How to Use Deep Research Like a Pro

    Artificial Intelligence

    World-Consistent Video Diffusion With Explicit 3D Modeling

    Artificial Intelligence

    Deploy Amazon SageMaker Projects with Terraform Cloud

    Artificial Intelligence

    Data Science ETL Pipelines with DuckDB

    Artificial Intelligence

    New Amazon Bedrock Data Automation capabilities streamline video and audio analysis

  • Crypto
    Bitcoin Craters to $100K as $837M in Crypto Positions Vanish

    Bitcoin Craters to $100K as $837M in Crypto Positions Vanish

    SHA256 Strikes Back: Bitcoin’s Mining Algorithm Reclaims the Throne in 2025

    SHA256 Strikes Back: Bitcoin’s Mining Algorithm Reclaims the Throne in 2025

    NYSE Welcomes Circle in Major Crypto Public Listing

    NYSE Welcomes Circle in Major Crypto Public Listing

    ECB Slashes Rates Again—While the Fed Stands Frozen

    ECB Slashes Rates Again—While the Fed Stands Frozen

    Circle’s IPO Surpasses Expectations, Raising $1.05 Billion at $31 a Share

    Circle’s IPO Surpasses Expectations, Raising $1.05 Billion at $31 a Share

    Warning: Europe Might Irrevocably Make Bitcoin Illegal

    Warning: Europe Might Irrevocably Make Bitcoin Illegal

    Crypto to “Become Part of All Sectors” Under Trump: Kevin O’Leary

    “Bitcoin Might go to Infinite Against Dollar,” Says Tim Draper

    Elon Musk Inches Closer to Bitcoin Maximalism Amid US Debt Alarm

    Elon Musk Inches Closer to Bitcoin Maximalism Amid US Debt Alarm

    Grayscale: Bitcoin Demand Explodes as $5T Deficit Projection Fuels Fiat Doubt

    Grayscale: Bitcoin Demand Explodes as $5T Deficit Projection Fuels Fiat Doubt

  • Cybersecurity
    Cybersecurity

    Popular Chrome Extensions Leak API Keys, User Data via HTTP and Hardcoded Credentials

    Cybersecurity

    Critical Cisco ISE Auth Bypass Flaw Impacts Cloud Deployments on AWS, Azure, and OCI

    Cybersecurity

    Why Traditional DLP Solutions Fail in the Browser Era

    Cybersecurity

    HPE Issues Security Patch for StoreOnce Bug Allowing Remote Authentication Bypass

    Cybersecurity

    Critical 10-Year-Old Roundcube Webmail Bug Allows Authenticated Users Run Malicious Code

    Cybersecurity

    Android Trojan Crocodilus Now Active in 8 Countries, Targeting Banks and Crypto Wallets

    Cybersecurity

    Microsoft and CrowdStrike Launch Shared Threat Actor Glossary to Cut Attribution Confusion

    Cybersecurity

    New Chrome Zero-Day Actively Exploited; Google Issues Emergency Out-of-Band Patch

    Cybersecurity

    Qualcomm Fixes 3 Zero-Days Used in Targeted Android Attacks via Adreno GPU

  • Deals
    WD_BLACK 1TB SN7100 NVMe Internal Gaming SSD Solid State Drive – Gen4 PCIe, M.2 2280, Up…

    WD_BLACK 1TB SN7100 NVMe Internal Gaming SSD Solid State Drive – Gen4 PCIe, M.2 2280, Up…

    MagicFiber Microfiber Cleaning Cloth, 6 Pack – Premium Cloth for Glasses, Lens, Screens…

    MagicFiber Microfiber Cleaning Cloth, 6 Pack – Premium Cloth for Glasses, Lens, Screens…

    HyperX Alloy Origins 60 – Mechanical Gaming Keyboard, Ultra Compact 60% Form Factor,…

    HyperX Alloy Origins 60 – Mechanical Gaming Keyboard, Ultra Compact 60% Form Factor,…

    Apple iPad Pro 9.7in 256GB Gold WiFi + 4G Cellular ( )(Renewed)

    Apple iPad Pro 9.7in 256GB Gold WiFi + 4G Cellular ( )(Renewed)

    SEDETA 55″ White Corner Computer Desk with Storage, L Shaped Desk with Fabric Drawers…

    SEDETA 55″ White Corner Computer Desk with Storage, L Shaped Desk with Fabric Drawers…

    HUANUO Single Monitor Mount, 13 to 32 Inch Gas Monitor Arm, Adjustable Stand, Vesa Mount…

    HUANUO Single Monitor Mount, 13 to 32 Inch Gas Monitor Arm, Adjustable Stand, Vesa Mount…

    GMKtec Mini PC Intel Core i9 13900HK(Turbo 5.4GHz) 32GB DDR5 2TB SSD NVMe Mini Computers…

    GMKtec Mini PC Intel Core i9 13900HK(Turbo 5.4GHz) 32GB DDR5 2TB SSD NVMe Mini Computers…

    Fantasy Flight Games Arkham Horror The Card Game The Innsmouth Conspiracy Investigator…

    Fantasy Flight Games Arkham Horror The Card Game The Innsmouth Conspiracy Investigator…

    Goliath Go Go UFO Game | Alien Adventure Game with Non-Stop Slinky Action | 2-6 Players,…

    Goliath Go Go UFO Game | Alien Adventure Game with Non-Stop Slinky Action | 2-6 Players,…

  • Gaming
    Gungrave G.O.R.E. 03 – Mutant – Stage 3 – Infected Sewers | Easy | No Commentary | All Trophies

    Gungrave G.O.R.E. 03 – Mutant – Stage 3 – Infected Sewers | Easy | No Commentary | All Trophies

    EA Play: EA Sports FC 25 Is Coming to Game Pass Ultimate with EA Play

    EA Play: EA Sports FC 25 Is Coming to Game Pass Ultimate with EA Play

    Assassin's Creed Odyssey – Before You Buy

    Assassin's Creed Odyssey – Before You Buy

    SUPER MARIO MOVIE FINAL TRAILER REACTION | THE SUPER MARIO BROS. MOVIE DIRECT – 3.9.2023

    SUPER MARIO MOVIE FINAL TRAILER REACTION | THE SUPER MARIO BROS. MOVIE DIRECT – 3.9.2023

    Hogwarts Legacy Review – What Makes It SO GOOD?

    Hogwarts Legacy Review – What Makes It SO GOOD?

    Why Epic Games give Games for Free? – Explained in Tamil (& How?)

    Why Epic Games give Games for Free? – Explained in Tamil (& How?)

    The Dark Truth About Ubisoft

    The Dark Truth About Ubisoft

    RTX Blackwell GPUs Accelerate Video Editing

    RTX Blackwell GPUs Accelerate Video Editing

    Mr Meat – Full Episode | Gameplay Walkthrough Part 1 | Android Gameplay HD

    Mr Meat – Full Episode | Gameplay Walkthrough Part 1 | Android Gameplay HD

  • Tesla
    YONZEE Foldable Desk Food Tray (Carbon Edition) for Tesla Model Y Model 3 – Upgraded…

    YONZEE Foldable Desk Food Tray (Carbon Edition) for Tesla Model Y Model 3 – Upgraded…

    3PCS Car Micro Squeegee Curves Slot Tint Tool Set, Auto Vinyl Wrap Tool Kit, 3 in 1…

    3PCS Car Micro Squeegee Curves Slot Tint Tool Set, Auto Vinyl Wrap Tool Kit, 3 in 1…

    for Tesla Model Y 20 Inch Wheel Rim Protector Hubcaps Induction Wheels Rim Cover Tire…

    for Tesla Model Y 20 Inch Wheel Rim Protector Hubcaps Induction Wheels Rim Cover Tire…

    SIM&NAT RC Lipo Battery Charger Adapter Connector Splitter Cable, 8 in 1 Octopus Convert…

    SIM&NAT RC Lipo Battery Charger Adapter Connector Splitter Cable, 8 in 1 Octopus Convert…

    Tesla admits it would ‘suffer financial harm’ if its self-driving crash data becomes public

    Spigen Cybercase Adapter Case Designed for Tesla SAE J1772 Charging Adapter with Magnet…

    Spigen Cybercase Adapter Case Designed for Tesla SAE J1772 Charging Adapter with Magnet…

    Motrobe Tesla Cybertruck Center Console Organizer Tray [TPE+Anti-Scratch] Armrest…

    Motrobe Tesla Cybertruck Center Console Organizer Tray [TPE+Anti-Scratch] Armrest…

    Waterproof Car Covers Fit for 2024 Tesla Cybertruck, 6 Layers Upgrade 210T Windproof…

    Waterproof Car Covers Fit for 2024 Tesla Cybertruck, 6 Layers Upgrade 210T Windproof…

    Center Console Armrest Cover Compatible with Tesla Cybertruck 2024 2025, Upgrade…

    Center Console Armrest Cover Compatible with Tesla Cybertruck 2024 2025, Upgrade…

  • UFO
    The UFO 'cookbook': How the American government investigates the unexplainable

    The UFO 'cookbook': How the American government investigates the unexplainable

    Alien Research

    Alien Research

    Ancient Aliens: Ancient Alien DNA (Season 10) | History

    Ancient Aliens: Ancient Alien DNA (Season 10) | History

    SOJOS Retro Trendy Polarized Aviator Sunglasses for Women Men, Large Square Bold Pilot Shades SJ2385

    SOJOS Retro Trendy Polarized Aviator Sunglasses for Women Men, Large Square Bold Pilot Shades SJ2385

    Lily Nova | Astrophotographer, Contactee, & CE5 practitioner

    Lily Nova | Astrophotographer, Contactee, & CE5 practitioner

    I Hate Models – Intergalactic Emotional Breakdown [ARTS040]

    I Hate Models – Intergalactic Emotional Breakdown [ARTS040]

    Journey to End of Universe! #space #spaceexploration

    Journey to End of Universe! #space #spaceexploration

    Craziest Conspiracy Theories

    Craziest Conspiracy Theories

    UFO, Unidentified Flying Object, Round Sphere, or Weather Balloon

    UFO, Unidentified Flying Object, Round Sphere, or Weather Balloon

No Result
View All Result
Techcratic
No Result
View All Result
Home Hacker News

glassflow/clickhouse-etl: Real-time deduplication and temporal joins for streaming data

Hacker News by Hacker News
May 11, 2025
in Hacker News
Reading Time: 20 mins read
127 3
A A
0
Share on FacebookShare on XShare on LinkedIn

2025-05-11 09:33:00
github.com


GlassFlow Logo

Join our weekly office hours every Wednesday 15:00-18:00 CET

Join Next Office Hour

Slack
Email Support
Twitter

GlassFlow for ClickHouse Streaming ETL is a real-time stream processor designed to simplify data pipeline creation and management between Kafka and ClickHouse. It provides a powerful, user-friendly interface for building and managing real-time data pipelines with built-in support for deduplication and temporal joins.

Built specifically for data engineers, GlassFlow handles late-arriving events, ensures exactly-once correctness, and scales with high-throughput data. It delivers accurate, low-latency results from streaming data without compromising simplicity or performance. The tool’s intuitive web interface makes it easy to configure and monitor pipelines, while its robust architecture ensures reliable data processing.

  • Streaming Deduplication:

    • Real-time deduplication of Kafka streams before ingestion into ClickHouse
    • Configurable time windows up to 7 days for deduplication
    • Simple configuration of deduplication keys and time windows
    • One-click setup for deduplicated data pipelines
    • Prevents duplicate data from reaching ClickHouse
  • Temporal Stream Joins:

    • Join two Kafka streams in real-time
    • Configurable time windows up to 7 days for stream joins
    • Configure join keys and time windows through the UI
    • Simplified join setup process
    • Produce joined streams ready for ClickHouse ingestion
  • Built-in Kafka Connector:

    • Powered by NATS-Kafka Bridge
    • Automatic data extraction from Kafka topics
    • Seamless integration with Kafka clusters
    • No manual data pulling required
    • Supports multiple Kafka topics and partitions
    • Native support for JSON data types
  • Optimized ClickHouse Sink:

    • Native ClickHouse connection for maximum performance
    • Configurable batch sizes for efficient data ingestion
    • Adjustable wait times for optimal throughput
    • Built-in retry mechanisms
    • Automatic schema detection and management
    • Full support for JSON data types in ClickHouse
  • User-Friendly Interface: Web-based UI for pipeline configuration and management

  • Local Development: Includes demo setup with local Kafka and ClickHouse instances

  • Docker Support: Easy deployment using Docker and docker-compose

  • Self-Hosted: Open-source solution that can be self-hosted in your infrastructure

GlassFlow ClickHouse ETL Introduction
  • Docker and Docker Compose
  • Git
  1. Clone the repository:
git clone https://github.com/glassflow/clickhouse-etl.git
cd clickhouse-etl
  1. Start the services using Docker Compose:
  1. Access the web interface at http://localhost:8080 to configure your pipeline:
  • Connection Setup:
    • Configure Kafka source connection parameters
    • Configure ClickHouse sink connection parameters
  • Pipeline Configuration:
    • Select Kafka input streams
    • Define deduplication key and time window
    • For joins: specify join keys and time window for each stream
    • Configure ClickHouse table settings
  • Pipeline Management:
    • Start/stop pipelines
    • Check pipeline logs
  1. Access the web interface at http://localhost:8080
  2. Configure your source (Kafka) and sink (ClickHouse) connections
  3. Define your pipeline transformations:
    • Set up deduplication rules
    • Configure temporal joins between streams
    • Define data transformations
  4. Start the pipeline and monitor its progress

The project includes a comprehensive demo setup in the demos folder that provides a complete local testing environment. This setup includes:

  • Local Kafka instance with Zookeeper
  • Local ClickHouse instance
  • Sample data generation
  • Example pipeline configurations
  • Step-by-step instructions for running the demo

For detailed instructions on setting up and running the local testing environment, see the Demo README file.

The project consists of several key components:

  • GlassFlow API: Core ETL engine written in Go
  • Web UI: User interface for pipeline management
  • NATS: Message broker for internal communication
  • Kafka Bridge: Connector for Kafka integration using the NATS-Kafka Bridge

For information about third-party components and their licenses, please see our NOTICE file.

The pipeline configuration is defined in a JSON file that specifies the source, sink, and any transformations. While the web interface automatically generates this configuration, understanding its structure can be helpful for advanced users.

The pipeline configuration is a JSON object that defines how data flows from Kafka topics to ClickHouse tables. It consists of three main components:

  1. Source Configuration (Kafka)
  2. Sink Configuration (ClickHouse)
  3. Join Configuration (Optional)

Field Type Required Description
pipeline_id string Yes Unique identifier for the pipeline. Must be non-empty.
source object Yes Configuration for the Kafka source. See Source Configuration.
sink object Yes Configuration for the ClickHouse sink. See Sink Configuration.
join object No Configuration for joining multiple Kafka topics. See Join Configuration.

The source configuration defines how to connect to and consume from Kafka topics.

Field Type Required Description
type string Yes “kafka” is the only supported source
provider string No Kafka provider, e.g. “aiven”
topics array Yes List of Kafka topics to consume from. See Topic Configuration.
connection_params object Yes Kafka connection parameters. See Connection Parameters.

Field Type Required Description
brokers array Yes List of Kafka broker addresses (e.g., [“localhost:9092”]).
protocol string Yes Security protocol for Kafka connection (e.g., “SASL_SSL”).
mechanism string Yes Authentication mechanism (e.g., “SCRAM-SHA-256”).
username string Yes Username for Kafka authentication.
password string Yes Password for Kafka authentication.
root_ca string No Cert. file for Kafka authentication.

Each topic in the topics array has the following configuration:

Field Type Required Description
name string Yes Name of the Kafka topic.
consumer_group_initial_offset string Yes Initial offset for the consumer group (“earliest” or “newest”).
schema object Yes Event schema definition. See Schema Configuration.
deduplication object Yes Deduplication settings. See Deduplication Configuration.

Field Type Required Description
type string Yes Schema type (Currently only “json” is supported).
fields array Yes List of field definitions. See Field Configuration.

Field Type Required Description
name string Yes Field name.
type string Yes Field type (e.g., “String”, “Integer”).

Deduplication Configuration

Field Type Required Description
enabled boolean Yes Whether deduplication is enabled.
id_field string Yes Field name used for message deduplication.
id_field_type string Yes Type of the ID field (e.g., “string”).
time_window string Yes Time window for deduplication (e.g., “1h” for one hour).

The sink configuration defines how to connect to and write to ClickHouse.

Field Type Required Description
type string Yes Must be “clickhouse”.
host string Yes ClickHouse server hostname.
port integer Yes ClickHouse server port.
database string Yes ClickHouse database name.
username string Yes ClickHouse username.
password string Yes ClickHouse password.
table string Yes Target table name.
secure boolean No Whether to use secure connection. Defaults to false.
max_batch_size integer No Maximum number of records to batch before writing. Defaults to 1000.
max_delay_time string No Maximum delay time before the messages are flushed into the sink. Defaults to “10m”.
table_mapping array Yes List of field to column mappings. See Table Mapping Configuration.

Table Mapping Configuration

Each mapping in the table_mapping array has the following configuration:

Field Type Required Description
source_id string Yes Name of the source topic.
field_name string Yes Source field name.
column_name string Yes Target column name.
column_type string Yes Target column type.

The join configuration defines how to join data from multiple Kafka topics.

Field Type Required Description
enabled boolean Yes Whether joining is enabled.
type string Yes Join type (e.g., “temporal”).
sources array Yes List of sources to join. See Join Source Configuration.

Join Source Configuration

Each source in the sources array has the following configuration:

Field Type Required Description
source_id string Yes Name of the Kafka topic to join.
join_key string Yes Field name used for joining records.
time_window string Yes Time window for joining records (e.g., “1h” for one hour).
orientation string Yes Join orientation (“left” or “right”).

{
  "pipeline_id": "kafka-to-clickhouse-pipeline",
  "source": {
    "type": "kafka",
    "provider": "aiven",
    "connection_params": {
      "brokers": [
        "kafka-broker-0:9092",
        "kafka-broker-1:9092"
      ],
      "protocol": "SASL_SSL",
      "mechanism": "SCRAM-SHA-256",
      "username": "",
      "password": "",
      "root_ca": ""
    },
    "topics": [
      {
        "consumer_group_initial_offset": "earliest",
        "name": "user_logins",
        "schema": {
          "type": "json",
          "fields": [
            {
              "name": "session_id",
              "type": "string"
            },
            {
              "name": "user_id",
              "type": "string"
            },
            {
              "name": "timestamp",
              "type": "datetime"
            }
          ]
        },
        "deduplication": {
          "enabled": true,
          "id_field": "session_id",
          "id_field_type": "string",
          "time_window": "12h"
        }
      },
      {
        "consumer_group_initial_offset": "earliest",
        "name": "orders",
        "schema": {
          "type": "json",
          "fields": [
            {
              "name": "user_id",
              "type": "string"
            },
            {
              "name": "order_id",
              "type": "string"
            },
            {
              "name": "timestamp",
              "type": "datetime"
            }
          ]
        },
        "deduplication": {
          "enabled": true,
          "id_field": "order_id",
          "id_field_type": "string",
          "time_window": "12h"
        }
      }
    ]
  },
  "join": {
    "enabled": false,
    "type": "temporal",
    "sources": [
      {
        "source_id": "user_logins",
        "join_key": "user_id",
        "time_window": "1h",
        "orientation": "left"
      },
      {
        "source_id": "orders",
        "join_key": "user_id",
        "time_window": "1h",
        "orientation": "right"
      }
    ]
  },
  "sink": {
    "type": "clickhouse",
    "provider": "aiven",
    "host": "",
    "port": "12753",
    "database": "default",
    "username": "",
    "password": "",
    "secure": true,
    "max_batch_size": 1,
    "max_delay_time": "10m",
    "table": "user_orders",
    "table_mapping": [
      {
        "source_id": "user_logins",
        "field_name": "session_id",
        "column_name": "session_id",
        "column_type": "UUID"
      },
      {
        "source_id": "user_logins",
        "field_name": "user_id",
        "column_name": "user_id",
        "column_type": "UUID"
      },
      {
        "source_id": "orders",
        "field_name": "order_id",
        "column_name": "order_id",
        "column_type": "UUID"
      },
      {
        "source_id": "user_logins",
        "field_name": "timestamp",
        "column_name": "login_at",
        "column_type": "DataTime"
      },
      {
        "source_id": "orders",
        "field_name": "timestamp",
        "column_name": "order_placed_at",
        "column_type": "DateTime"
      }
    ]
  }
}

💡 Note: The web interface automatically generates this configuration based on user input, so manual editing is not required.

We welcome contributions! Please see our Contributing Guidelines for details.

This project is licensed under the Apache License 2.0.



Source Link


Keep your files stored safely and securely with the SanDisk 2TB Extreme Portable SSD. With over 69,505 ratings and an impressive 4.6 out of 5 stars, this product has been purchased over 8K+ times in the past month. At only $129.99, this Amazon’s Choice product is a must-have for secure file storage.

Help keep private content private with the included password protection featuring 256-bit AES hardware encryption. Order now for just $129.99 on Amazon!


Start your free Amazon Prime trial
today and unlock unlimited streaming and more!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!

BITCOIN

Bitcoin Logo

Bitcoin QR Code

bc1qlszw7elx2qahjwvaryh0tkgg8y68enw30gpvge

Scan the QR code with your crypto wallet app

DOGECOIN

Dogecoin Logo

Dogecoin QR Code

D64GwvvYQxFXYyan3oQCrmWfidf6T3JpBA

Scan the QR code with your crypto wallet app

ETHEREUM

Ethereum Logo

Ethereum QR Code

0xe9BC980DF3d985730dA827996B43E4A62CCBAA7a

Scan the QR code with your crypto wallet app

Please read the Privacy and Security Disclaimer on how Techcratic handles your support.

Disclaimer: As an Amazon Associate, Techcratic may earn from qualifying purchases.

Tags: Hacker News
Share161Tweet101Share28
Previous Post

THE CALLISTO PROTOCOL (Game Review)

Next Post

Puppy Adventures ~ Fairy Lake ~ Museum ~ Sauble Beach ~ Southampton Ontario

Hacker News

Hacker News

Stay updated with Hacker News, where technology meets entrepreneurial spirit. Get the latest on tech trends, startup news, and discussions from the tech community. Read the latest updates here at Techcratic.

Related Posts

Tokasaurus: An LLM Inference Engine for High-Throughput Workloads
Hacker News

Tokasaurus: An LLM Inference Engine for High-Throughput Workloads

June 5, 2025
1.3k
Engineer Fixes and Re-Installs Old Payphones, Provides Free Calls to the Public
Hacker News

Engineer Fixes and Re-Installs Old Payphones, Provides Free Calls to the Public

June 5, 2025
1.3k
Doubling Down on Open Source
Hacker News

Doubling Down on Open Source

June 4, 2025
1.3k
How We Reduced the Impact of Zombie Clients
Hacker News

How We Reduced the Impact of Zombie Clients

June 4, 2025
1.3k
When memory was measured in kilobytes: The art of efficient vision
Hacker News

When memory was measured in kilobytes: The art of efficient vision

June 4, 2025
1.3k
The History of R2E and the Micral
Hacker News

The History of R2E and the Micral

June 4, 2025
1.3k
Load More
Next Post
Puppy Adventures ~ Fairy Lake ~ Museum ~ Sauble Beach ~ Southampton Ontario

Puppy Adventures ~ Fairy Lake ~ Museum ~ Sauble Beach ~ Southampton Ontario

Everything You Need To Know About The Private Business Jet

Everything You Need To Know About The Private Business Jet

Who Makes RCA TVs And Where Are They Built?

Who Makes RCA TVs And Where Are They Built?

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Your Tech Resources

  • 30 Second Tech ™
  • AI
  • App Zone ™
  • Apple
  • Ars Technica
  • CNET
  • ComputerWorld
  • Crypto News
  • Cybersecurity
  • Endgadget
  • Fossbytes
  • Gaming
  • GeekWire
  • Gizmodo
  • Google News
  • Hacker News
  • Harvard Tech
  • I Like Cats ™
  • I Like Dogs ™
  • LifeHacker
  • MacRumors
  • Macworld
  • Mashable
  • Microsoft
  • MIT Tech
  • PC World
  • Photofocus
  • Physics
  • Random Tech
  • Retro Rewind ™
  • Robot Report
  • SiliconANGLE
  • SlashGear
  • Smartphone
  • StackSocial
  • Tech Art
  • Tech Careers
  • Tech Deals
  • Techcratic ™
  • TechCrunch
  • Techdirt
  • TechRepublic
  • Techs Got To Eat ™
  • TechSpot
  • Tesla
  • The Verge
  • TNW
  • Trusted Reviews
  • UFO
  • VentureBeat
  • Visual Capitalist
  • Weird Stuff
  • Wired
  • ZDNet

Tech News

  • 30 Second Tech ™
  • AI
  • AnandTech
  • Apple Insider
  • Ars Technica
  • CNET
  • ComputerWorld
  • Crypto News
  • Cybersecurity
  • Endgadget
  • ExtremeTech
  • Fossbytes
  • Gaming
  • GeekWire
  • Gizmodo

Tech News

  • Harvard Tech
  • MacRumors
  • Macworld
  • Mashable
  • Microsoft
  • MIT Tech
  • Physics
  • PC World
  • Random Tech
  • Retro Rewind ™
  • SiliconANGLE
  • SlashGear
  • Smartphone
  • StackSocial
  • Tech Careers

Tech News​

  • Tech Art
  • TechCrunch
  • Techdirt
  • TechRepublic
  • Techs Got To Eat ™
  • TechSpot
  • Tesla
  • The Verge
  • TNW
  • Trusted Reviews
  • UFO
  • VentureBeat
  • Visual Capitalist
  • Weird Stuff
  • Wired
  • ZDNet

Site Links

  • About Techcratic
  • Affiliate Disclaimer
  • Affiliate Link Policy
  • Contact Techcratic
  • Dealors Discount Store
  • Privacy and Security Disclaimer
  • Privacy Policy
  • RSS Feed
  • Site Map
  • Support Techcratic
  • Techcratic
  • Tech Deals
  • TOS
  • 𝕏
Click For A Secret Deal

Techcratic – Your All In One Tech Hub © 2020 – 2025
All Rights Reserved
∞

No Result
View All Result
  • Home
  • Apple
  • Gaming
  • Microsoft
  • AnandTech