MayankPratap/Samchika: A fast and light-weight multithreaded file processing library for Java.

Samchika (meaning File in Sanskrit) is a re-usable, easy-to-use, and extremely fast file processing library for the Java language.

It is built with a strong focus on multithreading to handle CPU-intensive file processing tasks in parallel, enabling high performance even with massive files.

• 🔁 Fully multithreaded – optimized for parallel file processing.
• 🧩 Simple API – just plug in your file path and logic.
• 📊 Optional runtime stats – time taken, memory used, thread-level info.
• 🧪 Ideal for processing and analyzing large text files (e.g. logs, datasets).
• 🌍 Open-source friendly – contributions are welcome!

Samchika excels in several scenarios where multithreaded file processing provides significant advantages:

• Log Analysis & Processing
• ETL (Extract, Transform, Load) Operations
• Large Text Corpus Processing
• Batch Report Generation
• Data Transformation Pipelines
• Real-time Data Processing

See the examples directory for detailed implementations of these use cases.

java// Transform a large CSV file with optimal performance

SmartFileProcessor.builder()
    .inputPath("large_dataset.csv")
    .outputPath("transformed_dataset.csv")
    .batchSize(10000)
    .lineProcessor(line -> line.toUpperCase())
    .displayStats(true)
    .build()
    .execute();

    
        jitpack.io
        https://jitpack.io
    



    
        com.github.mayankpratap
        samchika
        1.0.0
    

repositories {
    maven { url 'https://jitpack.io' }
}

dependencies {
    implementation 'com.github.mayankpratap:samchika:1.0.0'
}

import com.samchika.SmartFileProcessor;

public static void main(String[] args) {
    SmartFileProcessor processor = SmartFileProcessor.builder()
        .inputPath("input.txt")         // Path to the file to be processed
        .outputPath("output.txt")       // Path to write the output
        .lineProcessor(Main::processLine) // Your business logic per line
        .displayStats(true)             // Optional: display runtime stats
        .build();

    processor.execute();
}

Benchmarked against naïve BufferedReader-based implementations on files of various sizes:

✅ 200 MB

✅ 1 GB

✅ 5 GB

✅ 16 GB

Significant performance improvements were observed, especially in multi-core systems ( More than 70% performance gain )

The gain in time saved improves in comparison to naive code, as we increase the size of input file. Also for huge performance gain in time, the memory used for even large files ( 16GB ) is manageable ( ~ 800 MB ).

This library is licensed under the MIT License, which means you can freely use, modify, and distribute it, even in commercial applications. All we ask is that you include the original copyright notice and license text in any copy of the library or substantial portion of it.

This project was inspired by:

1. Shubham Maurya ( https://github.com/complex1 ) , a dear friend, who published a JavaScript library – which sparked the motivation to do something similar in Java.

2. A LinkedIn post discussing the challenges of processing large text files – which gave me the idea to solve it with an elegant API and fast multithreaded architecture.

Keep your files stored safely and securely with the SanDisk 2TB Extreme Portable SSD. With over 69,505 ratings and an impressive 4.6 out of 5 stars, this product has been purchased over 8K+ times in the past month. At only $129.99, this Amazon’s Choice product is a must-have for secure file storage.

Help keep private content private with the included password protection featuring 256-bit AES hardware encryption. Order now for just $129.99 on Amazon!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!