• About TC
  • Affiliate Disclaimer
  • Privacy Policy
  • TOS
  • Contact
Monday, June 9, 2025
Techcratic
  • TC
  • AI
    Artificial Intelligence

    10 Awesome OCR Models for 2025

    Artificial Intelligence

    5 Error Handling Patterns in Python (Beyond Try-Except)

    Artificial Intelligence

    Top 5 Alternative Data Career Paths and How to Learn Them for Free

    Artificial Intelligence

    Implementing Machine Learning Pipelines with Apache Spark

    Artificial Intelligence

    Learn Power BI for Free This Week

    Artificial Intelligence

    Build GraphRAG applications using Amazon Bedrock Knowledge Bases

    Artificial Intelligence

    How to Use Deep Research Like a Pro

    Artificial Intelligence

    World-Consistent Video Diffusion With Explicit 3D Modeling

    Artificial Intelligence

    Deploy Amazon SageMaker Projects with Terraform Cloud

  • Crypto
    ETF Weekly Flows: $129 Million Outflow for Bitcoin and $281 Million Inflow for Ether

    ETF Weekly Flows: $129 Million Outflow for Bitcoin and $281 Million Inflow for Ether

    DOGE Gets Distilled: Heritage Unleashes Dogecoin-Themed Bourbon

    DOGE Gets Distilled: Heritage Unleashes Dogecoin-Themed Bourbon

    Crypto ETFs centralize what was meant to be decentralized.

    Crypto ETFs centralize what was meant to be decentralized.

    Crypto Lost $1.64 Billion to Hackers in Q1 2025

    Why Is Crypto Down Today? – June 9, 2025

    The Blockchain Group Unveils $343 Million Capital Program to Boost Bitcoin Treasury Strategy

    The Blockchain Group Unveils $343 Million Capital Program to Boost Bitcoin Treasury Strategy

    Bitcoin Bull Cycle is Over: CryptoQuant CEO

    CEX Volumes Hit 2020 Lows as Market Shifts to HODL Mode

    Central African Republic to Launch Tokenized Land Sales on Solana

    Central African Republic to Launch Tokenized Land Sales on Solana

    XRP Added to Nasdaq Crypto US Settlement Price Index

    XRP Added to Nasdaq Crypto US Settlement Price Index

    Cetus Protocol Relaunches After $220M Hack, Restores Liquidity

    Cetus Protocol Relaunches After $220M Hack, Restores Liquidity

  • Cybersecurity
    Cybersecurity

    Malicious Browser Extensions Infect 722 Users Across Latin America Since Early 2025

    Cybersecurity

    Empower Users and Protect Against GenAI Data Loss

    Cybersecurity

    Popular Chrome Extensions Leak API Keys, User Data via HTTP and Hardcoded Credentials

    Cybersecurity

    Critical Cisco ISE Auth Bypass Flaw Impacts Cloud Deployments on AWS, Azure, and OCI

    Cybersecurity

    Why Traditional DLP Solutions Fail in the Browser Era

    Cybersecurity

    HPE Issues Security Patch for StoreOnce Bug Allowing Remote Authentication Bypass

    Cybersecurity

    Critical 10-Year-Old Roundcube Webmail Bug Allows Authenticated Users Run Malicious Code

    Cybersecurity

    Android Trojan Crocodilus Now Active in 8 Countries, Targeting Banks and Crypto Wallets

    Cybersecurity

    Microsoft and CrowdStrike Launch Shared Threat Actor Glossary to Cut Attribution Confusion

  • Deals
    GTRACING Gaming Chair with Footrest Speakers Video Game Chair Bluetooth Music Heavy Duty…

    GTRACING Gaming Chair with Footrest Speakers Video Game Chair Bluetooth Music Heavy Duty…

    RoboCop Rogue City (PS5)

    RoboCop Rogue City (PS5)

    My Universe: Puppies and Kittens – PlayStation 4

    My Universe: Puppies and Kittens – PlayStation 4

    Disney’s Little Mermaid: Ariel’s Undersea Adventure – Nintendo DS (Renewed)

    Disney’s Little Mermaid: Ariel’s Undersea Adventure – Nintendo DS (Renewed)

    Family Game Pack 2001- PlayStation (Renewed)

    Family Game Pack 2001- PlayStation (Renewed)

    StarTech.com Cisco GLC-T Compatible SFP Module – 1000BASE-T – SFP to RJ45 Cat6/Cat5e -…

    StarTech.com Cisco GLC-T Compatible SFP Module – 1000BASE-T – SFP to RJ45 Cat6/Cat5e -…

    5-in-1 Memory Card Reader, USB OTG Adapter & SD Card Reader for i-Phone/i-Pad, USB C and…

    5-in-1 Memory Card Reader, USB OTG Adapter & SD Card Reader for i-Phone/i-Pad, USB C and…

    Samsung Galaxy A23 5G SM-A236U Factory Unlocked 64GB Black (Renewed)

    Samsung Galaxy A23 5G SM-A236U Factory Unlocked 64GB Black (Renewed)

    ORICO USB3.0 External 4 Bay Hard Drive Enclosure for 2.5 3.5 inch SSD HDD with…

    ORICO USB3.0 External 4 Bay Hard Drive Enclosure for 2.5 3.5 inch SSD HDD with…

  • Gaming
    The Callisto Protocol showed me what makes a GOOD GAME (Raptor Review)

    The Callisto Protocol showed me what makes a GOOD GAME (Raptor Review)

    REDRAGON S101 GAMING KEYBOARD

    Sony’s fast InZone M10S QD-OLED gaming monitor is over $300 off

    starship troopers: extermination, 2

    starship troopers: extermination, 2

    As Dusk Falls – Part 5 | XBOX SERIES X |

    As Dusk Falls – Part 5 | XBOX SERIES X |

    DIABLO 4: VESSEL OF HATRED SUCKS! (Review after 100%)

    DIABLO 4: VESSEL OF HATRED SUCKS! (Review after 100%)

    How to pay your taxes in Dune: Awakening

    How to pay your taxes in Dune: Awakening

    Sonic Frontiers  (Nintendo Switch Review)

    Sonic Frontiers (Nintendo Switch Review)

    Review on Gotham Knights – 50+ Hours on Hard Mode

    Review on Gotham Knights – 50+ Hours on Hard Mode

    Gungrave G.O.R.E. 12 – Labyrinth – Stage 11 – Hong Kong Back Alleys Area 1 | Easy | No Comm | Plat

    Gungrave G.O.R.E. 12 – Labyrinth – Stage 11 – Hong Kong Back Alleys Area 1 | Easy | No Comm | Plat

  • Tesla
    4 PCS LED Reverse Lights, 4014 45SMD 6500K 800LM High Bright Brake Light Turn Signal…

    4 PCS LED Reverse Lights, 4014 45SMD 6500K 800LM High Bright Brake Light Turn Signal…

    4 Pack Trailer Ball Cover, 2.36In x 2.24In x 1.97In Waterproof Dustproof Towing Hitch…

    4 Pack Trailer Ball Cover, 2.36In x 2.24In x 1.97In Waterproof Dustproof Towing Hitch…

    ClimaTex Heavy Duty Car, Truck, Van, and SUV Automotive Floor Mat for Floor Protection,…

    ClimaTex Heavy Duty Car, Truck, Van, and SUV Automotive Floor Mat for Floor Protection,…

    2 Pcs Tow Hook Covers Compatible with Tesla Cybertruck Accessories 2024 2025 (Red)

    2 Pcs Tow Hook Covers Compatible with Tesla Cybertruck Accessories 2024 2025 (Red)

    MAXDOM Under Seat Storage Fit for 2024+ Tesla Cybertruck Rear Underseat Organizer Box…

    MAXDOM Under Seat Storage Fit for 2024+ Tesla Cybertruck Rear Underseat Organizer Box…

    Car USB Hub Charger for Tesla Model Y 2021-2024 and Model 3 2021-2023,Fast…

    Car USB Hub Charger for Tesla Model Y 2021-2024 and Model 3 2021-2023,Fast…

    CAR GUYS Tire Shine Spray | High Gloss & Satin Finish | Non-Greasy, UV Protection,…

    CAR GUYS Tire Shine Spray | High Gloss & Satin Finish | Non-Greasy, UV Protection,…

    7PCS Center Console Organizer Tray for 2024 2025 Tesla Model 3 Accessories, Armrest…

    7PCS Center Console Organizer Tray for 2024 2025 Tesla Model 3 Accessories, Armrest…

    Spigen Door Striker and Hinge Cover Designed for Tesla Model 3 & Y [Compatible with…

    Spigen Door Striker and Hinge Cover Designed for Tesla Model 3 & Y [Compatible with…

  • UFO
    NASA UAP Researchers Share Shocking UFO Evidence!

    NASA UAP Researchers Share Shocking UFO Evidence!

    UFOs Over Phoenix: Confessions of a 911 Operator [DVD]

    UFOs Over Phoenix: Confessions of a 911 Operator [DVD]

    Have Aliens Visited Earth? | COLOSSAL MYSTERIES

    Have Aliens Visited Earth? | COLOSSAL MYSTERIES

    MINDBLOWING Encounters Unraveling the Secrets of Higher Dimensions

    MINDBLOWING Encounters Unraveling the Secrets of Higher Dimensions

    Roswell: The After-Action Report

    Roswell: The After-Action Report

    Alien UFO theories: AskReddit #ufo #alien #extraterrestrial #askreddit #reddit #creepystories #scary

    Alien UFO theories: AskReddit #ufo #alien #extraterrestrial #askreddit #reddit #creepystories #scary

    Resident Alien: Season One [DVD]

    Resident Alien: Season One [DVD]

    Close Up Zoom Moon 1000x beatiful view #SHORTVIDEO

    Close Up Zoom Moon 1000x beatiful view #SHORTVIDEO

    Black Triangle UFO: The Truth Behind the TR-3B Mystery

    Black Triangle UFO: The Truth Behind the TR-3B Mystery

No Result
View All Result
  • TC
  • AI
    Artificial Intelligence

    10 Awesome OCR Models for 2025

    Artificial Intelligence

    5 Error Handling Patterns in Python (Beyond Try-Except)

    Artificial Intelligence

    Top 5 Alternative Data Career Paths and How to Learn Them for Free

    Artificial Intelligence

    Implementing Machine Learning Pipelines with Apache Spark

    Artificial Intelligence

    Learn Power BI for Free This Week

    Artificial Intelligence

    Build GraphRAG applications using Amazon Bedrock Knowledge Bases

    Artificial Intelligence

    How to Use Deep Research Like a Pro

    Artificial Intelligence

    World-Consistent Video Diffusion With Explicit 3D Modeling

    Artificial Intelligence

    Deploy Amazon SageMaker Projects with Terraform Cloud

  • Crypto
    ETF Weekly Flows: $129 Million Outflow for Bitcoin and $281 Million Inflow for Ether

    ETF Weekly Flows: $129 Million Outflow for Bitcoin and $281 Million Inflow for Ether

    DOGE Gets Distilled: Heritage Unleashes Dogecoin-Themed Bourbon

    DOGE Gets Distilled: Heritage Unleashes Dogecoin-Themed Bourbon

    Crypto ETFs centralize what was meant to be decentralized.

    Crypto ETFs centralize what was meant to be decentralized.

    Crypto Lost $1.64 Billion to Hackers in Q1 2025

    Why Is Crypto Down Today? – June 9, 2025

    The Blockchain Group Unveils $343 Million Capital Program to Boost Bitcoin Treasury Strategy

    The Blockchain Group Unveils $343 Million Capital Program to Boost Bitcoin Treasury Strategy

    Bitcoin Bull Cycle is Over: CryptoQuant CEO

    CEX Volumes Hit 2020 Lows as Market Shifts to HODL Mode

    Central African Republic to Launch Tokenized Land Sales on Solana

    Central African Republic to Launch Tokenized Land Sales on Solana

    XRP Added to Nasdaq Crypto US Settlement Price Index

    XRP Added to Nasdaq Crypto US Settlement Price Index

    Cetus Protocol Relaunches After $220M Hack, Restores Liquidity

    Cetus Protocol Relaunches After $220M Hack, Restores Liquidity

  • Cybersecurity
    Cybersecurity

    Malicious Browser Extensions Infect 722 Users Across Latin America Since Early 2025

    Cybersecurity

    Empower Users and Protect Against GenAI Data Loss

    Cybersecurity

    Popular Chrome Extensions Leak API Keys, User Data via HTTP and Hardcoded Credentials

    Cybersecurity

    Critical Cisco ISE Auth Bypass Flaw Impacts Cloud Deployments on AWS, Azure, and OCI

    Cybersecurity

    Why Traditional DLP Solutions Fail in the Browser Era

    Cybersecurity

    HPE Issues Security Patch for StoreOnce Bug Allowing Remote Authentication Bypass

    Cybersecurity

    Critical 10-Year-Old Roundcube Webmail Bug Allows Authenticated Users Run Malicious Code

    Cybersecurity

    Android Trojan Crocodilus Now Active in 8 Countries, Targeting Banks and Crypto Wallets

    Cybersecurity

    Microsoft and CrowdStrike Launch Shared Threat Actor Glossary to Cut Attribution Confusion

  • Deals
    GTRACING Gaming Chair with Footrest Speakers Video Game Chair Bluetooth Music Heavy Duty…

    GTRACING Gaming Chair with Footrest Speakers Video Game Chair Bluetooth Music Heavy Duty…

    RoboCop Rogue City (PS5)

    RoboCop Rogue City (PS5)

    My Universe: Puppies and Kittens – PlayStation 4

    My Universe: Puppies and Kittens – PlayStation 4

    Disney’s Little Mermaid: Ariel’s Undersea Adventure – Nintendo DS (Renewed)

    Disney’s Little Mermaid: Ariel’s Undersea Adventure – Nintendo DS (Renewed)

    Family Game Pack 2001- PlayStation (Renewed)

    Family Game Pack 2001- PlayStation (Renewed)

    StarTech.com Cisco GLC-T Compatible SFP Module – 1000BASE-T – SFP to RJ45 Cat6/Cat5e -…

    StarTech.com Cisco GLC-T Compatible SFP Module – 1000BASE-T – SFP to RJ45 Cat6/Cat5e -…

    5-in-1 Memory Card Reader, USB OTG Adapter & SD Card Reader for i-Phone/i-Pad, USB C and…

    5-in-1 Memory Card Reader, USB OTG Adapter & SD Card Reader for i-Phone/i-Pad, USB C and…

    Samsung Galaxy A23 5G SM-A236U Factory Unlocked 64GB Black (Renewed)

    Samsung Galaxy A23 5G SM-A236U Factory Unlocked 64GB Black (Renewed)

    ORICO USB3.0 External 4 Bay Hard Drive Enclosure for 2.5 3.5 inch SSD HDD with…

    ORICO USB3.0 External 4 Bay Hard Drive Enclosure for 2.5 3.5 inch SSD HDD with…

  • Gaming
    The Callisto Protocol showed me what makes a GOOD GAME (Raptor Review)

    The Callisto Protocol showed me what makes a GOOD GAME (Raptor Review)

    REDRAGON S101 GAMING KEYBOARD

    Sony’s fast InZone M10S QD-OLED gaming monitor is over $300 off

    starship troopers: extermination, 2

    starship troopers: extermination, 2

    As Dusk Falls – Part 5 | XBOX SERIES X |

    As Dusk Falls – Part 5 | XBOX SERIES X |

    DIABLO 4: VESSEL OF HATRED SUCKS! (Review after 100%)

    DIABLO 4: VESSEL OF HATRED SUCKS! (Review after 100%)

    How to pay your taxes in Dune: Awakening

    How to pay your taxes in Dune: Awakening

    Sonic Frontiers  (Nintendo Switch Review)

    Sonic Frontiers (Nintendo Switch Review)

    Review on Gotham Knights – 50+ Hours on Hard Mode

    Review on Gotham Knights – 50+ Hours on Hard Mode

    Gungrave G.O.R.E. 12 – Labyrinth – Stage 11 – Hong Kong Back Alleys Area 1 | Easy | No Comm | Plat

    Gungrave G.O.R.E. 12 – Labyrinth – Stage 11 – Hong Kong Back Alleys Area 1 | Easy | No Comm | Plat

  • Tesla
    4 PCS LED Reverse Lights, 4014 45SMD 6500K 800LM High Bright Brake Light Turn Signal…

    4 PCS LED Reverse Lights, 4014 45SMD 6500K 800LM High Bright Brake Light Turn Signal…

    4 Pack Trailer Ball Cover, 2.36In x 2.24In x 1.97In Waterproof Dustproof Towing Hitch…

    4 Pack Trailer Ball Cover, 2.36In x 2.24In x 1.97In Waterproof Dustproof Towing Hitch…

    ClimaTex Heavy Duty Car, Truck, Van, and SUV Automotive Floor Mat for Floor Protection,…

    ClimaTex Heavy Duty Car, Truck, Van, and SUV Automotive Floor Mat for Floor Protection,…

    2 Pcs Tow Hook Covers Compatible with Tesla Cybertruck Accessories 2024 2025 (Red)

    2 Pcs Tow Hook Covers Compatible with Tesla Cybertruck Accessories 2024 2025 (Red)

    MAXDOM Under Seat Storage Fit for 2024+ Tesla Cybertruck Rear Underseat Organizer Box…

    MAXDOM Under Seat Storage Fit for 2024+ Tesla Cybertruck Rear Underseat Organizer Box…

    Car USB Hub Charger for Tesla Model Y 2021-2024 and Model 3 2021-2023,Fast…

    Car USB Hub Charger for Tesla Model Y 2021-2024 and Model 3 2021-2023,Fast…

    CAR GUYS Tire Shine Spray | High Gloss & Satin Finish | Non-Greasy, UV Protection,…

    CAR GUYS Tire Shine Spray | High Gloss & Satin Finish | Non-Greasy, UV Protection,…

    7PCS Center Console Organizer Tray for 2024 2025 Tesla Model 3 Accessories, Armrest…

    7PCS Center Console Organizer Tray for 2024 2025 Tesla Model 3 Accessories, Armrest…

    Spigen Door Striker and Hinge Cover Designed for Tesla Model 3 & Y [Compatible with…

    Spigen Door Striker and Hinge Cover Designed for Tesla Model 3 & Y [Compatible with…

  • UFO
    NASA UAP Researchers Share Shocking UFO Evidence!

    NASA UAP Researchers Share Shocking UFO Evidence!

    UFOs Over Phoenix: Confessions of a 911 Operator [DVD]

    UFOs Over Phoenix: Confessions of a 911 Operator [DVD]

    Have Aliens Visited Earth? | COLOSSAL MYSTERIES

    Have Aliens Visited Earth? | COLOSSAL MYSTERIES

    MINDBLOWING Encounters Unraveling the Secrets of Higher Dimensions

    MINDBLOWING Encounters Unraveling the Secrets of Higher Dimensions

    Roswell: The After-Action Report

    Roswell: The After-Action Report

    Alien UFO theories: AskReddit #ufo #alien #extraterrestrial #askreddit #reddit #creepystories #scary

    Alien UFO theories: AskReddit #ufo #alien #extraterrestrial #askreddit #reddit #creepystories #scary

    Resident Alien: Season One [DVD]

    Resident Alien: Season One [DVD]

    Close Up Zoom Moon 1000x beatiful view #SHORTVIDEO

    Close Up Zoom Moon 1000x beatiful view #SHORTVIDEO

    Black Triangle UFO: The Truth Behind the TR-3B Mystery

    Black Triangle UFO: The Truth Behind the TR-3B Mystery

No Result
View All Result
Techcratic
No Result
View All Result
Home Hacker News

Tokasaurus: An LLM Inference Engine for High-Throughput Workloads

Hacker News by Hacker News
June 5, 2025
in Hacker News
Reading Time: 10 mins read
129 1
A A
0
Share on FacebookShare on XShare on LinkedIn

2025-06-05 17:27:00
scalingintelligence.stanford.edu


We’re releasing Tokasaurus, a new LLM inference engine optimized for throughput-intensive workloads. With small models, Tokasaurus benefits from very low CPU overhead and dynamic Hydragen grouping to exploit shared prefixes. For larger models, Tokasaurus supports async tensor parallelism for GPUs with NVLink and a fast implementation of pipeline parallelism for GPUs without. On throughput-focused benchmarks, Tokasaurus can outperform vLLM and SGLang by up to 3x+.


Table of Contents


Intro

As LLMs get smarter, faster, and cheaper, the community keeps finding new ways to use them. Our own recent work has explored using models to scan every file in a codebase, sample 10,000 attempts for math and code problems, and collaborate with other models to minimize cloud costs. Inference is now also an important part of the training process, where we use models to generate synthetic data or as part of RL pipelines that generate and train on model completions.

Crucially, these new inference workloads look quite different than the original LLM use case of serving a chatbot. Here, we care primarily about the total time and cost required to complete a large batch of sequences, and we care much less (if at all) about the individual latency of a single generation. In other words, we want high throughput!

Open-source inference engines (i.e. dedicated systems for running efficient LLM inference) like FlexGen, vLLM, and SGLang have been enormously valuable to the community. Inspired by and learning from these projects, we built a new engine, Tokasaurus, designed from the ground up to handle throughput-focused workloads. We’ve optimized Tokasaurus for efficiently serving large and small models alike, allowing it to outperform existing engines on throughput benchmarks. In the rest of this blog, we’ll walk through some of these optimizations and show off a few settings where Tokasaurus really shines.


Optimizing Small Models

To benchmark Tokasaurus with small models, we’ll use two workloads:

  • Completing chatbot prompts from the ShareGPT dataset (this is a common benchmark for testing inference engines).
  • Reproducing an experiment from Large Language Monkeys, where we take 128 problems from the GSM8K math dataset and sample 1024 answers to every problem. The distinguishing feature of this workload is that there’s a lot of prefix sharing across sequences.
Tokasaurus small models
Tokasaurus large batch sampling

Tokasaurus outperforms vLLM and SGLang on both of these benchmarks, in particular achieving over 2x the throughput of other engines on the Large Language Monkeys workload. Two main features contribute to these wins with small models:

Minimizing CPU Overhead

LLM engines perform many different tasks on the CPU, like handling web requests, tokenizing inputs/detokenizing outputs, managing KV cache allocation, and preparing inputs for the model. If these CPU-side tasks cause the GPU-side model to stall, throughput can take a big hit. To avoid these stalls, inference engines commonly make many CPU-side tasks asynchronous: while the GPU runs a forward pass for batch N, the CPU-side of the engine post-processes the results from batch N-1 and prepares the inputs for batch N+1.

Tokasaurus goes one step further, making the CPU-side of the engine (what we call the manager) both asynchronous and adaptive. The manager’s goal is to maintain a deep queue of inputs for the model to run forward passes on. The manager monitors the size of this queue and can detect if the model is close to exhausting it (and therefore stalling the GPU). In these cases, the manager will automatically start skipping optional steps (like checking for stop strings and onboarding new sequences) until the model’s input queue is sufficiently deep again. This combination of asynchrony and adaptivity lets Tokasaurus serve small models with much less CPU overhead.

Dynamic Prefix Identification and Exploration

Prefix sharing comes up all the time in LLM inference — not just when repeatedly sampling like in the Large Language Monkeys benchmark, but also when asking many questions about a long document or reusing a system prompt across many chatbot conversations.

Shared prefixes allow attention to be computed more efficiently. We first explored this idea last year with Hydragen (aka cascade attention and bifurcated attention), but at the time we didn’t address the problem of detecting these shared prefixes in an engine where sequences are constantly starting and finishing. With Tokasaurus, we solve this detection problem by running a greedy depth-first search algorithm before every model forward pass that iteratively finds the longest shared prefixes possible. Hydragen is most impactful for small models, which spend a relatively larger fraction of total FLOPs on attention.


Optimizing Bigger Models

Tokasaurus can also efficiently serve bigger models across multiple GPUs! Here, the most important optimizations are our implementations of pipeline parallelism (PP) and tensor parallelism (TP), which allow us to maximize throughput on GPUs with or without NVLink.

Pipeline Parallelism for the GPU Poor

One of our original goals with Tokasaurus was to efficiently run multi-GPU inference on our lab’s L40S GPUs, which don’t have fast inter-GPU NVLink connections. Without NVLink, the communication costs incurred running TP across a node of eight GPUs are substantial. Therefore, efficient support for PP (which requires much less inter-GPU communication) was a high priority. PP needs a large batch in order to run efficiently, since batches from the manager are subdivided into microbatches that are spread out across pipeline stages. When optimizing for throughput, we’re generally already using the largest batch size that fits in GPU memory, so PP is often a natural fit for throughput-focused workloads. When benchmarking against vLLM’s and SGLang’s pipeline parallel implementations using Llama-3.1-70B on eight L40S GPUs, Tokasaurus improves throughput by over 3x:

Tokasaurus small models

Async Tensor Parallel for the GPU Rich

If you do have GPUs with NVLink (e.g. B200s and certain models of H100s and A100s), Tokasaurus has something for you too! Models in Tokasaurus can be torch compiled end-to-end, allowing us to take advantage of Async Tensor Parallelism (Async-TP). This is a relatively new feature in PyTorch that can overlap inter-GPU communication with other computations, partially hiding the cost of communication. In our benchmarks, we found that Async-TP adds a lot of CPU overhead to the model forward pass and only starts improving throughput with very large batch sizes (e.g. 6k+ tokens). Tokasaurus maintains torch-compiled versions of our models with and without Async-TP enabled, allowing us to automatically switch on Async-TP whenever the batch size is big enough:

Tokasaurus small models

Try it Out

Tokasaurus started as an internal lab effort to run our inference experiments faster, and we’re excited to share it more broadly! You can check out the Tokasaurus code on GitHub and install the package from PyPI with:

Currently, we support models from the Llama-3 and Qwen-2 families and support any combination of data, tensor, and pipeline parallel within a single node.

Tokasaurus is written in pure Python (although we do use attention and sampling ops from the excellent FlashInfer package). We hope that this makes the engine easier to fork and hack on, à la GPT-fast.

Benchmarking Details

The commands for reproducing our benchmarks are available here. For each benchmark, we configure all engines with the same KV cache size and maximum number of running requests. We’ve made a best effort to tune each engine’s remaining parameters. We report the average throughput across runs after completing a warmup run. For each benchmark, all engines are run on the same machine.

We use this script from SGLang for our ShareGPT benchmarks and this custom script for the Large Language Monkeys benchmark. To standardize our benchmarking scripts and interface, all experiments send requests through the OpenAI API. We also experimented with vLLM’s Python API (i.e. LLM.generate()) on the Large Language Monkeys benchmark with Llama-1B and measured roughly a 5% throughput increase (thanks to the vLLM team for the tip!).

Acknowledgements

Huge thanks to Prime Intellect and Together AI for providing us with compute for this project.

Also, we’re grateful to Dan Biderman, Simon Guo, Manat Kaur, and Avanika Narayan for beta testing the engine!


If you find Tokasaurus useful, please use the following citation:

@misc{juravsky2025tokasaurus,
  author       = {Jordan Juravsky and Ayush Chakravarthy and Ryan Ehrlich and Sabri Eyuboglu and Bradley Brown and Joseph Shetaye and Christopher R{\'e} and Azalia Mirhoseini},
  title        = {Tokasaurus: An LLM Inference Engine for High-Throughput Workloads},
  year         = {2025},
  howpublished = {\url{https://scalingintelligence.stanford.edu/blogs/tokasaurus/}}
}

Source Link


Keep your files stored safely and securely with the SanDisk 2TB Extreme Portable SSD. With over 69,505 ratings and an impressive 4.6 out of 5 stars, this product has been purchased over 8K+ times in the past month. At only $129.99, this Amazon’s Choice product is a must-have for secure file storage.

Help keep private content private with the included password protection featuring 256-bit AES hardware encryption. Order now for just $129.99 on Amazon!


Start your free Amazon Prime trial
today and unlock unlimited streaming and more!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!

BITCOIN

Bitcoin Logo

Bitcoin QR Code

bc1qlszw7elx2qahjwvaryh0tkgg8y68enw30gpvge

Scan the QR code with your crypto wallet app

DOGECOIN

Dogecoin Logo

Dogecoin QR Code

D64GwvvYQxFXYyan3oQCrmWfidf6T3JpBA

Scan the QR code with your crypto wallet app

ETHEREUM

Ethereum Logo

Ethereum QR Code

0xe9BC980DF3d985730dA827996B43E4A62CCBAA7a

Scan the QR code with your crypto wallet app

Please read the Privacy and Security Disclaimer on how Techcratic handles your support.

Disclaimer: As an Amazon Associate, Techcratic may earn from qualifying purchases.

Tags: Hacker News
Share161Tweet101Share28
Previous Post

TS Imagine develops customer service bot atop Snowflake

Next Post

Samsung is giving away free 32-inch smart monitors, here’s how to qualify

Hacker News

Hacker News

Stay updated with Hacker News, where technology meets entrepreneurial spirit. Get the latest on tech trends, startup news, and discussions from the tech community. Read the latest updates here at Techcratic.

Related Posts

Askannz/munal-os: An experimental operating system fully written in Rust, with a unikernel design, cooperative scheduling and a security model based on WASM sandboxing.
Hacker News

Askannz/munal-os: An experimental operating system fully written in Rust, with a unikernel design, cooperative scheduling and a security model based on WASM sandboxing.

June 9, 2025
1.3k
The New Godel Prize Winner Tastes Great and is Less Filling
Hacker News

The New Godel Prize Winner Tastes Great and is Less Filling

June 9, 2025
1.3k
my first attempt at iOS app development
Hacker News

my first attempt at iOS app development

June 8, 2025
1.3k
binfmtc – binfmt_misc C scripting interface
Hacker News

binfmtc – binfmt_misc C scripting interface

June 8, 2025
1.3k
Stop Vibe Coding. Start Cyborg Coding. | by Chase | Jun, 2025
Hacker News

Stop Vibe Coding. Start Cyborg Coding. | by Chase | Jun, 2025

June 7, 2025
1.3k
Discovering a JDK Race Condition, and Debugging it in 30 Minutes with Fray
Hacker News

Discovering a JDK Race Condition, and Debugging it in 30 Minutes with Fray

June 7, 2025
1.3k
Load More
Next Post
FIRE STICK

Samsung is giving away free 32-inch smart monitors, here's how to qualify

Cute Puppies Running to Me Every Morning | Feeding Biscuits & Playing Together

Cute Puppies Running to Me Every Morning | Feeding Biscuits & Playing Together

Bitcoin Craters to $100K as $837M in Crypto Positions Vanish

Bitcoin Craters to $100K as $837M in Crypto Positions Vanish

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Your Tech Resources

  • 30 Second Tech ™
  • AI
  • App Zone ™
  • Apple
  • Ars Technica
  • CNET
  • ComputerWorld
  • Crypto News
  • Cybersecurity
  • Endgadget
  • Fossbytes
  • Gaming
  • GeekWire
  • Gizmodo
  • Google News
  • Hacker News
  • Harvard Tech
  • I Like Cats ™
  • I Like Dogs ™
  • LifeHacker
  • MacRumors
  • Macworld
  • Mashable
  • Microsoft
  • MIT Tech
  • PC World
  • Photofocus
  • Physics
  • Random Tech
  • Retro Rewind ™
  • Robot Report
  • SiliconANGLE
  • SlashGear
  • Smartphone
  • StackSocial
  • Tech Art
  • Tech Careers
  • Tech Deals
  • Techcratic ™
  • TechCrunch
  • Techdirt
  • TechRepublic
  • Techs Got To Eat ™
  • TechSpot
  • Tesla
  • The Verge
  • TNW
  • Trusted Reviews
  • UFO
  • VentureBeat
  • Visual Capitalist
  • Weird Stuff
  • Wired
  • ZDNet

Tech News

  • 30 Second Tech ™
  • AI
  • AnandTech
  • Apple Insider
  • Ars Technica
  • CNET
  • ComputerWorld
  • Crypto News
  • Cybersecurity
  • Endgadget
  • ExtremeTech
  • Fossbytes
  • Gaming
  • GeekWire
  • Gizmodo

Tech News

  • Harvard Tech
  • MacRumors
  • Macworld
  • Mashable
  • Microsoft
  • MIT Tech
  • Physics
  • PC World
  • Random Tech
  • Retro Rewind ™
  • SiliconANGLE
  • SlashGear
  • Smartphone
  • StackSocial
  • Tech Careers

Tech News​

  • Tech Art
  • TechCrunch
  • Techdirt
  • TechRepublic
  • Techs Got To Eat ™
  • TechSpot
  • Tesla
  • The Verge
  • TNW
  • Trusted Reviews
  • UFO
  • VentureBeat
  • Visual Capitalist
  • Weird Stuff
  • Wired
  • ZDNet

Site Links

  • About Techcratic
  • Affiliate Disclaimer
  • Affiliate Link Policy
  • Contact Techcratic
  • Dealors Discount Store
  • Privacy and Security Disclaimer
  • Privacy Policy
  • RSS Feed
  • Site Map
  • Support Techcratic
  • Techcratic
  • Tech Deals
  • TOS
  • 𝕏
Click For A Secret Deal

Techcratic – Your All In One Tech Hub © 2020 – 2025
All Rights Reserved
∞

No Result
View All Result
  • Home
  • Apple
  • Gaming
  • Microsoft
  • AnandTech