• About TC
  • Affiliate Disclaimer
  • Privacy Policy
  • TOS
  • Contact
Monday, June 9, 2025
Techcratic
  • TC
  • AI
    Artificial Intelligence

    10 Awesome OCR Models for 2025

    Artificial Intelligence

    5 Error Handling Patterns in Python (Beyond Try-Except)

    Artificial Intelligence

    Top 5 Alternative Data Career Paths and How to Learn Them for Free

    Artificial Intelligence

    Implementing Machine Learning Pipelines with Apache Spark

    Artificial Intelligence

    Learn Power BI for Free This Week

    Artificial Intelligence

    Build GraphRAG applications using Amazon Bedrock Knowledge Bases

    Artificial Intelligence

    How to Use Deep Research Like a Pro

    Artificial Intelligence

    World-Consistent Video Diffusion With Explicit 3D Modeling

    Artificial Intelligence

    Deploy Amazon SageMaker Projects with Terraform Cloud

  • Crypto
    ETF Weekly Flows: $129 Million Outflow for Bitcoin and $281 Million Inflow for Ether

    ETF Weekly Flows: $129 Million Outflow for Bitcoin and $281 Million Inflow for Ether

    DOGE Gets Distilled: Heritage Unleashes Dogecoin-Themed Bourbon

    DOGE Gets Distilled: Heritage Unleashes Dogecoin-Themed Bourbon

    Crypto ETFs centralize what was meant to be decentralized.

    Crypto ETFs centralize what was meant to be decentralized.

    Crypto Lost $1.64 Billion to Hackers in Q1 2025

    Why Is Crypto Down Today? – June 9, 2025

    The Blockchain Group Unveils $343 Million Capital Program to Boost Bitcoin Treasury Strategy

    The Blockchain Group Unveils $343 Million Capital Program to Boost Bitcoin Treasury Strategy

    Bitcoin Bull Cycle is Over: CryptoQuant CEO

    CEX Volumes Hit 2020 Lows as Market Shifts to HODL Mode

    Central African Republic to Launch Tokenized Land Sales on Solana

    Central African Republic to Launch Tokenized Land Sales on Solana

    XRP Added to Nasdaq Crypto US Settlement Price Index

    XRP Added to Nasdaq Crypto US Settlement Price Index

    Cetus Protocol Relaunches After $220M Hack, Restores Liquidity

    Cetus Protocol Relaunches After $220M Hack, Restores Liquidity

  • Cybersecurity
    Cybersecurity

    Malicious Browser Extensions Infect 722 Users Across Latin America Since Early 2025

    Cybersecurity

    Empower Users and Protect Against GenAI Data Loss

    Cybersecurity

    Popular Chrome Extensions Leak API Keys, User Data via HTTP and Hardcoded Credentials

    Cybersecurity

    Critical Cisco ISE Auth Bypass Flaw Impacts Cloud Deployments on AWS, Azure, and OCI

    Cybersecurity

    Why Traditional DLP Solutions Fail in the Browser Era

    Cybersecurity

    HPE Issues Security Patch for StoreOnce Bug Allowing Remote Authentication Bypass

    Cybersecurity

    Critical 10-Year-Old Roundcube Webmail Bug Allows Authenticated Users Run Malicious Code

    Cybersecurity

    Android Trojan Crocodilus Now Active in 8 Countries, Targeting Banks and Crypto Wallets

    Cybersecurity

    Microsoft and CrowdStrike Launch Shared Threat Actor Glossary to Cut Attribution Confusion

  • Deals
    GTRACING Gaming Chair with Footrest Speakers Video Game Chair Bluetooth Music Heavy Duty…

    GTRACING Gaming Chair with Footrest Speakers Video Game Chair Bluetooth Music Heavy Duty…

    RoboCop Rogue City (PS5)

    RoboCop Rogue City (PS5)

    My Universe: Puppies and Kittens – PlayStation 4

    My Universe: Puppies and Kittens – PlayStation 4

    Disney’s Little Mermaid: Ariel’s Undersea Adventure – Nintendo DS (Renewed)

    Disney’s Little Mermaid: Ariel’s Undersea Adventure – Nintendo DS (Renewed)

    Family Game Pack 2001- PlayStation (Renewed)

    Family Game Pack 2001- PlayStation (Renewed)

    StarTech.com Cisco GLC-T Compatible SFP Module – 1000BASE-T – SFP to RJ45 Cat6/Cat5e -…

    StarTech.com Cisco GLC-T Compatible SFP Module – 1000BASE-T – SFP to RJ45 Cat6/Cat5e -…

    5-in-1 Memory Card Reader, USB OTG Adapter & SD Card Reader for i-Phone/i-Pad, USB C and…

    5-in-1 Memory Card Reader, USB OTG Adapter & SD Card Reader for i-Phone/i-Pad, USB C and…

    Samsung Galaxy A23 5G SM-A236U Factory Unlocked 64GB Black (Renewed)

    Samsung Galaxy A23 5G SM-A236U Factory Unlocked 64GB Black (Renewed)

    ORICO USB3.0 External 4 Bay Hard Drive Enclosure for 2.5 3.5 inch SSD HDD with…

    ORICO USB3.0 External 4 Bay Hard Drive Enclosure for 2.5 3.5 inch SSD HDD with…

  • Gaming
    The Callisto Protocol showed me what makes a GOOD GAME (Raptor Review)

    The Callisto Protocol showed me what makes a GOOD GAME (Raptor Review)

    REDRAGON S101 GAMING KEYBOARD

    Sony’s fast InZone M10S QD-OLED gaming monitor is over $300 off

    starship troopers: extermination, 2

    starship troopers: extermination, 2

    As Dusk Falls – Part 5 | XBOX SERIES X |

    As Dusk Falls – Part 5 | XBOX SERIES X |

    DIABLO 4: VESSEL OF HATRED SUCKS! (Review after 100%)

    DIABLO 4: VESSEL OF HATRED SUCKS! (Review after 100%)

    How to pay your taxes in Dune: Awakening

    How to pay your taxes in Dune: Awakening

    Sonic Frontiers  (Nintendo Switch Review)

    Sonic Frontiers (Nintendo Switch Review)

    Review on Gotham Knights – 50+ Hours on Hard Mode

    Review on Gotham Knights – 50+ Hours on Hard Mode

    Gungrave G.O.R.E. 12 – Labyrinth – Stage 11 – Hong Kong Back Alleys Area 1 | Easy | No Comm | Plat

    Gungrave G.O.R.E. 12 – Labyrinth – Stage 11 – Hong Kong Back Alleys Area 1 | Easy | No Comm | Plat

  • Tesla
    4 PCS LED Reverse Lights, 4014 45SMD 6500K 800LM High Bright Brake Light Turn Signal…

    4 PCS LED Reverse Lights, 4014 45SMD 6500K 800LM High Bright Brake Light Turn Signal…

    4 Pack Trailer Ball Cover, 2.36In x 2.24In x 1.97In Waterproof Dustproof Towing Hitch…

    4 Pack Trailer Ball Cover, 2.36In x 2.24In x 1.97In Waterproof Dustproof Towing Hitch…

    ClimaTex Heavy Duty Car, Truck, Van, and SUV Automotive Floor Mat for Floor Protection,…

    ClimaTex Heavy Duty Car, Truck, Van, and SUV Automotive Floor Mat for Floor Protection,…

    2 Pcs Tow Hook Covers Compatible with Tesla Cybertruck Accessories 2024 2025 (Red)

    2 Pcs Tow Hook Covers Compatible with Tesla Cybertruck Accessories 2024 2025 (Red)

    MAXDOM Under Seat Storage Fit for 2024+ Tesla Cybertruck Rear Underseat Organizer Box…

    MAXDOM Under Seat Storage Fit for 2024+ Tesla Cybertruck Rear Underseat Organizer Box…

    Car USB Hub Charger for Tesla Model Y 2021-2024 and Model 3 2021-2023,Fast…

    Car USB Hub Charger for Tesla Model Y 2021-2024 and Model 3 2021-2023,Fast…

    CAR GUYS Tire Shine Spray | High Gloss & Satin Finish | Non-Greasy, UV Protection,…

    CAR GUYS Tire Shine Spray | High Gloss & Satin Finish | Non-Greasy, UV Protection,…

    7PCS Center Console Organizer Tray for 2024 2025 Tesla Model 3 Accessories, Armrest…

    7PCS Center Console Organizer Tray for 2024 2025 Tesla Model 3 Accessories, Armrest…

    Spigen Door Striker and Hinge Cover Designed for Tesla Model 3 & Y [Compatible with…

    Spigen Door Striker and Hinge Cover Designed for Tesla Model 3 & Y [Compatible with…

  • UFO
    NASA UAP Researchers Share Shocking UFO Evidence!

    NASA UAP Researchers Share Shocking UFO Evidence!

    UFOs Over Phoenix: Confessions of a 911 Operator [DVD]

    UFOs Over Phoenix: Confessions of a 911 Operator [DVD]

    Have Aliens Visited Earth? | COLOSSAL MYSTERIES

    Have Aliens Visited Earth? | COLOSSAL MYSTERIES

    MINDBLOWING Encounters Unraveling the Secrets of Higher Dimensions

    MINDBLOWING Encounters Unraveling the Secrets of Higher Dimensions

    Roswell: The After-Action Report

    Roswell: The After-Action Report

    Alien UFO theories: AskReddit #ufo #alien #extraterrestrial #askreddit #reddit #creepystories #scary

    Alien UFO theories: AskReddit #ufo #alien #extraterrestrial #askreddit #reddit #creepystories #scary

    Resident Alien: Season One [DVD]

    Resident Alien: Season One [DVD]

    Close Up Zoom Moon 1000x beatiful view #SHORTVIDEO

    Close Up Zoom Moon 1000x beatiful view #SHORTVIDEO

    Black Triangle UFO: The Truth Behind the TR-3B Mystery

    Black Triangle UFO: The Truth Behind the TR-3B Mystery

No Result
View All Result
  • TC
  • AI
    Artificial Intelligence

    10 Awesome OCR Models for 2025

    Artificial Intelligence

    5 Error Handling Patterns in Python (Beyond Try-Except)

    Artificial Intelligence

    Top 5 Alternative Data Career Paths and How to Learn Them for Free

    Artificial Intelligence

    Implementing Machine Learning Pipelines with Apache Spark

    Artificial Intelligence

    Learn Power BI for Free This Week

    Artificial Intelligence

    Build GraphRAG applications using Amazon Bedrock Knowledge Bases

    Artificial Intelligence

    How to Use Deep Research Like a Pro

    Artificial Intelligence

    World-Consistent Video Diffusion With Explicit 3D Modeling

    Artificial Intelligence

    Deploy Amazon SageMaker Projects with Terraform Cloud

  • Crypto
    ETF Weekly Flows: $129 Million Outflow for Bitcoin and $281 Million Inflow for Ether

    ETF Weekly Flows: $129 Million Outflow for Bitcoin and $281 Million Inflow for Ether

    DOGE Gets Distilled: Heritage Unleashes Dogecoin-Themed Bourbon

    DOGE Gets Distilled: Heritage Unleashes Dogecoin-Themed Bourbon

    Crypto ETFs centralize what was meant to be decentralized.

    Crypto ETFs centralize what was meant to be decentralized.

    Crypto Lost $1.64 Billion to Hackers in Q1 2025

    Why Is Crypto Down Today? – June 9, 2025

    The Blockchain Group Unveils $343 Million Capital Program to Boost Bitcoin Treasury Strategy

    The Blockchain Group Unveils $343 Million Capital Program to Boost Bitcoin Treasury Strategy

    Bitcoin Bull Cycle is Over: CryptoQuant CEO

    CEX Volumes Hit 2020 Lows as Market Shifts to HODL Mode

    Central African Republic to Launch Tokenized Land Sales on Solana

    Central African Republic to Launch Tokenized Land Sales on Solana

    XRP Added to Nasdaq Crypto US Settlement Price Index

    XRP Added to Nasdaq Crypto US Settlement Price Index

    Cetus Protocol Relaunches After $220M Hack, Restores Liquidity

    Cetus Protocol Relaunches After $220M Hack, Restores Liquidity

  • Cybersecurity
    Cybersecurity

    Malicious Browser Extensions Infect 722 Users Across Latin America Since Early 2025

    Cybersecurity

    Empower Users and Protect Against GenAI Data Loss

    Cybersecurity

    Popular Chrome Extensions Leak API Keys, User Data via HTTP and Hardcoded Credentials

    Cybersecurity

    Critical Cisco ISE Auth Bypass Flaw Impacts Cloud Deployments on AWS, Azure, and OCI

    Cybersecurity

    Why Traditional DLP Solutions Fail in the Browser Era

    Cybersecurity

    HPE Issues Security Patch for StoreOnce Bug Allowing Remote Authentication Bypass

    Cybersecurity

    Critical 10-Year-Old Roundcube Webmail Bug Allows Authenticated Users Run Malicious Code

    Cybersecurity

    Android Trojan Crocodilus Now Active in 8 Countries, Targeting Banks and Crypto Wallets

    Cybersecurity

    Microsoft and CrowdStrike Launch Shared Threat Actor Glossary to Cut Attribution Confusion

  • Deals
    GTRACING Gaming Chair with Footrest Speakers Video Game Chair Bluetooth Music Heavy Duty…

    GTRACING Gaming Chair with Footrest Speakers Video Game Chair Bluetooth Music Heavy Duty…

    RoboCop Rogue City (PS5)

    RoboCop Rogue City (PS5)

    My Universe: Puppies and Kittens – PlayStation 4

    My Universe: Puppies and Kittens – PlayStation 4

    Disney’s Little Mermaid: Ariel’s Undersea Adventure – Nintendo DS (Renewed)

    Disney’s Little Mermaid: Ariel’s Undersea Adventure – Nintendo DS (Renewed)

    Family Game Pack 2001- PlayStation (Renewed)

    Family Game Pack 2001- PlayStation (Renewed)

    StarTech.com Cisco GLC-T Compatible SFP Module – 1000BASE-T – SFP to RJ45 Cat6/Cat5e -…

    StarTech.com Cisco GLC-T Compatible SFP Module – 1000BASE-T – SFP to RJ45 Cat6/Cat5e -…

    5-in-1 Memory Card Reader, USB OTG Adapter & SD Card Reader for i-Phone/i-Pad, USB C and…

    5-in-1 Memory Card Reader, USB OTG Adapter & SD Card Reader for i-Phone/i-Pad, USB C and…

    Samsung Galaxy A23 5G SM-A236U Factory Unlocked 64GB Black (Renewed)

    Samsung Galaxy A23 5G SM-A236U Factory Unlocked 64GB Black (Renewed)

    ORICO USB3.0 External 4 Bay Hard Drive Enclosure for 2.5 3.5 inch SSD HDD with…

    ORICO USB3.0 External 4 Bay Hard Drive Enclosure for 2.5 3.5 inch SSD HDD with…

  • Gaming
    The Callisto Protocol showed me what makes a GOOD GAME (Raptor Review)

    The Callisto Protocol showed me what makes a GOOD GAME (Raptor Review)

    REDRAGON S101 GAMING KEYBOARD

    Sony’s fast InZone M10S QD-OLED gaming monitor is over $300 off

    starship troopers: extermination, 2

    starship troopers: extermination, 2

    As Dusk Falls – Part 5 | XBOX SERIES X |

    As Dusk Falls – Part 5 | XBOX SERIES X |

    DIABLO 4: VESSEL OF HATRED SUCKS! (Review after 100%)

    DIABLO 4: VESSEL OF HATRED SUCKS! (Review after 100%)

    How to pay your taxes in Dune: Awakening

    How to pay your taxes in Dune: Awakening

    Sonic Frontiers  (Nintendo Switch Review)

    Sonic Frontiers (Nintendo Switch Review)

    Review on Gotham Knights – 50+ Hours on Hard Mode

    Review on Gotham Knights – 50+ Hours on Hard Mode

    Gungrave G.O.R.E. 12 – Labyrinth – Stage 11 – Hong Kong Back Alleys Area 1 | Easy | No Comm | Plat

    Gungrave G.O.R.E. 12 – Labyrinth – Stage 11 – Hong Kong Back Alleys Area 1 | Easy | No Comm | Plat

  • Tesla
    4 PCS LED Reverse Lights, 4014 45SMD 6500K 800LM High Bright Brake Light Turn Signal…

    4 PCS LED Reverse Lights, 4014 45SMD 6500K 800LM High Bright Brake Light Turn Signal…

    4 Pack Trailer Ball Cover, 2.36In x 2.24In x 1.97In Waterproof Dustproof Towing Hitch…

    4 Pack Trailer Ball Cover, 2.36In x 2.24In x 1.97In Waterproof Dustproof Towing Hitch…

    ClimaTex Heavy Duty Car, Truck, Van, and SUV Automotive Floor Mat for Floor Protection,…

    ClimaTex Heavy Duty Car, Truck, Van, and SUV Automotive Floor Mat for Floor Protection,…

    2 Pcs Tow Hook Covers Compatible with Tesla Cybertruck Accessories 2024 2025 (Red)

    2 Pcs Tow Hook Covers Compatible with Tesla Cybertruck Accessories 2024 2025 (Red)

    MAXDOM Under Seat Storage Fit for 2024+ Tesla Cybertruck Rear Underseat Organizer Box…

    MAXDOM Under Seat Storage Fit for 2024+ Tesla Cybertruck Rear Underseat Organizer Box…

    Car USB Hub Charger for Tesla Model Y 2021-2024 and Model 3 2021-2023,Fast…

    Car USB Hub Charger for Tesla Model Y 2021-2024 and Model 3 2021-2023,Fast…

    CAR GUYS Tire Shine Spray | High Gloss & Satin Finish | Non-Greasy, UV Protection,…

    CAR GUYS Tire Shine Spray | High Gloss & Satin Finish | Non-Greasy, UV Protection,…

    7PCS Center Console Organizer Tray for 2024 2025 Tesla Model 3 Accessories, Armrest…

    7PCS Center Console Organizer Tray for 2024 2025 Tesla Model 3 Accessories, Armrest…

    Spigen Door Striker and Hinge Cover Designed for Tesla Model 3 & Y [Compatible with…

    Spigen Door Striker and Hinge Cover Designed for Tesla Model 3 & Y [Compatible with…

  • UFO
    NASA UAP Researchers Share Shocking UFO Evidence!

    NASA UAP Researchers Share Shocking UFO Evidence!

    UFOs Over Phoenix: Confessions of a 911 Operator [DVD]

    UFOs Over Phoenix: Confessions of a 911 Operator [DVD]

    Have Aliens Visited Earth? | COLOSSAL MYSTERIES

    Have Aliens Visited Earth? | COLOSSAL MYSTERIES

    MINDBLOWING Encounters Unraveling the Secrets of Higher Dimensions

    MINDBLOWING Encounters Unraveling the Secrets of Higher Dimensions

    Roswell: The After-Action Report

    Roswell: The After-Action Report

    Alien UFO theories: AskReddit #ufo #alien #extraterrestrial #askreddit #reddit #creepystories #scary

    Alien UFO theories: AskReddit #ufo #alien #extraterrestrial #askreddit #reddit #creepystories #scary

    Resident Alien: Season One [DVD]

    Resident Alien: Season One [DVD]

    Close Up Zoom Moon 1000x beatiful view #SHORTVIDEO

    Close Up Zoom Moon 1000x beatiful view #SHORTVIDEO

    Black Triangle UFO: The Truth Behind the TR-3B Mystery

    Black Triangle UFO: The Truth Behind the TR-3B Mystery

No Result
View All Result
Techcratic
No Result
View All Result
Home AI

Implementing Machine Learning Pipelines with Apache Spark

AI by AI
June 3, 2025
in AI
Reading Time: 7 mins read
127 10
A A
0
Share on FacebookShare on XShare on LinkedIn

Jayita Gulati
2025-06-03 08:00:00
www.kdnuggets.com

Machine Learning Pipelines with Apache Spark
Image by Editor (Kanwal Mehreen) | Canva

 

Apache Spark is a tool for working with big data. It is free to use and very fast. Spark can manage large amounts of data that don’t fit in a computer’s memory. A machine learning pipeline is a series of steps to prepare data and train models. These steps include collecting data, cleaning it, selecting important features, training the model, and checking how well it works.

Spark makes it easy to build these pipelines. With Spark, companies can quickly analyze large amounts of data and create machine learning models. This helps them make better decisions based on the information they have. In this article, we will explain how to set up and use machine learning pipelines in Spark.

 

Components of a Machine Learning Pipeline in Spark

 
Spark’s MLlib library has many built-in tools. These tools can be linked together to build a complete machine learning process.

 

Transformers

Transformers change data in some way. They take a DataFrame and return a modified version of it. These are used for tasks like encoding categorical data or scaling numerical features. Examples include StringIndexer (for encoding) and StandardScaler (for scaling). Transformers are reusable and don’t change the original data permanently.

 

Estimators

Estimators learn from data to create models. They include algorithms like LogisticRegression and RandomForestClassifier. Estimators use a fit method to train on data, and they output a Model object that can make predictions.

 

Pipeline

A Pipeline is a tool to connect transformers and estimators into a single workflow. By organizing them in sequence, data flows smoothly from one step to the next. Pipelines make it easy to retrain models, repeat processes, and adjust parameters.

Let’s go through a basic example of building a classification pipeline to predict customer churn. In this pipeline, we’ll:

  1. Load the Data: Import the dataset into Spark for processing.
  2. Preprocess the Data: Clean and prepare the data for modeling.
  3. Setup the Model: Prepare the logistic regression model.
  4. Train the Model: Fit a machine learning model to the data.
  5. Evaluate the Model: Check how well the model performs.

 

Initialize Spark Session and Load Dataset

 
First, we use SparkSession.builder to set up the session. Then, we load the customer churn dataset. This churn data is about bank customers who have closed their accounts.

from pyspark.sql import SparkSession

# Initialize Spark session
spark = SparkSession.builder.appName("MLPipeline").getOrCreate()

# Load dataset
data = spark.read.csv("/content/Customer Churn.csv", header=True, inferSchema=True)

# Show the first few rows of the dataset
data.show(5)

 
dataset

 

Data Preprocessing

 
First, we check the data for any missing values. If there are missing values, we remove those rows to make sure the data is complete. Next, we convert categorical data into numerical format so that the computer can understand it. We do this using methods like StringIndexer and OneHotEncoder. Finally, we combine all the features into a single vector and scale the data.

from pyspark.sql import functions as F
from pyspark.ml.feature import StringIndexer, OneHotEncoder, VectorAssembler, StandardScaler

# Check for missing values
missing_values = data.select([F.count(F.when(F.isnan(c) | F.col(c).isNull(), c)).alias(c) for c in data.columns])

# Drop rows with any missing values
data = data.na.drop()  

# Identify categorical columns
categorical_columns = ['country', 'gender', 'credit_card', 'active_member']

# Create a list to hold the stages of the pipeline
stages = []

# Apply StringIndexer to convert categorical columns to numerical indices
for column in categorical_columns:
    indexer = StringIndexer(inputCol=column, outputCol=column + "_index")
    stages.append(indexer)

    # Apply OneHotEncoder for categorical features
    encoder = OneHotEncoder(inputCols=[column + "_index"], outputCols=[column + "_ohe"])
    stages.append(encoder)

label_column = 'churn'  # The label column
feature_columns = [column + "_ohe" for column in categorical_columns]

# Add numerical columns to the features list
numerical_columns = ['credit_score', 'age', 'tenure', 'balance', 'products_number', 'estimated_salary']
feature_columns += numerical_columns

# Create VectorAssembler to combine all feature columns
vector_assembler = VectorAssembler(inputCols=feature_columns, outputCol="features")
stages.append(vector_assembler)

# Scale the features using StandardScaler
scaler = StandardScaler(inputCol="features", outputCol="scaled_features", withMean=True, withStd=True)
stages.append(scaler)

 

Logistic Regression Model Setup

 
We import LogisticRegression from pyspark.ml.classification. Next, we create a logistic regression model by using LogisticRegression().

from pyspark.ml.classification import LogisticRegression
from pyspark.ml import Pipeline

# Logistic Regression Model
lr = LogisticRegression(featuresCol="scaled_features", labelCol=label_column)
stages.append(lr)

# Create and Run the Pipeline
pipeline = Pipeline(stages=stages)

 

Model Training and Predictions

 
We split the dataset into training and testing sets. Then, we fit the pipeline model to the training data and make predictions on the test data.

# Split data into training and testing sets
train_data, test_data = data.randomSplit([0.8, 0.2], seed=42)

# Fit the model
pipeline_model = pipeline.fit(train_data)

# Make Predictions
predictions = pipeline_model.transform(test_data)

# Show the predictions
predictions.select("prediction", label_column, "scaled_features").show(10)

 
predictions
 

Model Evaluation

 
We import MulticlassClassificationEvaluator from pyspark.ml.evaluation to evaluate our model’s performance. We calculate the accuracy, precision, recall, and F1 score using the predictions from our model. Finally, we stop the Spark session to free up resources.

from pyspark.ml.evaluation import MulticlassClassificationEvaluator

# Accuracy
evaluator_accuracy = MulticlassClassificationEvaluator(labelCol=label_column, predictionCol="prediction", metricName="accuracy")
accuracy = evaluator_accuracy.evaluate(predictions)
print(f"Accuracy: {accuracy}")

# Precision
evaluator_precision = MulticlassClassificationEvaluator(labelCol=label_column, predictionCol="prediction", metricName="weightedPrecision")
precision = evaluator_precision.evaluate(predictions)
print(f"Precision: {precision}")

# Recall
evaluator_recall = MulticlassClassificationEvaluator(labelCol=label_column, predictionCol="prediction", metricName="weightedRecall")
recall = evaluator_recall.evaluate(predictions)
print(f"Recall: {recall}")

# F1 Score
evaluator_f1 = MulticlassClassificationEvaluator(labelCol=label_column, predictionCol="prediction", metricName="f1")
f1_score = evaluator_f1.evaluate(predictions)
print(f"F1 Score: {f1_score}")

# Stop Spark session
spark.stop()

 
evaluation

 

Conclusion

 
In this article, we learned about machine learning pipelines in Apache Spark. Pipelines help organize each step of the ML process. We started by loading and cleaning the customer churn dataset. Then, we transformed the data and created a logistic regression model. After training the model, we made predictions on new data. Finally, we evaluated the model’s performance using accuracy, precision, recall, and F1 score.
 
 

Jayita Gulati is a machine learning enthusiast and technical writer driven by her passion for building machine learning models. She holds a Master’s degree in Computer Science from the University of Liverpool.

Source Link



Shark AI Ultra Voice Control Robot Vacuum

Transform your cleaning routine with the Shark AI Ultra Voice Control Robot Vacuum! This high-tech marvel boasts over 32,487 ratings, an impressive 4.2 out of 5 stars, and has been purchased over 900 times in the past month. Perfect for keeping your home spotless with minimal effort, this vacuum is now available for the unbeatable price of $349.99!

Don’t miss out on this limited-time offer. Order now and let Shark AI do the work for you!


Start your free Amazon Prime trial
today and unlock unlimited streaming and more!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!

BITCOIN

Bitcoin Logo

Bitcoin QR Code

bc1qlszw7elx2qahjwvaryh0tkgg8y68enw30gpvge

Scan the QR code with your crypto wallet app

DOGECOIN

Dogecoin Logo

Dogecoin QR Code

D64GwvvYQxFXYyan3oQCrmWfidf6T3JpBA

Scan the QR code with your crypto wallet app

ETHEREUM

Ethereum Logo

Ethereum QR Code

0xe9BC980DF3d985730dA827996B43E4A62CCBAA7a

Scan the QR code with your crypto wallet app

Please read the Privacy and Security Disclaimer on how Techcratic handles your support.

Disclaimer: As an Amazon Associate, Techcratic may earn from qualifying purchases.

Tags: AI NEWS
Share170Tweet106Share30
Previous Post

Brianna Bryan Interview – Blue’s Clues & You: Rainbow Puppy Adventures

Next Post

ERP/Database Specialist

AI

AI

Explore the dynamic realm of AI, where breakthroughs and trends are shaping the future. Stay informed and see how AI is making an impact. Don’t miss the crucial updates—read the latest articles here at Techcratic.

Related Posts

Artificial Intelligence
AI

10 Awesome OCR Models for 2025

June 6, 2025
1.3k
Artificial Intelligence
AI

5 Error Handling Patterns in Python (Beyond Try-Except)

June 6, 2025
1.3k
Artificial Intelligence
AI

Top 5 Alternative Data Career Paths and How to Learn Them for Free

June 5, 2025
1.3k
Artificial Intelligence
AI

Learn Power BI for Free This Week

June 2, 2025
1.4k
Artificial Intelligence
AI

Build GraphRAG applications using Amazon Bedrock Knowledge Bases

June 2, 2025
1.3k
Artificial Intelligence
AI

How to Use Deep Research Like a Pro

June 2, 2025
1.4k
Load More
Next Post
2 HOUR JOB SEARCH

ERP/Database Specialist

Smartphone

Adobe Photoshop finally launches on Android

Klarna Has a New BNPL Debit Card That Seems Like a Credit Card. Is It Worth Getting?

Klarna Has a New BNPL Debit Card That Seems Like a Credit Card. Is It Worth Getting?

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Your Tech Resources

  • 30 Second Tech ™
  • AI
  • App Zone ™
  • Apple
  • Ars Technica
  • CNET
  • ComputerWorld
  • Crypto News
  • Cybersecurity
  • Endgadget
  • Fossbytes
  • Gaming
  • GeekWire
  • Gizmodo
  • Google News
  • Hacker News
  • Harvard Tech
  • I Like Cats ™
  • I Like Dogs ™
  • LifeHacker
  • MacRumors
  • Macworld
  • Mashable
  • Microsoft
  • MIT Tech
  • PC World
  • Photofocus
  • Physics
  • Random Tech
  • Retro Rewind ™
  • Robot Report
  • SiliconANGLE
  • SlashGear
  • Smartphone
  • StackSocial
  • Tech Art
  • Tech Careers
  • Tech Deals
  • Techcratic ™
  • TechCrunch
  • Techdirt
  • TechRepublic
  • Techs Got To Eat ™
  • TechSpot
  • Tesla
  • The Verge
  • TNW
  • Trusted Reviews
  • UFO
  • VentureBeat
  • Visual Capitalist
  • Weird Stuff
  • Wired
  • ZDNet

Tech News

  • 30 Second Tech ™
  • AI
  • AnandTech
  • Apple Insider
  • Ars Technica
  • CNET
  • ComputerWorld
  • Crypto News
  • Cybersecurity
  • Endgadget
  • ExtremeTech
  • Fossbytes
  • Gaming
  • GeekWire
  • Gizmodo

Tech News

  • Harvard Tech
  • MacRumors
  • Macworld
  • Mashable
  • Microsoft
  • MIT Tech
  • Physics
  • PC World
  • Random Tech
  • Retro Rewind ™
  • SiliconANGLE
  • SlashGear
  • Smartphone
  • StackSocial
  • Tech Careers

Tech News​

  • Tech Art
  • TechCrunch
  • Techdirt
  • TechRepublic
  • Techs Got To Eat ™
  • TechSpot
  • Tesla
  • The Verge
  • TNW
  • Trusted Reviews
  • UFO
  • VentureBeat
  • Visual Capitalist
  • Weird Stuff
  • Wired
  • ZDNet

Site Links

  • About Techcratic
  • Affiliate Disclaimer
  • Affiliate Link Policy
  • Contact Techcratic
  • Dealors Discount Store
  • Privacy and Security Disclaimer
  • Privacy Policy
  • RSS Feed
  • Site Map
  • Support Techcratic
  • Techcratic
  • Tech Deals
  • TOS
  • 𝕏
Click For A Secret Deal

Techcratic – Your All In One Tech Hub © 2020 – 2025
All Rights Reserved
∞

No Result
View All Result
  • Home
  • Apple
  • Gaming
  • Microsoft
  • AnandTech