Spark Structured Streaming 3.0 : All You Need to Know

Spark Structured Streaming 3.0 : All You Need to Know

Spark Structured Streaming 3.0 : All You Need to Know

Get to hands on from the first hour and travel through the concepts and details to emerge out master at the end

Language: english

Note: 4.5/5 (19 notes) 552 students

Instructor(s): Amit Ranjan

Last update: 2020-08-30

What you’ll learn

  • In Depth exploration of Spark Structured Streaming 3.0 using Python API.
  • Get introduced to Apache Kafka on a high level in the process.
  • Understand the nuances of Stream Processing in Apache Spark
  • Discover various features Spark provides out of the box for Stream Processing

 

Requirements

  • Understanding of Spark SQL and Python (or pyspark) will be helpful

 

Description

Getting faster action from the data is the need of many industries and Stream Processing helps doing just that. But it comes with its own set of theories, challenges and best practices.

Apache Spark has seen tremendous development being in stream processing. The rich features of Spark Structured Streaming introduces a learning curve and this course is aimed at bringing all those concepts in a friendly and easy to reflect manner. Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. You can express your streaming computation the same way you would express a batch computation on static data. The Spark SQL engine will take care of running it incrementally and continuously and updating the final result as streaming data continues to arrive. It allows data engineers and data scientists to process real-time data from various sources including (but not limited to) Kafka, Flume, and Amazon Kinesis.

This illustrative course will build your foundational knowledge. You will learn the differences between batch & stream processing, programming model, the APIs and the challenges specific to stream processing. Quickly we’ll move to understand the concepts of stream processing with wide varieties of examples & hands-on, dealing with inner working and taking a use case towards the end. All of this activity will be on cloud using Spark 3.0.

 

Who this course is for

  • Data Engineers looking to expand their skill set, Data Scientists who wish want hands on working with stream processing and Technical Architects who want to evaluate the Spark Structured Streaming for their use cases

 

Course content

  • First Steps with Spark Structured Streaming
    • Need and Challenges of Stream Processing
    • Concepts of Spark Structured Streaming
    • Structure of Spark Structured Streaming Application
    • Writing the first Structured Streaming Application
    • Basics of Spark Structured Streaming
  • Resources
    • Links to resources
  • Deep dive into the structured streaming
    • Understanding output modes
    • Windows in stream processing
    • Watermarking in stream processing
    • Quick Introduction to Kafka
    • Diving deep into the Structured Streaming
  • Integrating Spark Structured Streaming with Kafka
    • Watermarking with Kafka Source
    • Use Case: Get materialized advertisements by joining Kafka topics
  • Applying Structured Streaming in Production and Road to Expertise
    • Structured Streaming in Production
    • System Designs involving Spark Structured Streaming
    • Road to Expertise
    • Final Quiz

 

Spark Structured Streaming 3.0 : All You Need to KnowSpark Structured Streaming 3.0 : All You Need to Know

Time remaining or 558 enrolls left

 

Don’t miss any coupons by joining our Telegram group 

Udemy Coupon Code 100% off | Udemy Free Course | Udemy offer | Course with certificate