Course Outline
Introduction
Overview of Spark Streaming Features and Architecture
- Supported data sources
- Core APIs
Preparing the Environment
- Dependencies
- Spark and streaming context
- Connecting to Kafka
Processing Messages
- Parsing inbound messages as JSON
- ETL processes
- Starting the streaming context
Performing a Windowed Stream Processing
- Slide interval
- Checkpoint delivery configuration
- Launching the environment
Prototyping the Processing Code
- Connecting to a Kafka topic
- Retrieving JSON from data source using Paw
- Variations and additional processing
Streaming the Code
- Job control variables
- Defining values to match
- Functions and conditions
Acquiring Stream Output
- Counters
- Kafka output (matched and non-matched)
Troubleshooting
Summary and Conclusion
Requirements
- Experience with Python and Apache Kafka
- Familiarity with stream-processing platforms
Audience
- Data engineers
- Data scientists
- Programmers
Testimonials (5)
Examples/exercices perfectly adapted to our domain
Luc - CS Group
Course - Scaling Data Analysis with Python and Dask
The trainer was very available to answer all te kind of question I did
Caterina - Stamtech
Course - Developing APIs with Python and FastAPI
It was a though course as we had to cover a lot in a short time frame. Our trainer knew a lot about the subject and delivered the content to address our requirements. It was lots of content to learn but our trainer was helpful and encouraging. He answered all our questions with good detail and we feel that we learned a lot. Exercises were well prepared and tasks were tailored accordingly to our needs. I enjoyed this course
Bozena Stansfield - New College Durham
Course - Build REST APIs with Python and Flask
Przekazanie wiedzy praktycznej oraz doświadczeń trenera.
Rumel Mateusz - Pojazdy Szynowe PESA Bydgoszcz SA
Course - GUI Programming with Python and PyQt
As I was the only participant the training could be adapted to my needs.