Beam College 2025 Sessions

This session provides an overview of Apache Beam and its place in the generative AI space. What problem does Beam solve? What are its advantages? Why and how are companies using it?

In this introductory session, Sascha provides a technical overview of Apache Beam and how it enables developers to build data pipelines that run on various frameworks like Spark and Flink. It supports both batch and stream processing, providing flexibility for diverse data needs. Beam’s SDKs are available in Go, Python, and Java, allowing developers to choose their preferred language.

In this session we explain how to implement a complex ML pipeline with Apache Beam. The pipeline we will build takes audio data, convert it to text, classify it to identify the topic or subject, feed it to a LLM and then take the output of the model and turn it back to voice.

This tutorial demonstrates how to perform streaming inference with Apache Beam and Google AI Studio’s Gemini model, focusing on a geography-based example to get country capitals.

This session dives into Apache Beam’s streaming primitives, focusing on reading from unbounded sources using Splittable DoFn and Unbounded Source, windowing strategies, and triggers with accumulation modes.

This presentation introduces vector-based semantic search and Retrieval Augmented Generation (RAG), demonstrating how to build scalable pipelines for using Apache Beam. We’ll start by explaining fundamental concepts like chunking, embeddings and vector similarity. Then we’ll explore semantic search applications before extending to full RAG systems.

Real-time anomaly detection is essential for identifying unexpected patterns and critical events in streaming data. This talk addresses the unique algorithmic challenges of anomaly detection in streaming environments and introduces a new feature within Apache Beam designed for this purpose. We will demonstrate how to seamlessly integrate both online and pre-trained offline anomaly detection models into Beam pipelines, empowering users to build robust, scalable, and real-time anomaly detection systems.

Learn how you can create low-code and no-code Beam YAML jobs in the Cloud Dataflow UI.

In this talk, we explore a new way of authoring and running your Beam pipelines; via the YAML SDK! Learn how you can split your pipeline infrastructure from your complex processing logic.

Discover what makes Managed IO a major leap towards a more unified, flexible, and upgrade-friendly connector ecosystem. This talk dives into the motivation and design behind Managed IO, highlighting a few key goals:

Explore real-time streaming pipelines with Kafka I/O. This session will share best practices for optimizing Kafka I/O performance and cost-efficiency, including strategies like redistribute transforms and offset-based deduplication. We will also cover integrating Dataflow with Google Managed Kafka for scalable data processing.

The stateful processing interface in Apache Beam serves as a versatile tool for data processing, empowering users with advanced capabilities to handle complex workflows. This session will delve into the diverse functionalities provided by stateful processing, illustrating their practical applications through clear and concise code examples.