This presentation introduces vector-based semantic search and Retrieval Augmented Generation (RAG), demonstrating how to build scalable pipelines for using Apache Beam. We’ll start by explaining fundamental concepts like chunking, embeddings and vector similarity. Then we’ll explore semantic search applications before extending to full RAG systems.

The presentation walks through implementing both semantic search and RAG pipelines using Apache Beam’s ML components, covering data ingestion, chunking, embedding generation, vector database integration, and similarity search. By the end, students will understand the theoretical foundations of both systems and have practical knowledge of how to implement them at scale using Apache Beam’s distributed processing capabilities.