Speaker
Description
In modern data science, the "Data Lakehouse" is a powerful way to manage large-scale data. However, for many teams, the "last mile" of integration—connecting Machine Learning platforms like Kubeflow to big data processing engines like Apache Spark—remains a major technical challenge. In a shared Kubernetes cluster, managing security, user permissions, and connection strings across different user profiles often leads to manual errors and security risks.
In this session, we will explore an open-source solution to this problem using Juju and Charmed Operators. We will walk through a "Multi-Integrator" architecture where each Kubeflow profile receives its own secure connection to the data platform. Instead of manual configuration, we use automated integrators to inject Spark Service Accounts, configurations, and security secrets directly into user namespaces.
By using Charmed Spark and Charmed Kubeflow, we can create a modular system where data scientists can move from a simple notebook to a distributed Spark job without worrying about the underlying YAML files or credentials. We will discuss the logic of this design, how it handles multi-tenancy, and how it simplifies the path to a production-ready data lakehouse on Ubuntu.
Biography
Bikalpa Dhakal is a software engineer and open-source enthusiast based in Pokhara, Nepal. He currently works in the Analytics and Processing team at Canonical, where he helps maintain the Charmed Apache Spark project and its repositories. Before joining Canonical, he worked for companies such as Logpoint. More recently at Canonical, he was involved in the technical integration between Charmed Apache Spark and Charmed Kubeflow. Outside of his professional work, Bikalpa's hobbies include cinema, traveling and aviation.
What audience can learn
Participants will learn about a practical architectural pattern for connecting Apache Spark with Kubeflow using Juju and Charmed Operators. The session explains how automated integrators can be used to manage security credentials and configurations across different user profiles in a shared Kubernetes cluster. By the end of the talk, attendees will have a better understanding of the technical steps required to build a modular data lakehouse and how to handle common integration challenges between data processing and machine learning workflows.
Things to know or prepare for this session
A basic understanding of Kubernetes (concepts like pods and namespaces) is recommended. You don't need to be an expert in Spark or Kubeflow, but knowing what they are used for will help you follow the session.
Summary
Connecting ML workflows to big data processing is difficult in multi-user environments. This talk shows how to automate these connections using Juju and Charmed Operators. We will explain a practical architecture that securely integrates Charmed Spark with Charmed Kubeflow, allowing data scientists
| Difficulty level | Intermediate |
|---|---|
| Language | English |