Speaker
Description
What happens when high-performance native systems require secure, distributed access to modern large-scale data processing infrastructure?
This talk covers an open-source C++ client implementation built on the Spark Connect protocol, enabling Native Systems to communicate with remote Spark clusters using gRPC and Protobuf.
The talk explores:
- The architecture of Spark Connect and its decoupled client-server model
- Designing a secure, distributed query interface from native C++
- Authentication and production deployment considerations
- Leveraging gRPC for structured remote execution
- Packaging and building the client for Ubuntu & other Linux environments (Linux Package distribution)
- The role of open standards in expanding big data ecosystems
All these aspects demonstrate how systems-level engineering can integrate directly with cloud-native data platforms while preserving performance, portability, and security.
By extending Spark access beyond JVM and Python, this project enables integration with embedded systems, financial engines, scientific computing workloads, and other performance-sensitive environments.
The goal is not to replace existing Spark APIs but to expand the ecosystem through open-source, cross-language interoperability.
Justification
This talk showcases the role of Linux and open-source engineering in providing secure and scalable distributed systems. It also showcases systems-level thinking by exploring how native C++ applications can be integrated with modern data infrastructure using open protocols and open-source tooling.
It also highlights community contribution and knowledge sharing, demonstrating how local engineering can contribute to extending global infrastructure. The ideas are applicable to developers who are interested in Linux, distributed systems, and package distribution.
| Technical level | intermediate |
|---|---|
| Where are you based? | Nyeri |
| Submission type | Talk |