Senior Software Engineer - Confluent Kafka

**Introduction** At IBM Software, we transform client challenges into solutions. Building the world’s leading AI-powered, cloud-native products that shape the future of business and society. Our legacy of innovation creates endless opportunities for IBMers to learn, grow, and make an impact on a global scale. Working in Software means joining a team fueled by curiosity and collaboration. You’ll work with diverse technologies, partners, and industries to design, develop, and deliver solutions that power digital transformation. With a culture that values innovation, growth, and continuous learning, IBM Software places you at the heart of IBM’s product and technology landscape. Here, you’ll have the tools and opportunities to advance your career while creating software that changes the world. With Confluent, data doesn’t sit still. We put information in motion, streaming in near real time so organizations can react faster, build smarter, and deliver experiences as dynamic as the world around them. **Your role and responsibilities** About Product & Team A software engineer on this team primarily contributes by implementing and deploying the business logic inside some KORA components that run in production. Their code defines how the system behaves, how features are delivered, and how the platform responds under real customer workloads. KORA is the cloud native kafka engine. More about KORA [1] https://www.confluent.io/blog/cloud-native-kafka-kora-vldb-award/ About the Role We are a team of passionate engineers who love solving complex distributed systems and infrastructure problems. We are building Kora from the ground up to be a true managed service for clients who demand high levels of availability and performance at the lowest cost of ownership. In this role, you will be working on building highly available, reliable, secure, scalable, and multi-tenant Kora's compute layer. You will be instrumental in driving the technical solutions end-to-end, working closely with the team and key stakeholders to design, architect, and develop top-notch solutions. Here are examples of some of the challenges we've taken on in recent years: [2] Kora: A Cloud-Native Event Streaming Platform For Kafka [3] Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable Consumer Groups [4] From On-Prem to Cloud-Native: Multi-Tenancy in Confluent Cloud [5] Speed, Scale, Storage: Our Journey from Apache Kafka to Performance in Confluent Cloud [6] Benchmarking Confluent Cloud What You’ll Do As an L3 Senior Software Engineer, you are a mostly self-sufficient team member who owns and drives well-scoped to moderately complex problems end-to-end in your product area and helps other engineers with specific tasks when needed. In this role you will: * Design, implement, and maintain services that power catalog, publishing, distribution, and lifecycle management for connectors and other Kafka/Flink artifacts, within a well-defined portion of the Marketplace & Hub domain. * Deliver large features independently and with high quality, including tests, logging, and error handling, contributing to bigger initiatives owned by the team. * Build a solid understanding of the design and architecture of your product area (e.g., specific Marketplace/Hub services, data models, and APIs) and use that to make sound design and implementation choices. * Participate actively in technical processes for your team: design/one-pager reviews, RFC/KIP discussions where relevant, and PR reviews, providing constructive feedback to peers. * Improve performance, scalability, reliability, and security of Marketplace & Hub services through incremental changes: refactors, dependency upgrades, schema evolution, and automation of existing manual workflows. * Take shared ownership of operational excellence by handling on-call responsibilities, following runbooks, performing production operations safely, and contributing to better monitoring, alerting, and incident remediation. * Collaborate with engineers across teams (Kafka Connect, Partner Engineering, Security, Billing, UX) on well-scoped cross-team efforts, unblocking yourself and others as needed. * Help maintain a high engineering bar on the team through good coding practices, thoughtful reviews, and reliable delivery on commitments. What You Will Do * Driven and excited about challenges of delivering complex projects in production with a focus on quality * Customer focused - making customers more successful by taking on their most challenging problems motivates you * Have a strong sense of teamwork and able to make decisions which benefit the team and company * Identify root causes, and get beyond treating symptoms - motivated to dig deep and solve hard problems References Visible links 1. https://www.confluent.io/blog/cloud-native-kafka-kora-vldb-award/ 2. https://www.vldb.org/pvldb/vol16/p3822-povzner.pdf 3. https://www.confluent.io/events/current/2023/apache-kafkas-next-gen-rebalance-protocol-towards-more-stable-and-scalable/ 4. https://www.confluent.io/blog/cloud-native-multi-tenant-kafka-with-confluent-cloud/ 5. https://www.confluent.io/blog/from-apache-kafka-to-confluent-cloud-optimizing-for-speed-scale-storage/ 6. https://assets.confluent.io/m/2d7c883a8aa6a71d/original/20200501-WP-Benchmark\_Your\_Dedicated\_Apache\_Kafka\_Cluster\_on\_Confluent\_Cloud.pdf **Required technical and professional expertise** * BS, MS, or PhD in computer science or a related field, or equivalent work experience * 4+ years of relevant cloud infrastructure/cloud networking experience * Strong fundamentals in distributed systems design and development * Experience building and operating large-scale systems in the Cloud * Solid understanding of basic systems operations (disk, network, operating systems, etc) * A self starter with the ability to work effectively in teams * Proficiency in Java, Scala, Go or other statically typed languages **Preferred technical and professional experience** * In-depth familiarity with the JVM * A strong background in distributed storage systems or databases * Experience in the areas of resource management and QoS * Experience/knowledge with public clouds (AWS, Azure or GCP) * Interest in evangelism (giving talks at tech conferences, writing blog posts evangelizing Kafka) * Experience working on databases or data storage IBM is committed to creating a diverse environment and is proud to be an equal-opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, caste, genetics, pregnancy, disability, neurodivergence, age, veteran status, or other characteristics. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.