This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Data Engineer – Databricks based in the United States.
This role sits at the core of a modern, AI-driven data platform powering enterprise revenue intelligence solutions. You will be responsible for building, optimizing, and scaling high-performance Databricks pipelines that unify ERP and CRM data into actionable insights used across product, ML, and analytics teams. The position combines hands-on engineering with architectural ownership, ensuring data systems are reliable, cost-efficient, and production-ready at scale. You will work across multi-cloud environments (Azure and AWS), supporting both legacy modernization and new customer onboarding. The environment is highly collaborative, involving close partnership with product managers, ML engineers, and enterprise architects. This is a high-impact role where data quality, performance, and governance directly influence customer outcomes and platform intelligence. You will also contribute to operational excellence through monitoring, CI/CD practices, and incident response in a global, SLA-driven setting.
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Data Engineer – Databricks based in the United States.
This role sits at the core of a modern, AI-driven data platform powering enterprise revenue intelligence solutions. You will be responsible for building, optimizing, and scaling high-performance Databricks pipelines that unify ERP and CRM data into actionable insights used across product, ML, and analytics teams. The position combines hands-on engineering with architectural ownership, ensuring data systems are reliable, cost-efficient, and production-ready at scale. You will work across multi-cloud environments (Azure and AWS), supporting both legacy modernization and new customer onboarding. The environment is highly collaborative, involving close partnership with product managers, ML engineers, and enterprise architects. This is a high-impact role where data quality, performance, and governance directly influence customer outcomes and platform intelligence. You will also contribute to operational excellence through monitoring, CI/CD practices, and incident response in a global, SLA-driven setting.
Accountabilities:
- Own the design, development, and production support of Databricks-based data pipelines powering ERP and CRM data ingestion, transformation, and delivery at scale.
- Monitor and optimize production data workflows, ensuring SLA compliance, system reliability, and rapid incident resolution across distributed environments.
- Lead performance tuning and cost optimization initiatives across Databricks workloads, improving throughput and reducing compute costs.
- Drive migration of legacy ETL/ELT systems to modern Databricks architecture, ensuring seamless and automated transitions.
- Architect and maintain Delta Lake structures, including schema design, partitioning strategies, and incremental processing frameworks.
- Implement robust data quality, observability, and monitoring systems to ensure integrity across pipelines and ML inputs.
- Support multi-tenant data onboarding, ensuring secure, isolated, and validated pipelines for enterprise customers.
- Collaborate cross-functionally with ML, product, and enterprise architecture teams to align data infrastructure with platform needs.
- Maintain operational readiness through documentation, CI/CD practices, runbooks, and participation in on-call support rotations.
- 4+ years of experience in data engineering with strong exposure to production-grade data pipelines.
- At least 2+ years of hands-on experience with Databricks and/or Apache Spark ecosystems across Azure and/or AWS.
- Strong proficiency in Python, PySpark, and SQL for building scalable data transformations and workflows.
- Deep experience with Delta Lake, including schema evolution, ACID transactions, and incremental/streaming processing.
- Proven expertise in performance tuning and optimizing Databricks workloads in production environments.
- Solid understanding of PostgreSQL and relational database design, including query optimization and integration patterns.
- Experience working with legacy ETL systems (e.g., SSIS, Informatica, or custom pipelines) and modernization efforts.
- Familiarity with multi-tenant architectures, data isolation strategies, and enterprise-grade data governance.
- Strong collaboration skills with cross-functional teams including data science, product, and infrastructure.
- Knowledge of CI/CD practices, version control, and automated testing for data engineering workflows.
- Competitive compensation package with performance-based incentives
- Comprehensive healthcare coverage for employees and families
- 401(k) retirement plan with company matching
- Unlimited paid time off and paid parental leave
- Health and wellness reimbursement programs
- Financial planning and legal support services
- Employee discounts on travel, entertainment, and wellness programs
- Career development, training, and educational resources
- Employee referral bonus program
- Flexible, remote-friendly work environment with a strong focus on work-life balance