Lead Product Engineer – Observability (Enterprise App Monitoring) - Remote

At Allstate, great things happen when our people work together to protect families and their belongings from life’s uncertainties. And for more than 90 years, our innovative drive has kept us a step ahead of our customers’ evolving needs. From advocating for seat belts, air bags and graduated driving laws, to being an industry leader in pricing sophistication, telematics, and, more recently, device and identity protection. 

Job Description

Allstate is seeking a forward-thinking Lead Product Engineer to drive innovation in its AI-powered Observability platform. This role sits at the intersection of AI engineering, observability, and software development, with a strong focus on automation, intelligent insights, and self-healing systems across hybrid and multi-cloud environments.

You will lead the design and development of next-generation observability solutions powered by agentic AI, enabling proactive detection, diagnosis, and remediation of issues. This role emphasizes building scalable platforms, intelligent automation, and developer-centric tooling that enhance reliability, performance, and operational efficiency.

As a technical leader, you will shape product strategy, drive engineering excellence, and deliver AI-driven monitoring and automation capabilities that transform how applications and platforms are operated across the enterprise.

Key Responsibilities

AI & Intelligent Automation

• Design and build agentic AI solutions for observability, including autonomous agents capable of anomaly detection, root cause analysis, and automated remediation.
• Embed AI/ML capabilities into observability platforms to enable predictive insights, anomaly detection, and intelligent alerting.
• Develop AIOps frameworks that reduce noise, automate workflows, and improve incident response times.
• Leverage LLMs and AI orchestration frameworks to create self-service diagnostics and operational assistants.
• Architect and implement end-to-end observability solutions across logs, metrics, traces, and events.
• Enhance observability platforms (Datadog, Dynatrace, New Relic, AppDynamics, OTEL) with custom integrations, automation, and AI enhancements.
• Build real-time insights and health analytics to ensure system reliability, scalability, and performance.
• Lead development of scalable, reusable platform components and APIs for observability and automation.
• Build developer-first tools and frameworks that simplify instrumentation, monitoring, and diagnostics.
• Apply modern software engineering practices including TDD, CI/CD, microservices, and cloud-native design.
• Develop full-stack solutions (backend services, dashboards, automation tooling) using Java/Spring Boot, Python, Node.js, and React.
• Drive automation-first strategies to eliminate manual operations and improve efficiency.
• Build self-healing systems that automatically detect and remediate issues.
• Lead initiatives in infrastructure-as-code, Observability-as-code and automated deployment pipelines.
• Engineer highly resilient systems by focusing on performance, scalability, and fault tolerance.
• Define and evolve enterprise observability architecture, incorporating AI and automation as core principles.
• Collaborate with architecture, platform, and security teams to align solutions with enterprise standards.
• Champion standardization, reusable patterns, and platform scalability across the organization.
• Act as a technical leader and mentor, guiding teams in AI, observability, and engineering best practices.
• Partner with cross-functional teams to deliver integrated, high-impact platform solutions.
• Communicate technical concepts and strategies effectively to both technical and executive audiences.
• Foster a culture of innovation, automation, and continuous improvement.

Essential Skills

• 5+ years of engineering experience building software, platforms, or automation solutions, with a strong emphasis on observability and distributed systems.
• 4+ years of hands-on experience with observability platforms (Datadog, Dynatrace, OTEL).
• 3+ years of software development experience using Java (Spring Boot), Python, Node.js, or React.
• 2+ years of hands-on experience building agentic AI systems (autonomous agents, AI workflows, or AI-native applications).
• Proven ability to design and implement AI-driven automation and AIOps solutions.
• Experience integrating LLMs, orchestration frameworks, or AI pipelines into production systems.
• Strong experience with Kubernetes and cloud-native architectures.
• Hands-on experience in hybrid environments (on-prem + cloud) across Linux and Windows.
• Expertise in API development, event-driven systems, and microservices architecture.
• Familiarity with CI/CD pipelines, infrastructure-as-code, and DevOps practices.
• Strong problem-solving skills with a focus on automation and innovation.
• Excellent communication skills to articulate complex technical solutions.
• Passion for emerging technologies, particularly AI in observability and operations.

Supervisory Responsibilities

• This job does not have supervisory duties.

(#LI-NJ1)

Skills

Agentic AI, Agentic AI, Agile Methodology, AI Agents, AI Ops, Anomaly Detection, API Development, AppDynamics, Application Monitoring, Application Programming Interface (API), Automation, Automation Solutions, Back-End Development, Building Architecture, Business, Business Direction, Business Objectives, Business Operations, Business Processes, Business Software, C (Programming Language), Cloud Infrastructure, Cloud Native, Collaboration, Collaborative Development {+ 65 more}

Compensation

Compensation offered for this role is 100,000.00 - 170,500.00 annually and is based on experience and qualifications.

The candidate(s) offered this position will be required to submit to a background investigation.

 

Joining our team isn’t just a job — it’s an opportunity. One that takes your skills and pushes them to the next level. One that encourages you to challenge the status quo. One where you can shape the future of protection while supporting causes that mean the most to you. Joining our team means being part of something bigger – a winning team making a meaningful impact.

Allstate generally does not sponsor individuals for employment-based visas for this position.

Effective July 1, 2014, under Indiana House Enrolled Act (HEA) 1242, it is against public policy of the State of Indiana and a discriminatory practice for an employer to discriminate against a prospective employee on the basis of status as a veteran by refusing to employ an applicant on the basis that they are a veteran of the armed forces of the United States, a member of the Indiana National Guard or a member of a reserve component.

For jobs in San Francisco, please click “here” for information regarding the San Francisco Fair Chance Ordinance.

For jobs in Los Angeles, please click “here” for information regarding the Los Angeles Fair Chance Initiative for Hiring Ordinance.

To view the “EEO Know Your Rights” poster click “here”. This poster provides information concerning the laws and procedures for filing complaints of violations of the laws with the Office of Federal Contract Compliance Programs.

To view the FMLA poster, click “here”. This poster summarizing the major provisions of the Family and Medical Leave Act (FMLA) and telling employees how to file a complaint.

It is the Company’s policy to employ the best qualified individuals available for all jobs. Therefore, any discriminatory action taken on account of an employee’s ancestry, age, color, disability, genetic information, gender, gender identity, gender expression, sexual and reproductive health decision, marital status, medical condition, military or veteran status, national origin, race (include traits historically associated with race, including, but not limited to, hair texture and protective hairstyles), religion (including religious dress), sex, or sexual orientation that adversely affects an employee's terms or conditions of employment is prohibited. This policy applies to all aspects of the employment relationship, including, but not limited to, hiring, training, salary administration, promotion, job assignment, benefits, discipline, and separation of employment.

 

Allstate provides a comprehensive technology setup, including a laptop, monitors, headset, keyboard, and mouse. Employees eligible to work from home also receive a monthly connectivity reimbursement to help offset internet costs.

 

When working from home, you must have a dedicated, private workspace free from distractions, along with appropriate desk and seating. Reliable internet is required, with minimum speeds of 50 MB download and 5 MB upload.