Principal AI Engineer

CareersNew York, NYPrincipal AI Engineer-20.5.26LoactionNew York, NYJob TypeFull-timeExperience5 - 10Remote work policyOn-siteVisa sponsorshipNoPreferred time zoneETAbout the roleThis role owns the design and implementation of AI agents that power intelligent workflows across our platform, from reasoning and automation to evidence-driven RCA, customer-facing copilots, internal operations, and product intelligence.You’re a strong Python engineer who understands modern agentic AI architecture and can turn ambiguous product needs into production-grade systems. You are comfortable building agents that use tools, retrieve context, reason over structured and unstructured data, execute workflows, and integrate deeply with backend services.This is not a prompt-only role. We’re looking for someone who can architect, build, test, deploy, and operate AI-powered features end to end.What you’ll buildAgentic AI systemsProduction-grade AI agents for different product and internal applications: site-level connectivity analysisevidence-first root cause analysis workflowscustomer-facing assistant and copilot experiencesinternal automation for operations, support, and engineering workflowsreasoning workflows over telemetry, incidents, site data, and knowledge sourcesModern agentic architecture: tool/function callingplanning and execution loopsstate and memory managementretrieval-augmented generationstructured outputsworkflow orchestrationmulti-agent patterns where appropriateguardrails, permissions, and safety constraintsobservability and evaluation frameworksAI-native product primitives: agent task modelscontext assembly pipelinestool registriesprompt/version managementhuman-in-the-loop review flowsagent execution traceseval datasets and regression testingPython backend + product engineeringBackend services and APIs in Python: FastAPI/Django/Flask-style servicesworkers and async jobsintegrations with internal systems and external APIsdata models supporting AI workflowsevent-driven and workflow-driven architecturesRetrieval and knowledge systems: vector searchhybrid searchdocument ingestionchunking and indexing strategiesmetadata filteringgrounding and citation workflowsProduction AI infrastructure: LLM provider integrationmodel routingcost and latency optimizationCachingrate limits and retriesmonitoring and debuggingfailure handling and fallback behaviorFeature delivery end to end: product scopingArchitectureImplementationTestingDeploymentObservabilityiteration based on user feedbackResponsibilitiesArchitect and implement production-grade AI agents that solve real business and product problems.Build agent workflows that can reason over Eino’s data, tools, telemetry, site models, incidents, and knowledge sources.Own agent architecture patterns across planning, memory, retrieval, tool execution, structured outputs, evals, and observability.Build and operate core Python backend services that support AI-powered product features.Work closely with product, engineering, and leadership to identify high-value agentic AI use cases.Move quickly from prototype to production while maintaining reliability, security, and maintainability.Establish testing and evaluation discipline for AI systems: unit/integration testsprompt and workflow regression testsagent evalsgolden datasetstrace reviewfailure analysisDrive practical AI engineering standards: correctness over demosgrounded outputsmeasurable qualityclear contracts between agents, tools, and backend servicesRequired qualificationsStrong experience building production Python systems, including services, APIs, workers, and backend infrastructure.Hands-on experience building AI agents, LLM-powered applications, RAG systems, workflow automation, or tool-using AI systems.Deep familiarity with modern agentic AI architectures, including tool/function calling, planning and execution loops, state and memory management, retrieval, structured outputs, guardrails, observability, and evaluation frameworks.Ability to...