data
Posted YesterdaySenior AI Engineer | US | Remote
at Grafana Labs
United StatesRemote
Responsibilities
- Own end-to-end development of multi-agent AI systems, from architecture and implementation through testing, deployment, and ongoing operation
- Build modular, composable agentic systems using orchestration frameworks (LangChain, CrewAI, Anthropic MCP, or similar) that operate 24/7 across teams
- Develop reusable agentic skills that agents invoke across interfaces (Slack, dashboards, internal apps, CLIs)
- Implement observability and feedback loops including logging, performance metrics, prompt iteration, model evaluation, and cost management
- Establish governance and compliance standards for AI workflows including access controls, audit trails, PII handling, and human-in-the-loop escalation paths
- Build MCP servers, APIs, CLIs, and microservices connecting AI models to business systems (BigQuery, Slack, CRMs, email, calendars, analytics tools)
- Architect data flows for retrieval-augmented generation (RAG), connecting LLMs to internal knowledge bases, customer data, and real-time business context
- Build serverless or containerized services (GCP Cloud Functions, Cloud Run) that scale with usage and integrate with Grafana's cloud infrastructure
- Design and deploy workflows using orchestration tools (n8n, Workato, or custom platforms) with CI/CD, testing, and production reliability standards
- Build systems designed for self-service with documentation, playbooks, and enablement materials that let partner teams operate independently
Requirements
- With Grafana Cloud's actually useful AI, organizations can see, understand, and act on all their disparate data to move at the speed of their ambitions.
- Today, more than 35 million users and 7,000+ customers – including Anthropic, Bloomberg, NVIDIA, Microsoft, and Salesforce – trust Grafana Labs to ensure reliability of their applications and systems, resolve incidents quickly, and optimize their telemetry to reduce noise and cost.
- Grafana Labs is seeking a Senior Engineer (AI & Automation) to own the AI agent infrastructure and automation platform that powers our Marketing Operations organization.
- You’ll build multi-agent architectures, LLM integrations, and backend services that connect AI models to internal and third-party data platforms.
- You’ll define the technical direction for the automation platform (data models, API contracts, shared libraries, reference architectures) and partner with Data Engineering, GTM Systems, and Field Operations to build scalable, self-service automation that eliminates manual work and drives operational efficiency.
- Agentic Systems & AI Infrastructure
- You'll have access to AI coding assistants (Claude Code, Gemini CLI, OpenAI Codex, and others of your choice within security guidelines).
- We encourage pragmatic AI-assisted development paired with strong code review and quality standards.
- experience with depth in backend development, systems integration, or data/analytics engineering 2+ years hands-on
- experience applying LLMs/AI to production workflows, not just prototypes
- Strong proficiency in Python and JavaScript/Node.js with Git-based workflows, code review practices, and testing discipline Hands-on
- experience with LLM frameworks and patterns including prompt engineering, RAG, function calling/tool use, structured output parsing, and evaluation •
- Deep familiarity with Google Cloud Platform, BigQuery, and serverless/containerized services (Cloud Functions, Cloud Run)
- Understanding of LLM failure modes and production mitigations including confidence thresholds, fallback logic, human escalation, and cost/latency management
- Proven ability to identify high-leverage problems, push back on low-impact requests, and deliver end-to-end with minimal direction
- Fluent with AI-assisted development tools (GitHub Copilot, Cursor, Claude Code). You use AI to build AI systems
- Experience with vector databases or retrieval pipelines (Pinecone, Weaviate, ChromaDB, Qdrant, pgvector)
- Familiarity with marketing or sales platforms (Salesforce, Customer.io, HubSpot, Marketo, Outreach) •
- Experience with frontend frameworks (React, Slack Block Kit) for building user-facing AI tool interfaces
- Observability tooling for AI systems (LangSmith, Weights & Biases, custom evaluation frameworks) •
- Experience with workflow orchestration platforms (n8n, Temporal, Prefect, Airflow)
- Familiarity with Model Context Protocol (MCP) or similar standards for connecting AI systems to data sources
- Active in open-source communities. Grafana is built on OSS and we value engineers who share that DNA
- Grafana Labs may utilize AI tools in its recruitment process to assist in matching information provided in CVs to job postings.
Experience
- 8+ years of software engineering
Benefits
- Clear technical communicator who can explain complex systems in simple terms to both engineers and business stakeholders Bonus Points •
- In the United States, the base compensation range for this role is USD $154,445 - USD $185,334.
- Actual compensation may vary based on level, experience, and skillset as assessed throughout the interview process.
- *Compensation ranges are country specific. If you are applying for this role from a different location than listed above, your recruiter will discuss your specific market’s defined pay range &
- Balance is Key - We operate a global annual leave policy of 30 days per annum. 3 days of your annual leave entitlement are reserved for Grafana Shutdown Days to allow the team to really disconnect. *We will comply with local legislation where applicable.
Contact
- Learn more at grafana.com and follow us on LinkedIn and X .
Additional details
- Grafana Labs, the company behind the open observability cloud, is founded on the principles of open source, open standards, open ecosystems, and open culture.
- Grafana Cloud, our fully managed observability platform, is flexible and built for scale.
- We are a 100% remote company with 1,600+ team members across 40+ countries, and we’re backed by leading investors including Lightspeed Venture Partners, Sequoia Capital, GIC, Coatue, J.P.
- We’re scaling fast and staying true to what makes us different: an open-source legacy, a global collaborative culture, and a passion for meaningful work.
- Our team thrives in an innovation-driven environment where transparency, autonomy, and trust fuel everything we do.
- You may not meet every requirement, and that’s okay. If this role excites you, we’d love you to raise your hand for what could be a truly career-defining opportunity.
- This is a remote opportunity and we are looking for candidates from the U.S. The Opportunity
- You’ll ship production systems that teams depend on daily.
- This is a high-autonomy role where you own the technical direction.
- You’ll identify the highest-leverage problems across Marketing, RevOps, and SDR teams, design the solutions, and ship them.