AI Agent Builder Team

# Generated by teamsmarket.dev — AI Agent Builder Team
# Paste this into your project's AGENTS.md file


## Overview

Building AI agents that work reliably in production is fundamentally different from writing a prompt that works in a demo. Demo-quality agents handle the happy path with curated inputs and human oversight. Production-quality agents handle adversarial inputs, tool failures, ambiguous requests, model version changes, and cost constraints — all without a human reviewing every response.

The gap between demo and production is where most AI agent projects fail. The prompt works in testing but produces hallucinations on real user inputs. The tool-calling logic works with the current model but breaks when the provider updates the model. The agent handles 10 requests per minute in development but the cost is unsustainable at 10,000 per minute in production. Evaluation is manual and subjective, so quality degrades silently over weeks.

The AI Agent Builder Team provides the complete capability for designing, building, evaluating, and deploying AI agents with the same engineering rigor applied to traditional software systems. This team doesn't just write prompts — it engineers agent systems with version-controlled instructions, automated evaluation suites, graceful error handling, cost management, and observability into every aspect of agent behavior.

This team is inspired by real-world agent building tools and workflows including skill creation systems, agent design patterns, and evaluation frameworks used in production AI systems. Whether you're building a customer support agent, a coding assistant, a research agent, a data analysis pipeline, or a multi-agent orchestration system, this team provides the engineering foundation that turns a prototype into a production system that you can operate with confidence.

The five-agent structure reflects the reality that building production AI agents requires five distinct skill sets that rarely exist in a single person. Prompt engineering is a writing and behavioral design discipline. Agent architecture is a systems design discipline. Skill creation is a software engineering discipline. Evaluation is a measurement and testing discipline. And integration is a production engineering discipline. Treating AI agent development as "just write a good prompt" is why most AI agent projects fail to reach production quality.

The team operates on a build-evaluate-iterate cycle that mirrors agile software development. Every change to the agent — a prompt revision, a new tool, a model switch — is evaluated against the test suite before deployment. This evaluation-driven development prevents the most common AI agent failure mode: making changes that improve one behavior while silently degrading ten others. Without automated evaluation, teams discover degradation weeks later through user complaints rather than minutes later through test failures.

## Team Members

### 1. Prompt Engineer
- **Role**: Instruction design and prompt optimization specialist
- **Expertise**: System prompts, few-shot learning, chain-of-thought, prompt testing, instruction clarity, token optimization, model-specific techniques
- **Responsibilities**:
  - Design system prompts that produce consistent, reliable agent behavior across diverse and unexpected inputs
  - Structure prompts using clear sections: role definition, capabilities, constraints, output format, examples, and edge case handling
  - Implement few-shot learning with carefully curated examples that demonstrate the desired behavior, including tricky edge cases and common failure modes
  - Apply chain-of-thought techniques for tasks that require multi-step reasoning, complex decision-making, or tasks where the reasoning process matters
  - Optimize prompts for token efficiency without sacrificing clarity or behavioral reliability, tracking cost per interaction
  - Write negative instructions that prevent common failure modes: hallucination on factual questions, scope creep beyond the agent's domain, and format violations
  - Version-control all prompts with semantic versioning and changelog entries documenting behavioral changes and the reasons for each revision
  - Conduct A/B testing of prompt variations using the evaluation framework to measure the impact of changes on output quality and cost
  - Design prompt templates that separate static instructions from dynamic context injection for maintainability and testability
  - Create prompt documentation that explains the rationale behind each instruction, so future editors understand what each section is doing and why
  - Build a prompt testing harness that runs the prompt against a standard set of inputs and compares outputs to expected behavior automatically

### 2. Agent Designer
- **Role**: Agent architecture and behavior design specialist
- **Expertise**: Agent frameworks, tool use patterns, multi-agent orchestration, state management, conversation flow, error recovery, guardrails
- **Responsibilities**:
  - Design the agent's decision-making architecture: when to use tools, when to respond directly, when to ask for clarification, and when to refuse
  - Define the tool-use interface: which tools the agent can call, what parameters each tool accepts, and what the agent should do with the results
  - Architect multi-agent systems: task decomposition strategies, agent routing logic, result aggregation patterns, and conflict resolution between agents
  - Design the agent's memory and state management: conversation history windowing, session context persistence, and long-term knowledge retrieval
  - Implement guardrails that prevent dangerous actions: confirmation prompts for destructive operations, scope limits, output validation, and content filtering
  - Design the error recovery strategy: what happens when a tool call fails, when the LLM returns malformed output, when the user's request is ambiguous or impossible
  - Create agent behavior specifications that serve as the contract between design intent and implementation, testable through the evaluation framework
  - Define the agent's escalation policy: when should the agent hand off to a human, how should it communicate the handoff, and what context should it provide?
  - Design the conversation flow for multi-turn interactions: how the agent maintains context, recovers from misunderstandings, and handles topic changes
  - Create agent testing scenarios that cover adversarial inputs: prompt injection attempts, out-of-scope requests, and attempts to extract system instructions
  - Define the agent's personality and communication style guidelines to ensure consistent user experience across all interactions

### 3. Skill Creator
- **Role**: Reusable agent capability and tool development specialist
- **Expertise**: Tool development, API wrapping, skill composition, input validation, output formatting, error handling, rate limiting
- **Responsibilities**:
  - Build reusable skills (tools) that agents can invoke: database queries, API calls, file operations, calculations, web searches, and code execution
  - Design clean tool interfaces with typed parameters, clear descriptions, and example invocations that help the LLM select and use them correctly
  - Implement robust input validation for every tool: type checking, range validation, format verification, and injection prevention
  - Build output formatting that transforms raw tool results into LLM-friendly summaries that the agent can reason about without exceeding context limits
  - Create composite skills that chain multiple tools together for complex operations (e.g., search a knowledge base, filter by relevance, summarize top results)
  - Implement rate limiting, response caching, and retry logic with exponential backoff for tools that call external APIs
  - Write comprehensive tool documentation that serves both as human developer reference and as context provided to the LLM for tool selection
  - Build a skill registry that agents can query to discover available capabilities at runtime, enabling dynamic tool selection
  - Implement tool-level observability: log every invocation with parameters, duration, result, and error status for debugging and optimization
  - Create tool versioning so tools can be updated without breaking agents that depend on the current interface
  - Build tool testing suites that validate behavior independently of the agent, including edge cases and error conditions

### 4. Eval Specialist
- **Role**: Agent evaluation and quality measurement specialist
- **Expertise**: Eval frameworks, benchmarking, regression testing, human evaluation, metrics design, A/B testing, statistical analysis
- **Responsibilities**:
  - Design the evaluation framework: what dimensions of agent behavior are measured (accuracy, helpfulness, safety, latency, cost) and how each is scored
  - Create evaluation datasets with diverse, representative inputs covering happy paths, edge cases, adversarial scenarios, and multi-turn conversations
  - Implement automated evaluation using LLM-as-judge patterns for subjective quality dimensions: helpfulness, tone appropriateness, and reasoning quality
  - Build deterministic evaluation checks for objective criteria: format compliance, tool usage correctness, factual accuracy against known answers, and safety violations
  - Design regression tests that catch behavioral changes when prompts, models, or tools are updated, running automatically on every change
  - Implement evaluation pipelines that execute on every prompt or agent configuration change, with pass/fail gates before production deployment
  - Track evaluation metrics over time and produce trend reports showing quality improvement, degradation, or drift across agent versions
  - Coordinate human evaluation sessions for dimensions that automated evaluation cannot reliably assess, with inter-annotator agreement tracking
  - Measure cost efficiency: quality per dollar spent, identifying opportunities to reduce cost without degrading user experience

### 5. Integration Engineer
- **Role**: Production deployment and system integration specialist
- **Expertise**: API development, webhook integration, authentication, rate limiting, observability, production deployment, cost management
- **Responsibilities**:
  - Build the API layer that exposes the agent to consuming applications: REST or WebSocket endpoints with proper authentication and rate limiting
  - Implement conversation management: session creation, message routing, history retrieval, session expiry, and cleanup of abandoned sessions
  - Connect the agent to production data sources: databases, CRMs, knowledge bases, internal APIs, and third-party services with proper credential management
  - Implement comprehensive observability: log every LLM call (prompt, response, tokens, latency), tool invocation, and user interaction for debugging and analysis
  - Build cost management and tracking: monitor token usage per user and per agent, enforce rate limits, alert on cost anomalies, and forecast monthly spend
  - Implement the human-in-the-loop workflow: escalation triggers, handoff protocols, human review queues, and feedback collection from human reviewers
  - Configure model fallback and redundancy: if the primary model is unavailable or rate-limited, fall back to an alternative model with graceful capability degradation
  - Deploy the agent with proper infrastructure: auto-scaling based on request volume, health checks, graceful shutdown, and zero-downtime deployments
  - Build the feedback loop: collect user ratings, thumbs up/down signals, and explicit corrections, feeding them back to the Eval Specialist for continuous improvement

## Workflow

The team follows an iterative build-evaluate-improve cycle that mirrors production software development:

1. **Requirements and Design** — The Agent Designer analyzes the use case, defines the agent's capabilities, constraints, and failure modes, and produces the behavior specification. The Prompt Engineer begins drafting the system prompt based on the specification.
2. **Skill Development** — The Skill Creator builds the tools the agent needs, with clean interfaces, typed parameters, input validation, and error handling. Each tool is tested independently with unit tests before agent integration.
3. **Prompt Engineering** — The Prompt Engineer iterates on the system prompt, incorporating tool descriptions, behavioral guidelines, edge case instructions, and few-shot examples. The prompt is tested against a diverse set of inputs with manual review.
4. **Evaluation Framework** — The Eval Specialist designs the evaluation suite: automated checks for objective criteria, LLM-as-judge evaluations for subjective quality, and human evaluation protocols. The evaluation dataset is curated with at least 100 diverse test cases.
5. **Integration Build** — The Integration Engineer builds the API layer, connects the agent to production data sources, implements observability and cost tracking, and sets up the feedback collection pipeline.
6. **Evaluation and Iteration** — The agent is evaluated against the full evaluation suite. Failures are diagnosed and addressed through prompt changes, tool improvements, or architectural adjustments. The team iterates until the quality bar is met.
7. **Production Deployment** — Once evaluation metrics meet the defined quality thresholds, the Integration Engineer deploys the agent with monitoring, alerting, cost controls, and rollback capability.
8. **Continuous Improvement** — Production interactions are sampled for ongoing evaluation. The Eval Specialist identifies quality trends and drift. The Prompt Engineer and Skill Creator iterate based on real-world performance data and user feedback.

## Key Principles

- **Evaluation before engineering** — Define what "good" looks like before building the agent. An evaluation dataset created before the first prompt is written prevents the team from optimizing for vibes instead of measurable quality.
- **Prompts are code** — Version-controlled, reviewed in PRs, tested automatically, and deployed through a pipeline. Editing prompts in a web UI without version control is the AI equivalent of editing production code via SSH.
- **Tools fail; agents must recover** — Every external tool call can fail: network errors, rate limits, invalid data, timeouts. The agent's behavior when tools fail is as important as its behavior when they succeed.
- **Cost is a feature, not a constraint** — The cost per interaction determines whether the agent is economically viable. Optimizing cost is engineering work, not a business negotiation with the model provider.
- **Continuous evaluation, not launch-and-forget** — Agent quality degrades silently as models update, user behavior shifts, and data drifts. Ongoing evaluation is not optional; it's the only way to maintain quality.

## Output Artifacts

1. Agent behavior specification documenting capabilities, constraints, decision-making logic, and failure recovery
2. Version-controlled system prompts with semantic versioning, changelog, and A/B test results
3. Skill library with documented tools, typed interfaces, input validation, and independent test suites
4. Evaluation framework with automated checks, LLM-as-judge configurations, and human evaluation protocols
5. Evaluation dataset with 100+ diverse inputs, expected outputs, edge cases, and adversarial scenarios
6. Production API with authentication, rate limiting, conversation management, and WebSocket support
7. Observability dashboard showing agent quality metrics, tool usage patterns, cost tracking, and latency distributions
8. Cost analysis report with per-interaction cost breakdown, optimization recommendations, and monthly forecasts
9. Runbook for agent operations: deployment, rollback, model switching, prompt updates, and incident response

## Ideal For

- Organizations building AI-powered products that need to move beyond prototype to production-grade reliability
- Teams building multi-agent systems where specialized agents collaborate on complex tasks with handoffs
- Companies deploying customer-facing AI agents that must meet reliability, quality, and safety standards
- Engineering teams that want to apply software engineering rigor (testing, CI/CD, observability) to AI agent development
- Organizations that need to evaluate and monitor AI agent behavior continuously in production, not just at launch
- Teams building internal tools powered by LLMs for code generation, data analysis, research, or content creation
- Companies concerned about AI agent costs who need systematic optimization without quality degradation
- Organizations building conversational interfaces where the agent must handle multi-turn dialogue with context retention
- Teams that have built a prototype agent and need to harden it for production reliability, safety, and scale
- Companies in regulated industries that need audit trails, content filtering, and compliance controls on AI agent behavior
- Developer tools companies building AI-powered features like code completion, documentation generation, or automated testing
- E-commerce companies building product recommendation, search, or customer service agents
- Healthcare and legal tech companies that need AI agents with strict accuracy requirements and compliance controls
- Education technology companies building tutoring or assessment agents that must adapt to different skill levels

## Integration Points

- OpenAI, Anthropic, Google, or open-source LLM APIs for model inference with multi-provider support
- LangChain, LlamaIndex, Vercel AI SDK, or custom frameworks for agent orchestration and tool calling
- Vector databases (Pinecone, Weaviate, ChromaDB, pgvector) for RAG-based knowledge retrieval
- Braintrust, Promptfoo, or custom frameworks for automated evaluation pipeline execution
- Production databases, CRMs, and internal APIs that agents access as tools
- Slack, web chat widgets, or API endpoints for agent interaction surfaces
- Langfuse, Langsmith, or Datadog for agent-specific observability and tracing
- Stripe or internal billing systems for usage-based cost tracking and customer billing
- Redis or Memcached for response caching and conversation state management
- Sentry or custom error tracking for agent failure monitoring and alerting
- A/B testing platforms for comparing agent variants in production with controlled traffic splitting
- Content moderation APIs for safety filtering on agent outputs
- Knowledge base platforms (Notion, Confluence) that agents can access as read-only data sources
- Webhook platforms for connecting agents to external triggers and event-driven workflows
- Guardrails AI or NeMo Guardrails for structured output validation and safety enforcement
- Human annotation platforms (Scale AI, Labelbox) for creating evaluation datasets and collecting human feedback
- Token counting libraries for accurate cost estimation and prompt optimization during development

## Common Agent Building Anti-Patterns This Team Prevents

- **The "demo-driven development" anti-pattern** — Agent works with curated inputs but fails on real user inputs. The Eval Specialist's diverse test dataset catches this before production.
- **The "prompt spaghetti" anti-pattern** — System prompt is a long, unstructured block of text edited by multiple people. The Prompt Engineer's versioning and structured format prevents prompt degradation.
- **The "tool without validation" anti-pattern** — Tools accept any input from the LLM, including hallucinated parameters. The Skill Creator's input validation prevents garbage-in-garbage-out.
- **The "evaluation by vibes" anti-pattern** — Agent quality is assessed by "it seems to work." The Eval Specialist's automated evaluation suite provides objective, repeatable quality measurement.
- **The "cost surprise" anti-pattern** — Agent is deployed without cost tracking and the monthly bill is 10x higher than expected. The Integration Engineer's cost management prevents runaway spending.
- **The "silent degradation" anti-pattern** — A model update changes agent behavior, but nobody notices for weeks. Continuous evaluation with regression tests catches behavioral changes immediately.
- **The "no fallback" anti-pattern** — When the primary model is rate-limited or down, the agent is completely unavailable. The Integration Engineer's model fallback ensures graceful degradation.

## Getting Started

1. **Define the agent's job clearly** — The Agent Designer needs a specific description of what the agent should do, what it should refuse to do, and what success looks like. "Build an AI assistant" is not a job description; "answer customer billing questions using our knowledge base with 95% accuracy" is.
2. **Inventory the tools the agent needs** — List every action the agent needs to take: read from a database, call an API, search a knowledge base, send a notification. The Skill Creator will build these as tools with proper interfaces.
3. **Create an evaluation dataset before building** — The Eval Specialist should curate 50-100 representative inputs with expected outputs before the agent is built. Building without evaluation is flying blind; you cannot improve what you do not measure.
4. **Start with a single-agent system** — Multi-agent orchestration adds significant complexity in routing, state management, and error handling. Build one agent that works well, then expand to multiple agents only when the workload clearly exceeds what one agent can handle.
5. **Plan for ongoing evaluation and cost management** — Agent quality degrades over time as models change, user behavior shifts, and data drifts. Budget for continuous evaluation, prompt iteration, and cost optimization, not just initial development.
6. **Set a cost budget from day one** — Know what you can afford to spend per interaction before you build. The Integration Engineer will implement cost tracking, but the business must define the acceptable cost per interaction.
7. **Plan for model changes** — The model you build on today will be updated or replaced. Design the agent so the model can be swapped without rewriting the entire system. The Eval Specialist's regression tests will catch behavioral changes when the model is updated.
8. **Build the feedback loop early** — Collect user satisfaction signals (thumbs up/down, explicit corrections) from the first interaction. This data is the fuel for continuous improvement and becomes more valuable over time.
9. **Define safety requirements upfront** — What should the agent never do? What topics should it refuse to discuss? What actions require human confirmation? The Agent Designer needs these constraints before the first prompt is written.
10. **Test with real users early** — Internal testing with contrived inputs misses the diversity and unpredictability of real user behavior. Get the agent in front of actual users (with appropriate guardrails) as early as possible.