Minion Templates: Backend Storage & CRUD API Feature

by Admin 53 views
Minion Templates: Backend Storage & CRUD API Feature

Hey everyone! Today, let's dive deep into an exciting new feature: Minion Templates. This feature focuses on backend storage and CRUD (Create, Read, Update, Delete) API functionality. If you've ever felt the pain of repeatedly configuring minions with the same permissions, this one's for you! We'll explore the problem it solves, the user impact, acceptance criteria, test scenarios, implementation details, and more.

Problem: The Tedious Task of Minion Configuration

Let's get straight to the issue. Currently, every time you create a minion, you have to manually configure its permission modes and allowed tools. For common minion types – think "Code Expert," "Web Researcher," or "Project Manager" – this becomes a repetitive and frankly, boring task. Not only is it time-consuming, but it's also prone to errors. Imagine accidentally giving a "Web Researcher" edit access – yikes! This is where minion templates come in to save the day.

Current behavior forces users to specify permission_mode and allowed_tools for each minion individually. There's no way to save common configurations, leading to repetitive work for frequently used minion types. This manual process significantly increases the risk of misconfiguration or inconsistency across minions, hindering efficient workflow and potentially compromising security. The lack of a centralized management system for minion configurations makes it difficult to maintain a standardized environment. This can lead to operational inefficiencies and higher administrative overhead as the number of minions grows.

Expected behavior, on the other hand, is a dream come true. Users will be able to create named templates with pre-defined permission configurations. These templates will be stored persistently, making them reusable across sessions. Think of it as saving your favorite minion setups! The system will support full CRUD operations on these templates, meaning you can list, create, update, and delete them as needed. Each template will include permission_mode and allowed_tools, with optional fields for default_role and default_system_prompt. This comprehensive approach aims to streamline minion creation and management, ensuring consistency and reducing errors.

User Impact: Streamlining Minion Creation

Think about it from a user's perspective. As a user who frequently creates similar minions, I need to define reusable permission templates so that I can quickly spawn minions with consistent configurations and avoid repetitive manual setup. This boils down to efficiency and consistency. No more second-guessing if you've configured everything correctly. Just select a template, and boom – you have a minion ready to go!

Let's look at some example templates to illustrate this:

Template: "Code Expert"
- permission_mode: "acceptEdits"
- allowed_tools: ["bash", "edit", "read", "write", "glob", "grep"]
- default_role: "Code review and refactoring specialist"
- description: "Can freely edit code and run tests"

Template: "Web Researcher"
- permission_mode: "default"
- allowed_tools: ["web_fetch", "web_search", "read"]
- default_role: "Documentation and research specialist"
- description: "Can search web and read docs, requires permission for edits"

Template: "Project Manager"
- permission_mode: "acceptEdits"
- allowed_tools: ["bash", "edit", "read", "send_comm", "spawn_minion", "create_channel"]
- default_role: "Overseer for delegation and coordination"
- description: "Can manage project structure and delegate tasks"

Template: "Safe Sandbox"
- permission_mode: "default"
- allowed_tools: ["read"]
- default_role: "Read-only analyst"
- description: "Highly restricted for safe experimentation"

These templates showcase the flexibility and power of the minion template feature. You can define templates for various roles, each with specific permissions and tools. This not only saves time but also promotes a more organized and secure environment.

Acceptance Criteria: What Makes a Good Template?

So, what are the key components of a minion template? Let's break down the acceptance criteria:

Template Data Model: This is the blueprint for our templates. It includes:

  • [ ] Template has unique ID (UUID): Ensures each template is uniquely identifiable.
  • [ ] Template has unique name (user-facing identifier): Allows users to easily recognize templates.
  • [ ] Template includes permission_mode (required): Defines the level of access the minion has.
  • [ ] Template includes allowed_tools list (optional): Specifies which tools the minion can use.
  • [ ] Template includes default_role (optional): Sets a default role for the minion.
  • [ ] Template includes default_system_prompt (optional): Provides a default system prompt for the minion.
  • [ ] Template includes description (optional, for documentation): Offers a way to describe the template's purpose.
  • [ ] Template includes created_at timestamp: Tracks when the template was created.
  • [ ] Template includes updated_at timestamp: Tracks when the template was last updated.

Template Storage: Where do these templates live?

  • [ ] Templates stored in data/templates/ directory: A dedicated directory for templates.
  • [ ] Each template saved as {template_id}.json: Templates are stored as JSON files for easy readability and manipulation.
  • [ ] Template persistence survives server restarts: Templates are not lost when the server restarts.
  • [ ] Templates loaded on server startup: Templates are automatically loaded when the server starts.

REST API Endpoints: How do we interact with the templates?

  • [ ] GET /api/templates - List all templates: Retrieves a list of all available templates.
  • [ ] GET /api/templates/{id} - Get specific template: Retrieves a specific template by its ID.
  • [ ] POST /api/templates - Create new template: Creates a new template.
  • [ ] PUT /api/templates/{id} - Update existing template: Updates an existing template.
  • [ ] DELETE /api/templates/{id} - Delete template: Deletes a template.

Validation: Ensuring data integrity.

  • [ ] Template name must be unique: Prevents naming conflicts.
  • [ ] Template name must be non-empty: A template must have a name.
  • [ ] permission_mode must be valid (default, acceptEdits, plan, bypassPermissions): Ensures a valid permission mode is selected.
  • [ ] allowed_tools must be valid tool names (or empty): Prevents the use of invalid tool names.
  • [ ] Cannot delete template if in use (optional safety feature): Prevents accidental deletion of templates in use.

Default Templates: Getting started easily.

  • [ ] System ships with example templates (Code Expert, Web Researcher, etc.): Provides pre-built templates for common use cases.
  • [ ] Default templates loaded on first run: Default templates are available from the start.
  • [ ] User can modify/delete default templates: Users have the flexibility to customize or remove default templates.

These acceptance criteria ensure that the minion template feature is robust, user-friendly, and meets the needs of various use cases.

Test Scenarios: Putting Templates to the Test

To ensure the quality of the minion template feature, we've defined several test scenarios using Gherkin syntax. These scenarios cover various aspects of template creation, retrieval, updating, and deletion.

Scenario: Create new template
  Given I am an authenticated user
  When I create template with:
    - name: "Code Expert"
    - permission_mode: "acceptEdits"
    - allowed_tools: ["bash", "edit", "read"]
  Then template is saved to data/templates/{id}.json
  And template appears in list of all templates
  And I receive template ID in response

Scenario: List all templates
  Given system has 3 templates
  When I request GET /api/templates
  Then I receive array of 3 templates
  And each template includes id, name, permission_mode

Scenario: Get specific template
  Given template "Code Expert" exists with ID "abc-123"
  When I request GET /api/templates/abc-123
  Then I receive full template details
  And response includes permission_mode and allowed_tools

Scenario: Update existing template
  Given template "Code Expert" exists
  When I update it to add "write" to allowed_tools
  Then template is updated in storage
  And updated_at timestamp is refreshed
  And GET request shows updated tools list

Scenario: Delete template
  Given template "Old Template" exists
  When I request DELETE /api/templates/{id}
  Then template file is deleted from data/templates/
  And template no longer appears in list

Scenario: Duplicate template name rejected
  Given template with name "Code Expert" exists
  When I try to create another template with name "Code Expert"
  Then I receive error: "Template name already exists"
  And no new template is created

Scenario: Invalid permission mode rejected
  Given I create template with permission_mode="invalid"
  When I submit the template
  Then I receive error: "Invalid permission mode"
  And template is not created

These scenarios cover the core functionalities of the minion template feature and ensure that it behaves as expected under various conditions.

Implementation Notes: Diving into the Code

Let's peek under the hood and see how this feature is implemented. We'll focus on the key components: the Template Data Model, the Template Manager, and the REST API Endpoints.

Template Data Model (src/models/minion_template.py - NEW FILE):

from dataclasses import dataclass, asdict
from datetime import datetime, timezone
from typing import List, Optional, Dict, Any
import json
from pathlib import Path

@dataclass
class MinionTemplate:
    """
    Reusable configuration template for minion creation.

    Stores permission mode, allowed tools, and default settings
    that can be applied when creating new minions.
    """
    template_id: str
    name: str
    permission_mode: str  # default, acceptEdits, plan, bypassPermissions
    allowed_tools: Optional[List[str]] = None
    default_role: Optional[str] = None
    default_system_prompt: Optional[str] = None
    description: Optional[str] = None
    created_at: Optional[datetime] = None
    updated_at: Optional[datetime] = None

    def __post_init__(self):
        if self.created_at is None:
            self.created_at = datetime.now(timezone.utc)
        if self.updated_at is None:
            self.updated_at = datetime.now(timezone.utc)
        if self.allowed_tools is None:
            self.allowed_tools = []

    def to_dict(self) -> Dict[str, Any]:
        """Convert to dictionary for JSON serialization."""
        data = asdict(self)
        data['created_at'] = self.created_at.isoformat()
        data['updated_at'] = self.updated_at.isoformat()
        return data

    @classmethod
    def from_dict(cls, data: Dict[str, Any]) -> 'MinionTemplate':
        """Create from dictionary loaded from JSON."""
        data['created_at'] = datetime.fromisoformat(data['created_at'])
        data['updated_at'] = datetime.fromisoformat(data['updated_at'])
        return cls(**data)

This data model uses a dataclass to define the structure of a minion template. It includes fields for template_id, name, permission_mode, allowed_tools, default_role, default_system_prompt, description, created_at, and updated_at. The to_dict and from_dict methods handle serialization and deserialization of the template data.

Template Manager (src/template_manager.py - NEW FILE):

import uuid
import json
from pathlib import Path
from typing import List, Optional, Dict
from src.models.minion_template import MinionTemplate

class TemplateManager:
    """Manages minion templates storage and retrieval."""

    def __init__(self, data_dir: Path):
        self.templates_dir = data_dir / "templates"
        self.templates_dir.mkdir(parents=True, exist_ok=True)

        # In-memory cache
        self.templates: Dict[str, MinionTemplate] = {}

    async def load_templates(self):
        """Load all templates from disk on startup."""
        self.templates.clear()

        for template_file in self.templates_dir.glob("*.json"):
            try:
                with open(template_file) as f:
                    data = json.load(f)
                    template = MinionTemplate.from_dict(data)
                    self.templates[template.template_id] = template
            except Exception as e:
                print(f"Error loading template {template_file}: {e}")

    async def create_template(
        self,
        name: str,
        permission_mode: str,
        allowed_tools: Optional[List[str]] = None,
        default_role: Optional[str] = None,
        default_system_prompt: Optional[str] = None,
        description: Optional[str] = None
    ) -> MinionTemplate:
        """Create a new template."""
        # Validate name uniqueness
        if any(t.name == name for t in self.templates.values()):
            raise ValueError(f"Template with name '{name}' already exists")

        # Validate permission_mode
        valid_modes = ["default", "acceptEdits", "plan", "bypassPermissions"]
        if permission_mode not in valid_modes:
            raise ValueError(f"Invalid permission_mode. Must be one of: {', '.join(valid_modes)}")

        # Create template
        template = MinionTemplate(
            template_id=str(uuid.uuid4()),
            name=name,
            permission_mode=permission_mode,
            allowed_tools=allowed_tools,
            default_role=default_role,
            default_system_prompt=default_system_prompt,
            description=description
        )

        # Save to disk
        await self._save_template(template)

        # Cache in memory
        self.templates[template.template_id] = template

        return template

    async def get_template(self, template_id: str) -> Optional[MinionTemplate]:
        """Get template by ID."""
        return self.templates.get(template_id)

    async def get_template_by_name(self, name: str) -> Optional[MinionTemplate]:
        """Get template by name."""
        for template in self.templates.values():
            if template.name == name:
                return template
        return None

    async def list_templates(self) -> List[MinionTemplate]:
        """List all templates."""
        return list(self.templates.values())

    async def update_template(
        self,
        template_id: str,
        name: Optional[str] = None,
        permission_mode: Optional[str] = None,
        allowed_tools: Optional[List[str]] = None,
        default_role: Optional[str] = None,
        default_system_prompt: Optional[str] = None,
        description: Optional[str] = None
    ) -> MinionTemplate:
        """Update existing template."""
        template = self.templates.get(template_id)
        if not template:
            raise ValueError(f"Template {template_id} not found")

        # Check name uniqueness if changing name
        if name and name != template.name:
            if any(t.name == name for t in self.templates.values()):
                raise ValueError(f"Template with name '{name}' already exists")
            template.name = name

        # Update fields if provided
        if permission_mode:
            valid_modes = ["default", "acceptEdits", "plan", "bypassPermissions"]
            if permission_mode not in valid_modes:
                raise ValueError(f"Invalid permission_mode")
            template.permission_mode = permission_mode

        if allowed_tools is not None:
            template.allowed_tools = allowed_tools

        if default_role is not None:
            template.default_role = default_role

        if default_system_prompt is not None:
            template.default_system_prompt = default_system_prompt

        if description is not None:
            template.description = description

        # Update timestamp
        template.updated_at = datetime.now(timezone.utc)

        # Save to disk
        await self._save_template(template)

        return template

    async def delete_template(self, template_id: str) -> bool:
        """Delete template."""
        if template_id not in self.templates:
            return False

        # Delete from disk
        template_file = self.templates_dir / f"{template_id}.json"
        if template_file.exists():
            template_file.unlink()

        # Remove from cache
        del self.templates[template_id]

        return True

    async def _save_template(self, template: MinionTemplate):
        """Save template to disk."""
        template_file = self.templates_dir / f"{template.template_id}.json"
        with open(template_file, 'w') as f:
            json.dump(template.to_dict(), f, indent=2)

    async def create_default_templates(self):
        """Create default example templates on first run."""
        # Check if any templates exist
        if self.templates:
            return

        # Create default templates
        defaults = [
            {
                "name": "Code Expert",
                "permission_mode": "acceptEdits",
                "allowed_tools": ["bash", "edit", "read", "write", "glob", "grep"],
                "default_role": "Code review and refactoring specialist",
                "description": "Can freely edit code and run tests"
            },
            {
                "name": "Web Researcher",
                "permission_mode": "default",
                "allowed_tools": ["web_fetch", "web_search", "read"],
                "default_role": "Documentation and research specialist",
                "description": "Can search web and read docs, requires permission for edits"
            },
            {
                "name": "Project Manager",
                "permission_mode": "acceptEdits",
                "allowed_tools": ["bash", "edit", "read", "send_comm", "spawn_minion", "create_channel"],
                "default_role": "Overseer for delegation and coordination",
                "description": "Can manage project structure and delegate tasks"
            },
            {
                "name": "Safe Sandbox",
                "permission_mode": "default",
                "allowed_tools": ["read"],
                "default_role": "Read-only analyst",
                "description": "Highly restricted for safe experimentation"
            }
        ]

        for default in defaults:
            await self.create_template(**default)

The TemplateManager class handles the CRUD operations for minion templates. It loads templates from disk on startup, stores them in an in-memory cache, and provides methods for creating, retrieving, updating, and deleting templates. It also includes validation logic to ensure data integrity and a method to create default templates on the first run.

REST API Endpoints (src/web_server.py):

# Add to ClaudeWebUI class

@app.get("/api/templates")
async def list_templates():
    """List all minion templates."""
    templates = await template_manager.list_templates()
    return [t.to_dict() for t in templates]

@app.get("/api/templates/{template_id}")
async def get_template(template_id: str):
    """Get specific template."""
    template = await template_manager.get_template(template_id)
    if not template:
        raise HTTPException(status_code=404, detail="Template not found")
    return template.to_dict()

@app.post("/api/templates")
async def create_template(request: Request):
    """Create new template."""
    data = await request.json()

    try:
        template = await template_manager.create_template(
            name=data.get("name"),
            permission_mode=data.get("permission_mode"),
            allowed_tools=data.get("allowed_tools"),
            default_role=data.get("default_role"),
            default_system_prompt=data.get("default_system_prompt"),
            description=data.get("description")
        )
        return template.to_dict()
    except ValueError as e:
        raise HTTPException(status_code=400, detail=str(e))

@app.put("/api/templates/{template_id}")
async def update_template(template_id: str, request: Request):
    """Update existing template."""
    data = await request.json()

    try:
        template = await template_manager.update_template(
            template_id=template_id,
            name=data.get("name"),
            permission_mode=data.get("permission_mode"),
            allowed_tools=data.get("allowed_tools"),
            default_role=data.get("default_role"),
            default_system_prompt=data.get("default_system_prompt"),
            description=data.get("description")
        )
        return template.to_dict()
    except ValueError as e:
        raise HTTPException(status_code=400, detail=str(e))

@app.delete("/api/templates/{template_id}")
async def delete_template(template_id: str):
    """Delete template."""
    success = await template_manager.delete_template(template_id)
    if not success:
        raise HTTPException(status_code=404, detail="Template not found")
    return {"deleted": True}

These REST API endpoints expose the minion template functionality to the outside world. They allow users to list, retrieve, create, update, and delete templates via HTTP requests. The endpoints use the TemplateManager to interact with the template data.

Initialization (main.py or web_server.py startup):

# On server startup
template_manager = TemplateManager(data_dir=Path("data"))
await template_manager.load_templates()
await template_manager.create_default_templates()  # Only creates if empty

This code snippet shows how the TemplateManager is initialized on server startup. It loads existing templates from disk and creates default templates if none exist.

Related Files: Where the Magic Happens

Here's a summary of the files involved in this feature:

  • src/models/minion_template.py - NEW: Template data model
  • src/template_manager.py - NEW: Template CRUD operations
  • src/web_server.py - REST API endpoints
  • data/templates/ - NEW: Template storage directory
  • main.py - Initialize TemplateManager on startup

Phase and Priority: Getting Things Done

This feature is part of Phase 5: Minion Templates - Foundation (Part 1 of 4). It's a Medium priority item because it enables the template feature, which is required before UI and MCP integration. Think of it as the foundation upon which we'll build the rest of the template functionality.

Related Issues: Building on Each Other

This feature is related to several other issues:

  • Required by Issue #2 (UI for selecting templates): We need templates before we can build a UI to select them.
  • Required by Issue #3 (list_templates MCP tool): MCP tools need to be able to access templates.
  • Required by Issue #4 (spawn_minion with template support): We need templates to spawn minions with pre-defined configurations.
  • Builds on Issue #145 (minion permission management): This feature leverages existing minion permission management capabilities.

Conclusion: A Solid Foundation for Minion Management

In conclusion, the Minion Templates feature is a significant step forward in simplifying minion management. By providing a way to define and reuse minion configurations, it saves time, reduces errors, and promotes consistency. This feature, focusing on backend storage and CRUD API functionality, lays a solid foundation for future enhancements and integrations. So, get ready to say goodbye to repetitive minion configurations and hello to a more streamlined workflow! This is going to make managing our minions so much easier, guys! We're building something really powerful here, and I'm excited to see how it evolves. Stay tuned for more updates, and let us know what you think! Your feedback is crucial as we continue to develop this feature. Thanks for reading!