`experiment_create`

Pack: ai-model-experiments · Endpoint: https://gateway.pipeworx.io/ai-model-experiments/mcp

Model Lab: run one prompt across many AI models at once and compare their outputs, cost, and latency side by side. Create and start an experiment: run each prompt against each model (× reps), collecting output, tokens, latency, and billed cost per cell. PREPAID: requires Pipeworx credit balance ≥ max_spend_usd (top up via experiment_topup); bills actual provider cost × 1.5 with a $0.10 minimum per experiment. ASYNC: returns experiment_id immediately — execution starts within ~1 minute; poll experiment_status until complete, then call experiment_results. Do NOT wait synchronously. Set summary:false to skip the AI-written model-comparison summary. Example: experiment_create({ name: “tone test”, prompts: [“Rewrite formally: …”], models: [“anthropic/claude-haiku-4.5”, “openai/gpt-5-mini”], reps: 2, max_spend_usd: 2 })

Parameters

Name	Type	Required	Description
`name`	string	no	Short experiment name
`prompts`	array	yes	Prompts to test (max 20)
`items`	string	no
`models`	array	yes	Model ids from experiment_models (max 12)
`items`	string	no
`reps`	number	no	Repetitions per prompt×model for variance, 1-5 (default 1)
`params`	object	no	Optional {system, temperature, max_tokens (default 512)}
`summary`	boolean	no	AI-written comparison of the models’ outputs when the run completes (default true)
`max_spend_usd`	number	yes	REQUIRED hard spend cap in USD for this experiment (max 100). Execution stops when reached.

Example call

Arguments

{
  "name": "model comparison",
  "prompts": [
    "Explain quantum computing in simple terms."
  ],
  "models": [
    "anthropic/claude-sonnet-4.5",
    "openai/gpt-4o-mini"
  ],
  "max_spend_usd": 5
}

curl

curl -X POST https://gateway.pipeworx.io/ai-model-experiments/mcp \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"experiment_create","arguments":{"name":"model comparison","prompts":["Explain quantum computing in simple terms."],"models":["anthropic/claude-sonnet-4.5","openai/gpt-4o-mini"],"max_spend_usd":5}}}'

TypeScript (@pipeworx/sdk)

import { Pipeworx } from '@pipeworx/sdk';
const pipeworx = new Pipeworx();

const result = await pipeworx.call('experiment_create', {
  "name": "model comparison",
  "prompts": [
    "Explain quantum computing in simple terms."
  ],
  "models": [
    "anthropic/claude-sonnet-4.5",
    "openai/gpt-4o-mini"
  ],
  "max_spend_usd": 5
});

More examples

{
  "name": "tone test",
  "prompts": [
    "Rewrite formally: hey there!",
    "Rewrite casually: Good morning."
  ],
  "models": [
    "anthropic/claude-haiku-4.5",
    "openai/gpt-4o-mini",
    "google/gemini-2.0-flash"
  ],
  "reps": 2,
  "params": {
    "temperature": 0.8,
    "max_tokens": 256
  },
  "summary": true,
  "max_spend_usd": 10
}

Connect

Add this to your MCP client config, or use one-click install buttons:

{
  "mcpServers": {
    "ai-model-experiments": {
      "url": "https://gateway.pipeworx.io/ai-model-experiments/mcp"
    }
  }
}

See Getting Started for client-specific install steps.

experiment_create