@pipeworx/ai-model-experiments

Connect: https://gateway.pipeworx.io/ai-model-experiments/mcp · Install: one-click buttons

Tools: 8

Model Lab — run the same prompt(s) across many AI models at once (Anthropic, OpenAI, Google, Meta, Mistral, DeepSeek, Qwen and more via OpenRouter) and compare their outputs, cost, and latency side by side, with an optional AI-written comparison summary when the run completes.

Tools

• experiment_models(search?, min_context?, max_price_per_mtok?, limit?) — browse ~300 available model ids with context window and our billed per-token price (provider cost × 1.5). Use the returned ids in experiment_create.
• experiment_estimate(prompts[], models[], reps?, params?, summary?) — free dry-run: cell count + estimated billed cost range for a spec, before creating it.
• experiment_create(name?, prompts[], models[], reps?, params?, summary?, max_spend_usd) — creates and starts an experiment (prompts × models × reps). Async: returns experiment_id immediately; execution happens on a separate cron worker within ~1 minute. Requires prepaid balance ≥ max_spend_usd.
• experiment_status(experiment_id) — cell counts by state, spend vs cap, whether complete. Poll this after create.
• experiment_results(experiment_id, include_outputs?) — per-model aggregates (latency, tokens, cost, error rate), per-cell outputs, and the AI-written comparison summary.
• experiment_list(limit?) — caller’s experiments, newest first.
• experiment_cancel(experiment_id) — skips pending cells (unbilled); in-flight cells finish and bill.
• experiment_topup(amount_usd?) — current balance + the x402 top-up flow.

Auth

Prepaid only — no free tier, no BYO-key mode. Every call to experiment_create requires a Pipeworx platform credit balance ≥ max_spend_usd (1 credit = $0.0001). Top up via x402: POST https://gateway.pipeworx.io/credits/topup?amount_usd=N returns an HTTP 402 payment challenge (USDC on Base); retry with a PAYMENT-SIGNATURE header to settle, and credits land instantly. See experiment_topup for the exact flow and current balance.

Billing is provider cost × 1.5, floored at $0.10/experiment. Model prices returned by experiment_models are already the billed (marked-up) price, never the raw provider cost.

Execution is handled by a separate cron worker (workers/experiment-runner, fires every minute) — a created experiment is not synchronous. Poll experiment_status rather than expecting experiment_create to block until done.

Data sources

• https://openrouter.ai/api/v1/models — the model catalog (id, context window, provider pricing) that experiment_models wraps and re-prices.
• Inference for each cell is executed against OpenRouter’s chat completions API by workers/experiment-runner, which also reads OpenRouter’s /generation endpoint for the actual per-call cost so the 1.5× markup is billed on real cost, not an estimate.

Full build plan, architecture, and current live-vs-planned status: docs/model-lab-plan.md.

Tools

• experiment_models — List AI models available for experiments (about 300 across Anthropic, OpenAI, Google, Meta, Mistral, DeepSeek, Qwen and more), with context window and OUR per-token prices (provider cost × 1.5 — what
• experiment_estimate — Dry-run cost estimate for an experiment BEFORE creating it — cell count and estimated billed cost range (at our 1.5× pricing) for prompts × models × reps. Free to call, no side effects, does not need
• experiment_create — Model Lab: run one prompt across many AI models at once and compare their outputs, cost, and latency side by side. Create and start an experiment: run each prompt against each model (× reps), collecti
• experiment_status — Progress of an experiment: cell counts by state (pending/running/ok/error/skipped), spend so far vs cap, and whether it is complete. Poll this after experiment_create (every few seconds). Example: exp
• experiment_results — Results of an experiment: per-model aggregates (mean latency, tokens, total billed cost, error rate), per-cell outputs, and the AI-written summary comparing how the models differed (if enabled). Use i
• experiment_list — List your experiments, newest first, with status and spend. Example: experiment_list({ limit: 10 })
• experiment_cancel — Cancel a running experiment: pending cells are skipped (not billed); in-flight cells finish and bill. Example: experiment_cancel({ experiment_id: ”…” })
• experiment_topup — How to add prepaid credits for experiments (and your current balance). Payment is x402 — USDC on Base, paid in-band by any wallet-equipped agent: POST https://gateway.pipeworx.io/credits/topup?amount_