Advanced

The Auto-Research Loop

What this gives you: Autonomous optimization for any metric in your business. Set a target - lower your ad costs, increase your booking rate, improve your landing page conversion - and Claude will test changes, measure results, and keep what works. You wake up to a report of what improved and why.

Most optimisation is guess-and-check. The auto-research loop makes it systematic - define a metric, run experiments automatically, keep what works, discard what does not, and compound improvements over time.

The Origin: Karpathy's Method

Andrej Karpathy, former head of AI at Tesla and one of the original OpenAI researchers, described a pattern for automated experimentation that is deceptively simple. The core idea: any system with a measurable output can be improved automatically if you can define a metric, a way to change the system, and a way to assess whether the change made things better or worse.

His context was neural network training and model evaluation. But the pattern maps perfectly onto business systems - sales copy, AI agent prompts, landing pages, ad creative. Any time you have a "thing" that produces a "result" you can measure, you can apply this loop.

The reason most people do not do this is that automated experimentation feels like a lot of setup for uncertain payoff. This guide shows you that the setup is far simpler than it looks, and the payoff compounds indefinitely.

The Loop Structure

# The auto-research loop in pseudocode

LOOP:
  1. Define metric      # What does "better" mean, as a number?
  2. Define change      # What is the variable you are changing?
  3. Define assessment  # How do you measure the metric objectively?

  REPEAT:
    a. Form hypothesis  # "Changing X will improve metric by Y%"
    b. Make the change  # Edit the prompt, copy, code, creative
    c. Run assessment   # Measure the metric after the change
    d. Compare          # Did it improve vs the control?
    e. If improved:     # Keep the change, log it, set new baseline
       keep + log
    f. If worse:        # Revert the change, log the failure
       revert + log
    g. Loop             # Form the next hypothesis and repeat

The power is in step (g). Each iteration builds on the last. Each successful change raises the baseline. After 20-30 iterations, the system is dramatically better than where it started - and you have a log of every change and its effect, which is itself a dataset for future improvements.

Application 1: Voice Agent Prompts

Metric: Booking rate (appointments booked / calls connected, as a percentage)
Change variable: The AI voice agent's system prompt
Assessment method: Pull call logs from Retell AI API, calculate booking rate over 50+ calls

#!/bin/bash
# auto-research/voice-agent/run-experiment.sh

source ~/.env

HYPOTHESIS="$1"      # What change are we making?
NEW_PROMPT_FILE="$2"  # Path to updated prompt
EXPERIMENT_ID=$(date +%Y%m%d-%H%M%S)

# 1. Upload new prompt to Retell AI
curl -s -X PATCH "https://api.retellai.com/v2/agent/$RETELL_AGENT_ID" \
  -H "Authorization: Bearer $RETELL_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"general_prompt\": \"$(cat $NEW_PROMPT_FILE | jq -Rs .)\"}"

echo "Prompt updated. Running for 48 hours..."
sleep 172800  # 48 hours

# 2. Pull call data and calculate booking rate
CALLS=$(curl -s "https://api.retellai.com/v2/list-calls?agent_id=$RETELL_AGENT_ID&limit=100" \
  -H "Authorization: Bearer $RETELL_API_KEY")

TOTAL=$(echo $CALLS | jq '.calls | length')
BOOKED=$(echo $CALLS | jq '.calls | map(select(.call_analysis.custom_analysis_data.booked == true)) | length')
RATE=$(echo "scale=2; $BOOKED / $TOTAL * 100" | bc)

# 3. Log results
echo "{
  \"experiment_id\": \"$EXPERIMENT_ID\",
  \"hypothesis\": \"$HYPOTHESIS\",
  \"total_calls\": $TOTAL,
  \"bookings\": $BOOKED,
  \"booking_rate\": $RATE,
  \"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"
}" >> auto-research/voice-agent/results.jsonl

echo "Booking rate: $RATE% ($BOOKED / $TOTAL calls)"

Over 10 experiments - each one a specific hypothesis about what might improve booking rate - you might find that adding social proof in the first 15 seconds increases bookings by 4%, using the prospect's first name in the opening increases by 6%, and shortening the pitch from 45 seconds to 30 seconds increases by 11%. These stack. You end up with a prompt that is 20%+ better than where you started, with a documented trail of what worked.

Application 2: Cold Email Sequences

Metric: Reply rate (positive replies / emails delivered, as a percentage)
Change variable: Subject line, opening line, or call-to-action
Assessment method: Pull campaign stats from your email tool (Instantly, Lemlist, etc.) after 200+ sends

# Claude Code prompt to run one experiment cycle

"You are running experiment #7 on the cold email sequence for solar installers.

Current control: Subject line 'Quick question about [Company]'
Current reply rate: 3.2% (baseline from last 500 sends)

Hypothesis to test: Replace generic 'quick question' with a specific observation
about their business. Example: 'Noticed you're still calling old leads manually, [Name]'

Task:
1. Generate 3 subject line variants based on this hypothesis
2. Create the A/B test in Instantly using the API
3. Set to run on the next 200 contacts in the sequence
4. Schedule a check-in in 5 days to evaluate results

Instantly API key is in ~/.env as INSTANTLY_API_KEY
Target campaign ID: camp_abc123"

Application 3: Landing Pages

Metric: Conversion rate (form submissions or CTA clicks / unique visitors)
Change variable: Headline, hero section, social proof, CTA copy, page structure
Assessment method: Pull from Google Analytics or Vercel Analytics after 500+ visitors

# Claude Code running a landing page experiment

"Run landing page experiment for rjmediahub.com/solar-demo.

Baseline: Current page. Conversion rate: 2.1% (from GA4, last 14 days, 800 visitors)

Hypothesis: The hero headline is too generic. Replacing with a specific outcome
claim ('We booked 23 extra consultations for SunPower London in 90 days')
will outperform the current 'AI That Fills Your Calendar'.

Steps:
1. Create a variant of the hero section with the new headline
2. Update the page to show variant B (or set up 50/50 split via edge middleware)
3. Log this experiment to experiments/landing-page/log.md
4. In 7 days, pull GA4 conversion data and compare

Current page: public/landing-solar.html
Experiments log: experiments/landing-page/log.md"

Application 4: Ad Creative

Metric: Cost per lead (total ad spend / leads generated)
Change variable: Ad image, headline, body copy, or audience targeting
Assessment method: Pull from Meta Ads Manager API after 5,000+ impressions

#!/usr/bin/env python3
# auto-research/ads/fetch-results.py
# Pull ad performance from Meta and log to experiments table

import os, json, requests
from datetime import datetime, timedelta

ACCESS_TOKEN = os.environ['META_ADS_TOKEN']
AD_ACCOUNT_ID = os.environ['META_AD_ACCOUNT_ID']

def get_ad_performance(ad_id: str, days: int = 7):
    since = (datetime.now() - timedelta(days=days)).strftime('%Y-%m-%d')

    url = f"https://graph.facebook.com/v19.0/{ad_id}/insights"
    params = {
        'access_token': ACCESS_TOKEN,
        'fields': 'spend,actions,cost_per_action_type,impressions,clicks',
        'time_range': json.dumps({'since': since, 'until': 'today'}),
    }

    resp = requests.get(url, params=params)
    data = resp.json()['data'][0]

    leads = next(
        (a['value'] for a in data.get('actions', [])
         if a['action_type'] == 'lead'), 0
    )

    return {
        'spend': float(data['spend']),
        'leads': int(leads),
        'cost_per_lead': float(data['spend']) / int(leads) if int(leads) > 0 else 999,
        'impressions': int(data['impressions']),
    }

Application 5: Website Performance

Metric: Lighthouse performance score (0-100)
Change variable: Image formats, JavaScript bundles, CSS loading, caching headers
Assessment method: Automated Lighthouse run via CLI after each code change

# Run Lighthouse and log the score
npx lighthouse https://yoursite.com \
  --output json \
  --output-path ./lighthouse-results.json \
  --chrome-flags="--headless"

# Extract the performance score
SCORE=$(cat lighthouse-results.json | jq '.categories.performance.score * 100')
echo "Performance score: $SCORE"

# Append to experiment log
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ), $SCORE, $HYPOTHESIS" >> performance-log.csv

The loop for Lighthouse optimisation is extremely fast - each experiment takes minutes, not days. You can run 10-20 iterations in an afternoon and take a page from a 60 to a 95 score systematically.

Full Implementation: Tracking Table

Every experiment needs a log. Without it, you are just guessing repeatedly with no institutional memory. Here is a real tracking example:

ID	Hypothesis	Before	After	Delta	Status
`VA-01`	Add social proof in opener	8.2%	9.1%	+0.9%	KEPT
`VA-02`	Use first name 3x vs 1x	9.1%	8.8%	-0.3%	REVERTED
`VA-03`	Shorten pitch 45s to 30s	9.1%	10.8%	+1.7%	KEPT
`VA-04`	Add urgency in close	10.8%	12.1%	+1.3%	KEPT
`VA-05`	Ask for referral at end	12.1%	11.9%	-0.2%	REVERTED

Result: 8.2% to 12.1% booking rate. +47% improvement from 5 experiments.

Tip: Log failures as carefully as successes. A hypothesis that made things worse is valuable information - it tells you what the system is not sensitive to, and prevents you from retesting the same bad idea six months later.

Setting Up Auto-Research as a Claude Code Skill

Once you understand the loop, you can codify it as a reusable skill so any agent in your system can run experiments:

# /home/jarvis/skills/auto-research.md

# Auto-Research Loop Skill

When asked to optimise any measurable system, follow this process:

## Step 1: Define the Experiment
- State the metric (must be a number)
- State what you are changing (one variable at a time)
- State how you will measure (specific API or tool)
- Set minimum sample size before evaluating (to avoid noise)

## Step 2: Run the Experiment
- Make the change in the actual system
- Log start time and current baseline metric
- Wait for minimum sample size to accumulate
- Do NOT check results before minimum sample - this creates false signals

## Step 3: Evaluate and Log
- Pull the metric using the defined measurement method
- Compare to baseline (percentage change, not absolute)
- If improved by more than 2%: KEEP and set new baseline
- If within 2%: INCONCLUSIVE, run more data or try different hypothesis
- If worse by more than 2%: REVERT and log
- Append to the tracking table in experiments/tracking.md

## Step 4: Next Hypothesis
- Review the tracking table for patterns
- What has worked? Double down on that direction.
- What has failed? Avoid that class of change.
- Form the next hypothesis and start again.

Logging Patterns: What to Record and Where

Every experiment log entry should capture:

Field	What to Record
Experiment ID	Sequential, easy to reference
Timestamp	When the experiment started AND when it was evaluated
Hypothesis	In plain English, one sentence maximum
What exactly changed	Enough detail to reproduce or revert
Sample size	How many calls/emails/visitors the assessment was based on
Metric before and after	Both absolute numbers and percentage change
Decision	KEPT, REVERTED, or INCONCLUSIVE (with reason)

Store these logs as JSONL files (one JSON object per line) for easy programmatic analysis later. As the log grows, you can run a Claude Code analysis over the entire experiment history to find patterns you have not spotted manually.

# Each log entry as JSONL
{"id":"VA-04","date":"2026-02-05","metric":"booking_rate","hypothesis":"Add urgency close","before":10.8,"after":12.1,"sample":82,"decision":"KEPT"}

# Analyse the full experiment log with Claude
claude -p "Analyse this experiment log. What categories of changes reliably improve
the booking rate? What categories reliably fail? What should the next 3
experiments test? Log: $(cat experiments/voice-agent/results.jsonl)"

Next Guide

10 Commands You Can Run Today

The Auto-Research Loop

The Origin: Karpathy's Method

The Loop Structure

Application 1: Voice Agent Prompts

Application 2: Cold Email Sequences

Application 3: Landing Pages

Application 4: Ad Creative

Application 5: Website Performance

Full Implementation: Tracking Table

Setting Up Auto-Research as a Claude Code Skill

Logging Patterns: What to Record and Where

Want this running automatically for your business?