Performance metrics & debugging

A backtest without metrics is just a chart. Quant Buffet reports risk-adjusted statistics so you can compare strategies honestly — and gives you an AI debugger with line numbers when the code breaks.

Metrics glossary

Reference points so you know what "good" means

Calibrate against these before getting excited. A long-only daily ETF strategy with a Sharpe of 0.8 is genuinely good. A Sharpe of 1.5 is exceptional and demands scrutiny. A Sharpe above 2 in a simple daily backtest almost always means look-ahead bias, an unrealistic cost assumption, or a survivor-picked universe. For context: Madoff's fabricated returns implied a Sharpe of roughly 2.5, and that implausibility was one of the earliest public clues.

Why drawdown matters more than CAGR

Drawdown is where backtests meet human beings. Two facts make it concrete. A 50% loss requires a 100% gain to recover, not 50%. And recovery can take an investing lifetime: the Nasdaq Composite peaked in March 2000 and did not durably reclaim that level until 2015 — roughly fifteen years of holding an underwater position. Meanwhile the fastest crash in modern history, February to March 2020, took only 33 days to fall about 34%.

Multiple testing: the statistics of trying again

If you test one strategy, a Sharpe of 1.0 is evidence. If you test a thousand and report the best, a Sharpe of 1.0 is close to what pure randomness delivers — the maximum of many noisy draws is large by construction. This is why the literature developed the deflated Sharpe ratio (Bailey and López de Prado), which penalises a result by how many attempts produced it.

• Count your attempts. Ten IDE re-runs with different lookbacks is ten tests, even if you only screenshot one.
• Prefer parameter plateaus. If 150, 200 and 250 days all work, the effect is probably real. If only 187 works, it is noise.
• Hold back a period. Fit your intuition on 2000–2015, then look at 2016–today once.
• Demand a reason. Every library strategy carries an economic rationale precisely because "the numbers liked it" is not one.

Debugging workflow

1. Read the error message and line number highlighted in the IDE gutter.
2. Compare your file against the contract: module-level ASSETS, plus make_on_day(prices) returning (on_day, ready).
3. Click Ask AI for syntax — you get where, why, and paste-ready snippets.
4. Consult API docs for imports, whitelist symbols, and template examples.
5. Re-run with a shorter `ASSETS` list and a later start to isolate data problems.

The five errors you will actually hit

# Slow: recomputes a 200-day mean on every single day (times out)
def on_day(engine, dt):
    window = prices.loc[:dt, "SPY"].tail(200)
    if prices.loc[dt, "SPY"] > window.mean():
        engine.set_target_weights(dt, {"SPY": 1.0})

# Fast: compute once, then look up
sma = prices["SPY"].rolling(200, min_periods=200).mean()

def on_day(engine, dt):
    if prices.loc[dt, "SPY"] > sma.loc[dt]:
        engine.set_target_weights(dt, {"SPY": 1.0})

# What the platform reports after a successful run
metrics = compute_metrics(result.equity, benchmark=spy_bh, trades_count=len(result.trades))
print(metrics["sharpe"], metrics["max_drawdown"], metrics["cagr"])