[{"data":1,"prerenderedAt":227},["ShallowReactive",2],{"explorations":3,"exploration-maestro-bench":15},[4],{"path":5,"title":6,"description":7,"date":8,"tags":9,"status":13,"icon":14},"\u002Fexplorations\u002Fmaestro-bench","MAESTRO-bench, comparing AI agents","How a benchmark tool for AI agents went from scripts and CSV files to a loop of dataset, experiment and analysis, grouped in studies.","2026-10-22",[10,11,12],"AI","Benchmark","Workflow","wip","lucide:flask-conical",{"id":16,"title":6,"body":17,"date":8,"description":7,"extension":219,"icon":14,"meta":220,"navigation":221,"path":5,"references":222,"seo":223,"status":13,"stem":224,"tags":225,"__hash__":226},"explorations\u002Fexplorations\u002Fmaestro-bench.md",{"type":18,"value":19,"toc":206},"minimark",[20,23,33,36,41,44,47,50,53,58,61,65,68,97,100,104,107,129,132,135,138,142,145,161,164,167,170,174,177,180,184],[21,22],"explorations-maestro-bench-logo",{},[24,25,26,27,32],"p",{},"MAESTRO-bench (MB) is a benchmark tool to compare AI agents, built for research. It sits next to two other pieces of my ",[28,29,31],"a",{"href":30},"\u002Fresearch","research",": Konsolidation, the study app for learners and teachers, and MAESTRO, the multi-agent AI backend behind it.",[24,34,35],{},"The agents of MAESTRO have a lot of settings: the architecture, the prompt, the model. Each configuration has to be analyzed, which raises two questions. How do we know which one is the best? And how do we prove it?",[37,38,40],"h2",{"id":39},"a-loop-of-three-steps","A loop of three steps",[24,42,43],{},"The answer fits in one loop. You prepare a dataset, you run an experiment on it, you analyze the result against a reference. Then you change one thing and start again, until the configuration is the best one.",[24,45,46],{},"The flow below plays on its own. Each pass changes one element (a prompt, a model, the rows) and the analysis compares it to a baseline. The values are synthetic.",[24,48,49],{},"Hover a step to see what it depends on, and click it to open its details. Pause the flow or change its speed from the bottom right.",[51,52],"explorations-maestro-bench-loop",{},[54,55,57],"h3",{"id":56},"what-each-step-holds","What each step holds",[24,59,60],{},"The dataset is a table of rows, for example learner responses with a reference grade. The experiment sends the chosen rows, all or a sample, to an agent (one prompt, one model) and records what comes back. The analysis compares those answers to the baseline, and to other experiments, with the indicators that fit the kind of value: RMSE, MAE or Pearson r for numbers, accuracy, macro F1 or Cohen's κ for categories. Each score comes with a 95% confidence interval from a bootstrap.",[37,62,64],{"id":63},"how-it-got-there","How it got there",[24,66,67],{},"The first answer was not a tool, it was a folder. Each version below fixes what hurt in the previous one.",[69,70,72,81,89],"compare",{"caption":71},"Three ways to run the same loop",[73,74,78],"compare-item",{"label":75,"note":76,"verdict":77},"Scripts and CSV files","After a month, nobody knows which file is the best one, nor which prompt or model made it.","fails",[24,79,80],{},"The first answer. Copy a script, edit it, run it again, save a CSV. The folder grows with every variation.",[73,82,86],{"label":83,"note":84,"verdict":85},"One experiment at a time","Everything is saved and analyzed, but each variation is still a manual round trip.","partial",[24,87,88],{},"MB records the prompt, the model and the answer of every launch, and the analysis is built in. A variation still means change, launch, wait, fetch.",[73,90,94],{"label":91,"note":92,"verdict":93},"Batches and studies","The variations run together, and the reasoning stays next to the results.","works",[24,95,96],{},"A batch launches all the variations at once. A study keeps them with the analyses and the notes of the team, and a link shares them.",[24,98,99],{},"Each version removes a chore of the previous one. Saving every launch replaces the folder, varying one element at a time keeps the comparison fair, the batch removes the waiting, and the study keeps the reasoning next to the results.",[37,101,103],{"id":102},"more-than-one-agent","More than one agent",[24,105,106],{},"An experiment does not have to stop at one agent. MAESTRO is a multi-agent backend, so MB can run a whole workflow of agents on each row, and the loop stays the same: dataset, experiment, analysis.",[24,108,109,110,114,115,114,118,114,121,124,125,128],{},"Two shapes cover most cases. Agents in a row, where the answer of the first becomes the input of the second. And agents side by side, which read the same row at the same time, followed by an aggregator that combines their grades with a formula. The aggregator can use arithmetic, comparisons, logic and functions such as ",[111,112,113],"code",{},"mean",", ",[111,116,117],{},"min",[111,119,120],{},"max",[111,122,123],{},"clamp"," or ",[111,126,127],{},"vote",".",[24,130,131],{},"The example below grades a student answer with two judges: one reads the content, the other the language, each with its own prompt and its own model. Press play, then click the aggregator and try the formulas. The answers and grades are synthetic.",[133,134],"explorations-maestro-bench-agents",{},[24,136,137],{},"Each agent is a variable of the experiment like any other: change the prompt of one judge, swap its model, or change the formula, and the analysis tells whether the workflow got better.",[37,139,141],{"id":140},"how-a-study-is-structured","How a study is structured",[24,143,144],{},"Four levels, from the question to a single launch:",[146,147,148,152,155,158],"ul",{},[149,150,151],"li",{},"A study groups what matters for one question.",[149,153,154],{},"A batch is a series of variations, launched together.",[149,156,157],{},"An experiment is one configuration run on the dataset.",[149,159,160],{},"An execution is one launch of an experiment, frozen as it was. It is the artifact, the piece of evidence behind a result.",[24,162,163],{},"Below, a study is drawn like the trace of a request: each level sits inside its parent, and the bars are time. The batch runs two experiments at once, a setting, and the others wait for a free slot. Click a row to see what it holds. The study is an example, with synthetic values.",[165,166],"explorations-maestro-bench-structure",{},[24,168,169],{},"The frozen execution is what makes a result checkable. In reproducibility work, such a piece of evidence is called an artifact, and every execution is one. The prompt, the model, the parameters, each answer, the cost and the time stay as they were, even if the prompt is edited later. A study can also be shared as a read-only link, with no account needed to open it.",[37,171,173],{"id":172},"where-it-landed","Where it landed",[24,175,176],{},"MB is now one interface for the whole loop, instead of scripts to copy. Everything is configurable (prompt, model, temperature, sampling, even several agents in a row) and nothing ties it to one provider: Mistral, Claude and GPT run side by side. Every launch is recorded answer by answer, and every score comes with its interval against a baseline.",[24,178,179],{},"It is used by teams: each project has its own workspace, with roles and its own API keys, so the models, prompts, datasets and experiments of a team stay theirs.",[37,181,183],{"id":182},"open-source-shared-ownership","Open source, shared ownership",[24,185,186,187,193,194,199,200,205],{},"The code is open source: it can be inspected, reused and adapted. MB is jointly owned by the ",[188,189,192],"entity-link",{"entity":190,"href":191},"ut","https:\u002F\u002Fwww.univ-toulouse.fr\u002F","Université de Toulouse"," and ",[188,195,198],{"entity":196,"href":197},"kosmos","https:\u002F\u002Fwww.kosmos-education.com\u002F","Kosmos Education",". It is being deployed on the infrastructure of the ",[188,201,204],{"entity":202,"href":203},"irit","https:\u002F\u002Fwww.irit.fr\u002F","IRIT",", which will host it.",{"title":207,"searchDepth":208,"depth":208,"links":209},"",2,[210,214,215,216,217,218],{"id":39,"depth":208,"text":40,"children":211},[212],{"id":56,"depth":213,"text":57},3,{"id":63,"depth":208,"text":64},{"id":102,"depth":208,"text":103},{"id":140,"depth":208,"text":141},{"id":172,"depth":208,"text":173},{"id":182,"depth":208,"text":183},"md",{},true,null,{"title":6,"description":7},"explorations\u002Fmaestro-bench",[10,11,12],"Q63qDGslfDZKyU05_mNgb2NOPNyBMVXyhMzIsg_2fqE",1792744397107]