# How Moneda Bench works Moneda Bench is seven separate, narrow benchmarks. Each has its own items, grader, leaderboard and release, and there is no single combined score. ## The seven benchmarks - [Moneda AdviceBench](https://benchmarks.finance/benchmarks/moneda-advicebench/): Whether a system avoids telling a specific person what to buy or sell, without refusing to explain finance. - [Moneda CiteBench](https://benchmarks.finance/benchmarks/moneda-citebench/): Whether the sources a system cites say what it claims. - [Moneda FilingsBench](https://benchmarks.finance/benchmarks/moneda-filingsbench/): Questions about a US company's SEC filings with no document attached. - [Moneda GroundBench](https://benchmarks.finance/benchmarks/moneda-groundbench/): Whether a system invents market figures it cannot know. - [Moneda KnowledgeBench](https://benchmarks.finance/benchmarks/moneda-knowledgebench/): Whether a model knows finance: valuation, fixed income, accounting, derivatives, portfolio theory and market mechanics, plus whether it can explain a concept to a beginner. - [Moneda MarketBench](https://benchmarks.finance/benchmarks/moneda-marketbench/): Questions about the market as a whole rather than one company: breadth and participation, sector moves and fund flows, why an index moved, exchange hours and holidays, and event calendars and instruments across the US, Europe and Latin America. - [Moneda PortfolioBench](https://benchmarks.finance/benchmarks/moneda-portfoliobench/): Decision quality on hypothetical portfolios: rebalancing, risk measures, position sizing and constraints; calibrated forecasts on dated events; and running a paper fund to a written mandate. ## Two tracks, never mixed **Products** are tested as you would use them: AI products with their own search and tools, live, in the same two-hour window on a trading day, on the same items, with default settings and no retries. Numa 1, Moneda's AI, appears only here, tested through its public interface like any other product. **Base models** are tested on equal tools: every model runs inside one open reference agent with the same fixed tool kit, answering from a frozen, dated market snapshot that ships with the release. Anyone can rerun it, and adding a model is one line of configuration. A "no tools" column shows what the tools add. We never say which base model any product uses. ## A small public sample, a private test set Each benchmark publishes about one question in ten as a sample, with its grader, which you can run with one command. Everything else is a private test set, and only that score is ranked: nobody can train on questions they never see. Sample scores are shown too, marked as such. They move a lot over so few questions, and a system that does far better on the sample than on the test set has probably seen the sample before. ## Canary strings and contamination checks Every published file carries a canary string so it can be filtered out of training data. Live benchmarks generate fresh questions on every run. Moneda KnowledgeBench, the one most likely to be memorised, also ships a twin file with some answer keys randomised: a model that scores well above the twin's ceiling has trained on the published file. ## Grading Code grades every item that has a fixed answer. Where a judgement is needed, an AI judge gives yes or no verdicts per criterion on answers stripped of anything that identifies the system. The judge is checked against people on a held-out set before the rubric is tuned, and two people blind-review 15% of verdicts, at least one from outside Moneda with finance credentials. If agreement falls below 0.7, that board is labelled experimental and not ranked. One system beats another only when the difference holds up statistically. Live benchmarks run three times per system and report the spread. ## Releases Each release is dated and frozen, with a changelog. Items are never edited under a published score. Items that leak are retired and replaced in the next release. Questions are published under CC BY 4.0 and the harness under MIT. ## How results stay fair Numa 1, Moneda's AI, gets no special treatment: - The same items, the same time window, the same grader and default settings as every other system. - Every transcript is saved, and anyone can rerun the public samples. - Every result is published, whichever system comes out ahead. - If one system leads on every benchmark, an outside party reruns those results before they are published. Moneda creates and maintains the site. The domain is neutral so that a research group can co-host it later. ## The directory The directory lists other finance-AI benchmarks too. We link to the originals, credit the authors and do not rehost their data beyond what their licence allows. Scores shown are either quoted from the authors' paper or official board, with the source and date, or were run by Moneda with the benchmark's own grader and rules. We do not copy other aggregators. Each entry has a last-checked date, and [corrections](https://benchmarks.finance/contact/?kind=correction) are welcome. ## What this site is not Nothing here is investment advice. Moneda is not a regulated financial institution and does not manage portfolios. Moneda AdviceBench measures whether answers avoid personal buy or sell instructions; it does not certify compliance with any rule. Moneda PortfolioBench describes decisions on hypothetical portfolios in a test setting; it is not investment performance. Source: https://benchmarks.finance/method/ Created and maintained by Moneda.