--- version: "0.1.2" level: copilot processes: design: pair implementation: copilot testing: auto documentation: auto review: pair deployment: copilot components: src: copilot functions: copilot eval: auto research: auto docs: auto scripts: copilot functions/data: none --- ## Notes Tutor (tutorman.dev) is an AI card search and deck builder for Magic: The Gathering. There are two separate things worth declaring here, and people usually mean the first one. **The product uses AI at runtime.** Every search is answered by a model. TypeSafe's Jev scores each of the 34,741 cards against your query, and deck building plus a few search types use Claude. Results can be wrong about rules, legality and prices, so check anything that matters before a tournament. **The code was written by AI, to a human's specification.** A human decided what this is, what it must do, what it must not do, and what evidence was required before anything shipped. AI wrote the implementation quickly, and was then pointed at its own work as a second opinion. ### What the human directed * **What it is.** An MTG tool, not an AI demo. The interface never mentions the model underneath, because players care about cards. Search reads plain English. The deck builder covers all 14 formats, not only Commander. * **What it may do.** The deck builder is free and open to everyone. Public deck pages are opt-in per deck. Sign-in is Google only. Raw IP addresses are never stored. Spend is capped daily and monthly, with kill switches in an admin page. * **Which capabilities were worth building.** Slang and keyword handling, deck revision, saved cards and decks, card list pages, and traffic analytics we own rather than a third party script. Rejected: routing search through a vector store, which measured worse than the simpler approach; turning thumbs-down feedback into automated test labels, because there is no evidence yet that players are accurate about what was missing; and storing IP addresses to get regional data. * **The standard for shipping.** Accuracy first, cost second. Risky changes go in a fork, never straight to production. Claims need numbers. A cost reduction is not a result until the accuracy suite has run, and an improvement smaller than the noise floor is not an improvement. * **What to test, and how to know it worked.** The human asked for regression baselines built from real logged queries, for a live test against the deployed site rather than a local mock, and for the accuracy study to be measured end to end on test data drawn from real traffic instead of hand-picked examples. ### What the AI did * **Implementation, for speed.** The application, the cloud functions, the styles, the build and the data pipelines. * **Measurement, as a second opinion.** The test suites and evaluation harnesses the human asked for: a search regression suite over real logged queries, security-rules tests, a live production harness, and an accuracy set of 30 real queries with 800 cards labelled by a different model from the one being graded, so the system under test is not its own judge. * **Reporting against itself.** Several of the AI's own proposals were killed by its own measurements once the human asked for proof. Re-ranking results by query condition had no measurable effect (t = 0.33). Routing multi-part queries to a reasoning model was worse and six times the price. Enlarging the search pool bought no accuracy at all. ### What review caught that automation did not Review is declared as `pair` for a specific reason. The bugs that mattered were found by a human using the site, not by the AI that wrote it or by the tests that AI wrote. * A colour filter quietly restricted "mana dorks" to green. The query never said green. Mana dorks simply usually are, so four fifths of the card pool was discarded before judging began. * Day counters rolled over at 8pm, because they were keyed in UTC. The dashboard was reporting spend for a day that had not started yet. * The card list pages were accurate and ugly, which would have converted nobody. * A settings form refused to accept the value production was already running. None of those were caught by the automated suites. They were caught by someone looking at the product and asking why it behaved that way. That is why the human stayed in the loop on design and review, rather than only approving deploys. ### Data Card names, rules text, prices and legality come from [Scryfall](https://scryfall.com). The concept tags that narrow each search were generated by Jev in a one-off pass. None of this is presented as human-written code. If you are judging the tool, the useful question is whether it gets Magic right. Measured against an independent judge, it reaches 96 to 100% of achievable precision on searches that ask for one thing, and about 63% on searches that ask a card to do two things at once. That second number is the weakest part of the product, and it is known to be. Tutor is unofficial Fan Content permitted under the Wizards of the Coast Fan Content Policy. Not approved or endorsed by Wizards. Portions of the materials used are property of Wizards of the Coast. ©Wizards of the Coast LLC.