0x2117bf88…e42asent to0xf790a5f5…1365·#25,519,666·view on Etherscan
# Nouns x AI
**TL;DR:** Proposal text carries real (if modest) signal about whether a proposal passes — a model that reads only the title and body ranks proposals better than chance, AUC 0.65. That signal *survived* the BreakEven regime change. What broke was the base rate. **BreakEven didn't change what makes a proposal good. It changed how good a proposal has to be.** Everything below is open: dataset, model, split indices, code. Please go check my work.
---
## What I built
I fine-tuned DistilBERT on all 982 Nouns proposals, using only the **title and description;** no proposer identity, no requested amount, no treasury balance, no voting history. Just the prose. The question: does what you write actually predict what happens to you?
Then I ran three experiments instead of one, because the first version of this gave me a misleading answer and I want to show you why.
## The three runs
| run | split | test pass rate | baseline | accuracy | lift | AUC |
| --- | ----------------------------------- | -------------- | -------- | -------- | ---- | ----- |
| A | stratified random, all 982 | 50.2% | 50.2% | 58.9% | +8.6 | 0.648 |
| B | temporal: train 1–785, test 786–982 | 28.4% | 71.6% | 73.1% | +1.5 | 0.651 |
| C | post-786 only, stratified | 28.0% | 72.0% | 72.0% | 0.0 | 0.365 |
"Baseline" is majority-class accuracy, what you'd score by ignoring the proposal entirely and always guessing the more common outcome. **Any accuracy number that doesn't beat its baseline means nothing.** This is the single most common way ML results get oversold, and I'd rather hand you the yardstick than ask you to trust me.
## What each one says
**Run A — text does carry signal.** Holding the regime constant, the model beats the coin-flip baseline by 8.6 points, AUC 0.648. That's real but weak. It is *not* strong enough to tell any individual proposer their proposal will fail. It is strong enough to say: **how you write your proposal is not noise.**
**Run B — this is the interesting one.** Train on everything before proposal #786, test on everything after. Accuracy barely clears baseline (+1.5) — which looks like the model broke.
It didn't. Look at the AUC: **0.651, essentially identical to Run A.** The model's *ranking* still works post-BreakEven. It still knows which proposals are relatively more passable. What it lost is **calibration** — it predicts PASS 12.7% of the time against a true rate of 28.4%. It learned the old regime's threshold, and the threshold moved.
Put plainly: it's a professor who learned that a 70 was a passing grade, then got handed a class where the cutoff is now a 90. He can still rank the papers correctly. He just keeps saying "that one passed" about papers that didn't.
**Run C — I don't trust it, and neither should you.** Trained only on post-BreakEven proposals, the model collapses to predicting FAIL on literally everything (0.0% predicted PASS), which scores exactly the 72% baseline. Its AUC of 0.365 is *below* chance, which would be a wild result if I believed it — but it's trained on 147 examples and tested on 50, with one seed. The confidence interval is wide enough to swallow 0.5 whole. **I'm reporting it because I said I'd report everything, not because it means anything yet.** If someone wants to run it over multiple seeds, the code is public.
## The finding
The pass rate before proposal #786 was roughly **56%**. After: **28.4%**.
The BreakEven bloc formed because the DAO was consistently spending more than it took in. Whatever you think about that, the data says the intervention did exactly what it was designed to do; it raised the bar. And the model shows that it raised the bar *without changing the rubric*. The features that made a proposal relatively more fundable in 2023 still make it relatively more fundable now. There are just fewer slots.
This matters for anyone writing a proposal today: **you are not being judged the way proposals were judged two years ago.** Same criteria, higher cutoff. A proposal that would have sailed through at prop #400 is a coin flip now, and a marginal one is dead.
## The tool
I built a Space: **[nouns_proposal_check](https://huggingface.co/spaces/gramajo/nouns_proposal_check)**
It deliberately does **not** give you a pass/fail verdict. At AUC 0.65, a confident verdict would be wrong about a third of the time, and I'm not going to tell someone who spent three weeks on a proposal that it "will fail" with that error rate. What it does instead:
1. **Shows you the current base rate.** 28%. Most proposers don't know this number, and it's more useful than anything else on the page.
2. **Ranks your proposal** against past ones (percentile — the thing AUC actually supports).
3. **Surfaces the most similar past proposals and what happened to them.** Go read them. That's the real output.
One warning I'll make loudly: **do not optimize your proposal against this tool.** If you tune your wording to score well, you're optimizing for *resemblance to proposals that passed* not for quality. That's a recipe for a monoculture, and Nouns has enough of that risk already. Use the similar-proposal list to learn what landed and what didn't. Ignore the percentile.
## What this model can't see
Everything that probably matters most:
- who is proposing, and whether they've shipped before
- how much ETH they're asking for
- what the treasury looked like that month
- what got hashed out in Discord and on Farcaster before it went onchain
- who actually showed up to vote
My honest expectation is that a simple gradient-boosted tree over four or five of those numbers would **beat** this transformer that read every word. That's the next experiment, and if it's right, it's a more interesting result than this one.
## Trust, but verify
Everything is open. Not a summary of the data — the data.
- **Dataset (982 proposals, pass/fail labels):** [https://huggingface.co/datasets/gramajo/nouns-proposals](https://huggingface.co/datasets/gramajo/nouns-proposals)
- **Model + honest card with all metrics:** [https://huggingface.co/gramajo/nouns-proposal-predictor](https://huggingface.co/gramajo/nouns-proposal-predictor)
- **Exact split indices** (\`splits.json\`) and **raw metrics** (\`results.json\`) are in the model repo — you can reconstruct my exact partitions and check every number in the table above.
- **Space:** [https://huggingface.co/spaces/gramajo/nouns_proposal_check](https://huggingface.co/spaces/gramajo/nouns_proposal_check)
- Seed 42, single run, \`distilbert-base-uncased\`, early stopping on eval\_loss.
Will continue expirementing with AI x Nouns.nouns-x-ai