Every AI product that reads lyrics says it gives feedback. Almost none of them can tell you how often that feedback is wrong. We can, because we built a harness that runs every one of Coach's critique prompts against lyric samples with a planted weakness, scores each reply on a keyword rubric, and hands the lyric-review replies to a second, stronger model to judge out of ten. The last committed run had 114 critiques in it. These are the numbers, and the failure modes are the useful part.
The headline
1,109 of 1,279 rubric points, 86.7 percent. The four lyric-review prompts (line feedback, full diagnosis, coach chat, scaffolded feedback) scored 90.6 percent over 76 runs. The judge gave those critiques a mean of 7.4 out of 10, caught the planted weakness 85 percent of the time, matched the depth to the student's belt 93 percent of the time, and stayed inside the student's genre 97 percent of the time. The two grading utilities (teach-back and song teardown) scored 79 percent, which is why those are the prompts we work on next.
Now the part a chatbot will not tell you. The single most common failure, 45 of the checks, was a critique that did not quote the line it was criticising. It said "the second verse relies on abstraction" without pointing at a word. That reads as insight and is almost useless to a writer, because you cannot rewrite a paragraph of opinion; you can rewrite a quoted line. Coach is now required to quote, and a claim that quotes a line which does not exist in the lyric is dropped before you see it. In an earlier run the model attributed a real song to two different wrong artists inside the same batch. Citations are now checked against a verified table and corrected in place.
The second failure, 23 checks, was missing the planted weakness altogether: a lyric written with a deliberate second-verse collapse into emotion labels came back with notes on rhyme scheme. The third, 18 checks, was length: replies that ran past the ceiling for the student's level. The fourth, 15 checks, was scoring that was too kind for the belt. A white-belt lyric built from clichés is a 4, not a 7, and the model wants to give a 7.
The rarest failures are the ones that worried us most. Once in this run, and four times in the run before it, the critique praised the exact thing it was hired to catch: "an earned escalation" for a second verse that was planted to collapse. Twice it manufactured a flaw in a clean lyric, because a critique with no weaknesses feels like a failure to a model trained to be helpful. Once it graded strong work as ordinary. Each of these now has its own check, and the praise trap check is the one we watch: a wrong note is recoverable, inverted praise sends you confidently in the wrong direction.
There is also the failure you never see
truncation. When the output budget was too small, the deepest critiques (purple belt and above, because those are the longest) were cut off mid-JSON and rendered as an empty diagnosis. Thirteen of sixty runs in the first baseline. It is zero now, and a truncated reply is refused outright rather than shown as a hollow read. If a tool ever hands you a critique with no notes, that is what happened.
What this means when you paste a lyric into any model: ask it to quote the line. Refuse a note that does not. Read the praise as carefully as the criticism, and be suspicious when the praise lands on the part you were unsure about. Expect it to be kinder than a co-writer would be. Expect it to lose the thread on a long song and give it one section at a time. None of that is a reason not to use the tool; it is the reason we score ours.
The drill on this page is the one most of those planted weaknesses were built around: show, do not tell. Every critique in the harness that scored 11 out of 11 did the same thing the drill asks of you, it found the line that names a feeling and replaced it with something you could photograph.
