A repeatable prompt testing plan you'll actually follow

Testing is a session, not a mood

Most people test prompts accidentally — they notice something while trying to finish a song, half-remember it, and never confirm it. That's how folklore forms. A test session is different: you're not trying to make a good song, you're trying to answer one question, and the output is a sentence in a log file.

Separate the two activities. Creative sessions are for making things; test sessions are for learning things. Mixing them means you'll abandon the test the moment something sounds promising.

The practical benefit shows up faster than people expect. After three or four sessions you stop reaching for words you've proven do nothing, and your prompts get shorter. Shorter prompts generate more consistently, which makes the next test cleaner. The whole thing compounds in the right direction.

The protocol

• Write the question first, in one sentence, before generating anything.
• Define the control: the exact baseline prompt, unchanged.
• Define the variant: the control with exactly one difference.
• Run each condition twice. Four generations total.
• Listen in random order without looking at which is which.
• Write one sentence of result, including 'no difference' if that's the answer.
• Decide: adopt, reject, or retest with a stronger version of the change.

Questions worth testing

Listening blind, cheaply

You will hear what you expect to hear if you know which file is which. You don't need a lab for this — rename the four exports to random numbers, note the mapping in a file you close, shuffle them in a player, and write your impressions before reopening the mapping.

It takes two extra minutes and it routinely overturns conclusions. A surprising number of 'obviously better' prompt tweaks evaporate under blind listening, which is exactly the point: those are the ones that would have wasted months of your attention.

One caveat: blind listening works best for questions with an obvious audible answer — is the vocal further forward, is the tempo steadier. For genuinely subjective questions like 'is this better', blind testing yourself still helps, but a second listener helps more. Send the four files to someone with no stake in the outcome and ask which one they'd keep.

The log that compounds

Three lines, three durable facts. Twenty test sessions gives you a document that outperforms any prompt guide on the internet, because it's calibrated to your genres, your taste and the model you actually use.

date | question | control | variant | result | verdict
2026-07-09 | does 'close-mic' pull vocal forward | base v3 | base+close-mic | yes, clearly, both runs | adopt
2026-07-09 | does BPM number matter | base v3 no tempo | +104 BPM | tighter, more consistent tempo | adopt
2026-07-10 | is 15-line prompt better than 6 | base v3 | expanded | worse, vaguer, adjectives averaged out | reject

Keep the plan small enough to survive

• One question per session. Two if they're genuinely independent.
• Cap it at twenty minutes; long sessions get abandoned and never repeated.
• Retest anything important after a model update — findings expire.
• Record null results; they're half of what you learn and nobody writes them down.
• Re-derive your baseline every few months from what you've adopted.

A testing habit fails for the same reason every habit fails: it was too ambitious on day one. Twenty minutes and one question is small enough that you'll actually do it next week, which is the only property that matters.

FAQ