How I trained our AI on thumbnails
Team tutorial for Marcus and Blendi · 2:32 · 1080p. What actually worked, checked against the data.
The method
- Test many methods side by side
Comparing image models found the one that fixed the face; a Nov 4 test confirmed it (73% face match vs 4 to 33%). - Point at the exact spot and name it
Named, concrete defects got fixed: blown-out windows became a rule and were gone the next round; hero-size and object-doesn't-read notes dropped within about a round. 936 marks over 40 review sessions. - Fix only what the note says
Spreading one note to every thumbnail broke 13 he liked. Change the system only for a repeated note. - Make a batch, and you pick
About 1 in 5 versions passed his eye (22%). The AI matched his verdicts 59% of the time vs 56% for a random pick, so the picking stays human. - Give concrete directions
“Match the background to the foreground” changed the result in one round. Abstract rules for better concepts went 0 for 4. - Hard rules only for what a check can catch
Checkable rules held once tuned (likeness floor 0.72 to 0.75, no blown-out windows). Taste rules in the AI judge skipped 5 of his 28 starred thumbnails and were removed. - Let the test decide
Of 305 finished A/B tests, the current thumbnail stayed in about 85%. Judge by the test's watch-time result, not the view graph: the yoyo video's views were already rising before its test.
Where the first version of this advice was wrong
- Don't let the AI (or rules written for it) judge taste. It agreed with his picks 59% of the time, a random pick 56%. Your eye picks; the A/B test decides.
- Fix only what the note says, and only concrete things. Spreading a note broke liked thumbnails, abstract concept rules went 0 for 4, and hard rules only work for checkable defects.
- Results take many rounds and most new thumbnails lose. Backgrounds took about 17 reviewed rounds and are still not A/B-proven; the current thumbnail keeps its spot in about 85% of tests. The yoyo view jump was not proof (views were already rising); the test's watch-time win was.
What's real and what's rebuilt
- Real recordings of Hafu: his round 4 Tella review line and his background round 4 page review (voice). The narration is his AI voice (voice G).
- Rebuilt from his recordings: the review pages are rebuilt from the exact page version he reviewed, at his window width; every red stroke is his recorded pen data on the picture he drew on. The replay follows his recorded scroll positions and stroke times.
- Real pictures: every thumbnail is a real output from that round or test; quotes are his words, verbatim.
Script, as built
Tap a time to jump the video there.
| 1. Where we started | ||
| 0:00 | BEFORE round 1 Lambo, Nov 2 (“this doesn't really look like me”) / AFTER the AI-made thumbnail that beat the original in test 371, I Tested 100 Years of Tech (Nov 8, 53% vs 28% of watch time) | “At the start of November / our AI thumbnails looked like this / A week later / this one beat our original on / YouTube” |
| 0:07 | How I trained our AI on thumbnails / What actually worked, checked against the data | “Here's what actually worked / checked against the data” |
| 0:11 | First: two designers. One too slow · One not good enough · I needed versions all day | “First / designers / One was too slow / the other wasn't good enough / and I needed versions all day” |
| 0:17 | Round 1, Nov 2 (too much text, things that don't read as what they are): arm (“This has way too much text”), minigun (“This doesn't really look like a minigun”) | “Then the AI had a go / Round 1 was rough / Too much text and things that / didn't look like what they were” |
| 0:24 | Same Lambo idea: round 2 (“this person does not look like me”), round 3 (“it looks a little creepy”) | “Round two / same Lambo / still not me / Round three / better / but creepy” |
| 0:30 | Many methods, side by side / A better image model + my picks of which faces looked like me | “We tested lots of methods side / by side / A better image model / and me picking which faces looked / like me / got it close in about five / rounds / It still slipped a few times / after” |
| 0:40 | Round 4 from his review video: Mario TV, face made by the AI vs his real face pasted in (pasting was later dropped) | “I think likeability / meaning like how much it looks like me / is basically solved / Now it's just about the expressions / and the emotions” |
| 2. How I review | ||
| 0:51 | His real background round 4 review page with all his marks, scrolling (rebuilt from the recording) | “Every round / I got a review page / I scroll / talk / and draw right on what's wrong” |
| 0:57 | Replay of his background round 4 review: his page, his recorded scroll, his pen strokes drawing as he drew them, his voice | “Not that one / Not that one / The colors are off / That one the colors are off again / This one color too vivid / This one color too dark / This one a little bit too much detail / in the background / with all the trees and the towers” |
| 1:14 | His real review pages with all marks: Nov 6 A/B batch, blind pick, background round 8. Counter: 936 marks, 40 review sessions | “Over 900 marks across 40 review / sessions” |
| 1:20 | BEFORE his mark on the round 14b booth window (“I don't like these blown-out windows”) / AFTER round 14c, window gone | “Naming the exact thing moved it / Blown out windows became a rule / and the next round / they were gone” |
| 1:27 | BEFORE fake-look round, relit orange (“you're putting an entire color cast on the thing”) / AFTER the place matched to him (“I think that one's the best”) | “Same with directions only I could / give / Match the background to me / not me to the background / That changed it in one round” |
| 3. What didn't work | ||
| 1:34 | Letting the AI judge: 59% matched my picks vs 56% a random pick. Rules for better ideas: 0 of 4 worked | “What didn't work? / Letting the AI judge / It matched my picks 59% of / the time / A random pick got 56% / Rules for better ideas didn't work / either” |
| 4. Let the test decide | ||
| 1:45 | 305 finished A/B tests (to Nov 17): about 85% the current thumbnail stays, 15% a loop thumbnail wins | “Since November 6 / The loop runs A/B tests on / YouTube by itself / The current thumbnail keeps its spot / in about 85% of tests” |
| 1:54 | Judge by the test, not the view graph: yoyo original (28% of watch time) / winner (41% of watch time) | “Judge by the test / not the view graph / This yoyo thumbnail won 41% of / watch time to 28 / but its views were already climbing / before the test” |
| 5. The method | ||
| 2:04 | The method: 7 steps | “Here's the method / Test many methods side by side / Point at the exact spot and / name it / Fix only what the note says / Make a batch and you pick / Give concrete directions / Hard rules only for things a / check can catch / And let the test decide” |
| 2:20 | Your biggest problem area, round after round: Marcus, composition. Blendi, sound (no A/B test for sound: ears and checks are the judge) | “Marcus / composition / Blendi / sound / where your ears and checks are / the judge / Expect lots of rounds / Stay on your biggest problem / inside your normal work” |