How I trained our AI on thumbnails

Team tutorial for Marcus and Blendi · 2:32 · 1080p. What actually worked, checked against the data.

The method

  1. Test many methods side by side
    Comparing image models found the one that fixed the face; a Nov 4 test confirmed it (73% face match vs 4 to 33%).
  2. Point at the exact spot and name it
    Named, concrete defects got fixed: blown-out windows became a rule and were gone the next round; hero-size and object-doesn't-read notes dropped within about a round. 936 marks over 40 review sessions.
  3. Fix only what the note says
    Spreading one note to every thumbnail broke 13 he liked. Change the system only for a repeated note.
  4. Make a batch, and you pick
    About 1 in 5 versions passed his eye (22%). The AI matched his verdicts 59% of the time vs 56% for a random pick, so the picking stays human.
  5. Give concrete directions
    “Match the background to the foreground” changed the result in one round. Abstract rules for better concepts went 0 for 4.
  6. Hard rules only for what a check can catch
    Checkable rules held once tuned (likeness floor 0.72 to 0.75, no blown-out windows). Taste rules in the AI judge skipped 5 of his 28 starred thumbnails and were removed.
  7. Let the test decide
    Of 305 finished A/B tests, the current thumbnail stayed in about 85%. Judge by the test's watch-time result, not the view graph: the yoyo video's views were already rising before its test.

Where the first version of this advice was wrong

  1. Don't let the AI (or rules written for it) judge taste. It agreed with his picks 59% of the time, a random pick 56%. Your eye picks; the A/B test decides.
  2. Fix only what the note says, and only concrete things. Spreading a note broke liked thumbnails, abstract concept rules went 0 for 4, and hard rules only work for checkable defects.
  3. Results take many rounds and most new thumbnails lose. Backgrounds took about 17 reviewed rounds and are still not A/B-proven; the current thumbnail keeps its spot in about 85% of tests. The yoyo view jump was not proof (views were already rising); the test's watch-time win was.

What's real and what's rebuilt

Script, as built

Tap a time to jump the video there.

1. Where we started
0:00BEFORE round 1 Lambo, Nov 2 (“this doesn't really look like me”) / AFTER the AI-made thumbnail that beat the original in test 371, I Tested 100 Years of Tech (Nov 8, 53% vs 28% of watch time)

“At the start of November / our AI thumbnails looked like this / A week later / this one beat our original on / YouTube”
AI voiceover in his voice (voice G)

0:07How I trained our AI on thumbnails / What actually worked, checked against the data

“Here's what actually worked / checked against the data”
AI voiceover in his voice (voice G)

0:11First: two designers. One too slow · One not good enough · I needed versions all day

“First / designers / One was too slow / the other wasn't good enough / and I needed versions all day”
AI voiceover in his voice (voice G)

0:17Round 1, Nov 2 (too much text, things that don't read as what they are): arm (“This has way too much text”), minigun (“This doesn't really look like a minigun”)

“Then the AI had a go / Round 1 was rough / Too much text and things that / didn't look like what they were”
AI voiceover in his voice (voice G)

0:24Same Lambo idea: round 2 (“this person does not look like me”), round 3 (“it looks a little creepy”)

“Round two / same Lambo / still not me / Round three / better / but creepy”
AI voiceover in his voice (voice G)

0:30Many methods, side by side / A better image model + my picks of which faces looked like me

“We tested lots of methods side / by side / A better image model / and me picking which faces looked / like me / got it close in about five / rounds / It still slipped a few times / after”
AI voiceover in his voice (voice G)

0:40Round 4 from his review video: Mario TV, face made by the AI vs his real face pasted in (pasting was later dropped)

“I think likeability / meaning like how much it looks like me / is basically solved / Now it's just about the expressions / and the emotions”
Real recording: his Tella review of round 4 (M1 thumbnail-system/v2/review3/review.mp4, 0:40)

2. How I review
0:51His real background round 4 review page with all his marks, scrolling (rebuilt from the recording)

“Every round / I got a review page / I scroll / talk / and draw right on what's wrong”
AI voiceover in his voice (voice G)

0:57Replay of his background round 4 review: his page, his recorded scroll, his pen strokes drawing as he drew them, his voice

“Not that one / Not that one / The colors are off / That one the colors are off again / This one color too vivid / This one color too dark / This one a little bit too much detail / in the background / with all the trees and the towers”
Real recording: his page review of background round 4, Nov 12 (Studio session 20261112T200019, notes 2 to 8), replayed on his page with his scroll and pen

1:14His real review pages with all marks: Nov 6 A/B batch, blind pick, background round 8. Counter: 936 marks, 40 review sessions

“Over 900 marks across 40 review / sessions”
AI voiceover in his voice (voice G)

1:20BEFORE his mark on the round 14b booth window (“I don't like these blown-out windows”) / AFTER round 14c, window gone

“Naming the exact thing moved it / Blown out windows became a rule / and the next round / they were gone”
AI voiceover in his voice (voice G)

1:27BEFORE fake-look round, relit orange (“you're putting an entire color cast on the thing”) / AFTER the place matched to him (“I think that one's the best”)

“Same with directions only I could / give / Match the background to me / not me to the background / That changed it in one round”
AI voiceover in his voice (voice G)

3. What didn't work
1:34Letting the AI judge: 59% matched my picks vs 56% a random pick. Rules for better ideas: 0 of 4 worked

“What didn't work? / Letting the AI judge / It matched my picks 59% of / the time / A random pick got 56% / Rules for better ideas didn't work / either”
AI voiceover in his voice (voice G)

4. Let the test decide
1:45305 finished A/B tests (to Nov 17): about 85% the current thumbnail stays, 15% a loop thumbnail wins

“Since November 6 / The loop runs A/B tests on / YouTube by itself / The current thumbnail keeps its spot / in about 85% of tests”
AI voiceover in his voice (voice G)

1:54Judge by the test, not the view graph: yoyo original (28% of watch time) / winner (41% of watch time)

“Judge by the test / not the view graph / This yoyo thumbnail won 41% of / watch time to 28 / but its views were already climbing / before the test”
AI voiceover in his voice (voice G)

5. The method
2:04The method: 7 steps

“Here's the method / Test many methods side by side / Point at the exact spot and / name it / Fix only what the note says / Make a batch and you pick / Give concrete directions / Hard rules only for things a / check can catch / And let the test decide”
AI voiceover in his voice (voice G)

2:20Your biggest problem area, round after round: Marcus, composition. Blendi, sound (no A/B test for sound: ears and checks are the judge)

“Marcus / composition / Blendi / sound / where your ears and checks are / the judge / Expect lots of rounds / Stay on your biggest problem / inside your normal work”
AI voiceover in his voice (voice G)