learn-ai-digital-human-with-phoebe / Session 5 of 6
Learn AI Digital Human with Phoebe · Session 5 of 6

When it is used against you

Four sessions in, you have been the one making the avatar. Tonight you are on the receiving end. Five documented cases, and every one of them arrives with a grade attached, because knowing which of these you may put in a board paper as fact is exactly as much of the lesson as the incidents themselves. You will also meet a widely quoted number that this course refuses to use, and find out why.

🟠 Hands-on Anyone who can authorise a payment or a credential reset And anyone who will be asked "is this real" at 6pm on a Friday 45 min
0-3 · Welcome 3-24 · Five cases, graded 24-40 · Build-along: the verification drill 40-45 · Q&A
Part 0

Two lessons running at once

This session teaches the misuse cases. It also teaches you to grade the evidence behind them, and the second lesson is the one that will still be useful in three years. Most published deepfake statistics do not survive being looked up. You are going to look them up.

🎫 Clear all three quiz questions to stamp your passport Live = presented in the 45 minutes Self-study = the full depth, after class

★ What you walk out with tonight. Five cases you can cite, each with the reason it earns its grade. One case that reads like the best story in the course and cannot be stated as fact. A control that costs nothing and comes straight from the FBI's own alert. And a two-number sentence that explains the entire shape of this problem.

LiveThe grading scale, and why a course bothers with one3 min▶

Every deck on this subject cites the same handful of incidents. Almost none of them say where the claim came from, and once you start checking, the pile separates cleanly into three heaps.

  • Grade A. A named victim organisation on the record, or the company publishing it about itself, or a dated report from a named research house with the finding quoted. Something you can hand to a regulator.
  • Grade B. A serious outlet, but the account rests on unnamed sources and the organisation declined to comment. Usable, if you say so out loud every single time.
  • Refuse. A figure with no traceable origin. It does not matter how often it is repeated or who repeated it last.
Why this matters more than it sounds

You will eventually stand in front of people who can approve spending, and you will be asked where a number came from. The moment one of your figures cannot be traced, the whole slide stops counting. Grading your own evidence before anyone else does is not pedantry. It is the only version of this work that survives contact with a sceptical CFO.

Self-studyWhat changes about your posture in this session2 min▶

Sessions 1 to 4 were about being the publisher: consent, likeness, disclosure, whether a face on screen helps anyone learn. Those are decisions you control.

This one is different. The attacker picked the tool, the timing and the channel. Nothing you decided in the first four sessions protects you here, and the defences that work turn out to have almost nothing to do with detecting synthetic media. They are about how a request is verified, not about how a face looks.

Part 1 · covers the grading ladder

Five cases, and what each one is actually worth

6 min live

Here is the whole session on one ladder. Read the right-hand column first, because the reason for the grade is the transferable part. The incidents will be replaced by newer incidents. The reasoning will not.

GRADE A Named victim or the company on the record, with independent confirmation Arup · 15 transfers to 5 accounts, HK$200 million · police confirmed LastPass · disclosed by LastPass itself the same day · no impact WPP · CEO to staff and a spokesperson, both on the record Graphika · a dated report with the finding quoted verbatim GRADE B A major outlet, unnamed sources, and the company declined to comment Ferrari · reported by Bloomberg · say reported by, never confirmed REFUSE No FBI or FinCEN dollar figure. Neither agency publishes one at all.
🔍 Click to zoom - the grade travels with the claim, or the claim does not travel
LiveArup, and the number everyone quotes4 min▶

January 2024. A finance employee at Arup joined a video conference in which the CFO and several colleagues were all synthetic, and made 15 transfers to 5 accounts totalling HK$200 million, about US$25 million.

Arup confirmed on the record that it "notified the police about an incident of fraud" and that "fake voices and images were used". Hong Kong Police confirmed the case and said it was the first bogus video conference call of its kind they had encountered.

Why this is Grade A. There is a named victim organisation making a statement about itself, plus a police force confirming the case independently. Two sources, neither of them a vendor with something to sell. That is the bar.

It is also the case that gets quoted most and taught worst. People reach for the dollar figure and stop there. The figure is the outcome, not the mechanism, and the mechanism is where the control goes.

LiveFerrari, which is the best story here and the weakest evidence4 min▶

An executive received a call from someone sounding like the CEO. He killed the attack by asking the caller to name the title of a book the CEO had recommended to him days earlier. The caller could not answer. That is a beautiful control, invented on the spot, and it is genuinely worth teaching.

★ How to say it, and this is the exact wording "Reported by Bloomberg: a Ferrari executive is said to have defeated a voice-cloning attempt by asking the caller to name a book the CEO had recommended days earlier. Ferrari declined to comment, and the account rests on unnamed sources."
✗ How it usually gets said, and why that is a problem "Ferrari confirmed that a deepfake attack on its CEO was stopped by a security question."

Ferrari confirmed nothing. Once you write "Ferrari confirmed" on a slide, you have manufactured a corporate statement that does not exist, and anybody who checks will find that out faster than you did.

The teaching moment. Grade B does not mean discard. It means the caveat is part of the claim and travels with it every time, in the deck, in the email, and out loud in the room. If a story is only impressive when you strip the caveat off it, it was never yours to tell.

Self-studyThe number this course will not put on a slide3 min▶

You will see decks citing "FBI data on deepfake losses". Go and look for it.

Both agencies did publish alerts. IC3 Alert I-120324-PSA is dated 3 December 2024. FinCEN FIN-2024-Alert004 is dated 13 November 2024. Neither publishes a loss total or an incident count for deepfake-enabled fraud. FinCEN says only that relevant suspicious activity reports have "increased", and a suspicious activity report is a suspicion, not a confirmed loss.

So any circulating federal dollar figure for deepfake fraud is unsourced. Not disputed, not contested, not "estimates vary". Unsourced. It has no document behind it.

Say the absence out loud. "There is no official loss figure, and the agencies that would publish one have not" is a stronger line in a board meeting than any invented number, because it is checkable and it is true. It also tells the room that you checked, which is the thing they are really assessing.

Part 2 · covers the anatomy

Where in the Arup sequence a control would have fitted

5 min live

Drawn as a sequence, the attack has an obvious seam. It is not at the point where the faces appear, because the faces were never checkable. It is one step later, at the request.

verification belongs here 1 A message arrives from the CFO 2 A video call is convened 3 Every other face on it is synthetic 4 Urgency, secrecy, a transfer request 5 15 transfers, 5 accounts, HK$200 million Everything before that line costs the attacker almost nothing. Everything after it is a wire transfer that has already left. Arup confirmed it "notified the police about an incident of fraud", and that "fake voices and images were used". Police called it the first of its kind.
🔍 Click to zoom - the employee could not check the faces, but the request was always checkable
LiveWhy the control never goes on the face4 min▶

Session 1 already settled this with evidence: across 56 papers and 86,155 participants, human detection of synthetic media ran at about 55 percent, with a confidence interval crossing chance. Asking an employee to spot a fake face is asking them to guess.

So a control placed at step 3 cannot work, no matter how much training you buy. A control placed at step 4 works every time, because it does not care whether the face was real. It asks a different question entirely: is this request legitimate, checked through a channel the attacker does not control.

The reframe worth carrying out of the room. This is not a detection problem wearing a fraud costume. It is a payment authorisation problem that has always existed, and synthetic video just removed the last informal check that used to sit on top of it, which was recognising a colleague's face.

Part 3 · covers the defence working

LastPass is worth more than Arup, and here is the argument

5 min live

Arup is the case everyone cites because of the number. LastPass is the case you should actually teach from, and the reason is simple: nothing was lost, so the record shows you the defence rather than the wreckage.

LiveWhat the LastPass employee actually noticed4 min▶

10 April 2024, disclosed by LastPass itself the same day. An employee "received a series of calls, texts, and at least one voicemail featuring an audio deepfake from a threat actor impersonating our CEO via WhatsApp".

The employee spotted two markers, ignored the contact, and reported it.

★ Marker one, and it has nothing to do with audio quality Contact arriving "outside of normal business communication channels".
★ Marker two, which is the oldest signal in fraud Forced urgency.

And the outcome, in the company's own words: "There was no impact to our company."

Why this beats Arup as a teaching case. Arup tells you what a loss looks like. LastPass tells you what the successful defence looked like from the inside, names the two things the human noticed, and comes from the company itself on the day. It is Grade A on the same test, and it carries an instruction rather than a warning.

Self-studyWPP, where the same two markers appear again3 min▶

May 2024. Attackers built a WhatsApp account from a public photo of CEO Mark Read, set up a Teams meeting, and used a voice clone plus YouTube footage while impersonating Read in the chat.

Read to staff: "Fortunately the attackers were not successful." A WPP spokesperson on the record: "Thanks to the vigilance of our people... the incident was prevented."

Note the construction of the attack. A public photo, a messaging account, publicly available video of the CEO speaking. Nothing about it required privileged access to WPP, and nothing about it required an expensive tool. Being a public-facing executive is the entire raw material.

Grade A, for the same reason as LastPass: the CEO and a named spokesperson are both on the record about their own company.

1 An urgent request on a channel you do not normally use 2 Do not evaluate the face or the voice. You cannot win there. 3 End it, then dial a number you already had for that person 4 Ask for the phrase you agreed before any of this happened IC3, IN ITS OWN WORDS "Create a secret word or phrase with your family to verify their identity." "Verify the identity ... by hanging up the phone ... call the number directly."
🔍 Click to zoom - every box after the first ignores the media entirely, which is why it works
Part 4 · covers the case that matters most here

Graphika, Wolf News, and 300 views

5 min live

This is the case that belongs in this specific course, because it is not fraud. It is a state-aligned influence operation using the exact consumer product you have spent four sessions learning to drive, at the exact consumer price, and the interesting part is that it failed.

LiveWhat Graphika found, quoted rather than paraphrased5 min▶

February 2023. Graphika, verbatim:

★ The finding, in the report's own words "this was the first time we observed a state-aligned operation promoting video footage of AI-generated fictitious people."

The pro-Chinese Spamouflage operation ran a fictitious outlet called Wolf News. Graphika traced its presenters by reverse image search to Synthesia stock avatars named "Anna" and "Jason", the same faces appearing in unrelated marketing videos. In one of those marketing videos, the presenter says:

✗ The same face, in a different job entirely "Hello, my name is Mr. Cruise. And I'm an avatar."

Two details make this the anchor case for the whole course. Subscriptions "start at $30 per month". And "none of the identified Spamouflage videos received more than 300 views".

Hold both halves of that. The barrier to a state-aligned actor producing synthetic news presenters was thirty dollars a month, which is genuinely alarming. And it did not work, which is genuinely useful. Production was never the hard part of an influence operation. Distribution and credibility were, and a stock avatar that also sells software in another video has neither.

Self-studyWhat "300 views" does and does not let you conclude3 min▶

It licenses one conclusion firmly: synthetic presenters did not, in this documented case, confer reach. Anyone arguing that AI video makes disinformation automatically effective has to explain this result.

It does not license complacency, and the reason is the same reverse image search that unmasked it. Those avatars were traceable because they were stock faces shared with paying customers. An operation willing to spend more, or to build its own likeness, removes that particular tell. The 300 views figure describes what happened to that operation, in that period, with those avatars.

Grade A, and worth saying why: a dated report from a named research house, with the finding quoted rather than summarised, and a vendor that confirmed on the record that the accounts were banned.

Build-along

The verification drill, run once in this room

★ 16 min · everyone builds

Nothing to install. This is a protocol you write down, and the only way to find out whether it works is to run it once while nobody is under pressure.

WHAT IT COSTS TO CLONE 3 sec of enrolled audio, unseen speaker Microsoft's VALL-E synthesises speech from a 3-second enrolled recording, keeping emotion and room acoustics. WHAT IT COSTS WHEN IT WORKS HK$200 million, about US$25 million Arup, from a single synthetic video conference US$6 million Federal forfeiture, New Hampshire robocall case Under 300 views Every Spamouflage video, at $30 per month Three seconds of audio on one side. On the other, all of that. The gap is not closing, so the control sits on the request, not the face.
🔍 Click to zoom - two numbers, and the whole shape of the problem sits between them

Pick the phrase, and pick it badly on purpose first. Someone will suggest a favourite film or a pet's name. Search that person publicly for sixty seconds and see how much of it you can find. The phrase has to be something that appears nowhere, which usually means it has to be arbitrary rather than meaningful.

Write down who holds it. A phrase known to forty people is a password on a whiteboard. Decide the smallest set: usually the people who can move money, plus the executives they would plausibly be asked to obey.

Decide the callback numbers now, not later. IC3's advice is explicit about hanging up and dialling a number you already have. That only works if the number exists somewhere trusted before the attack. If your only route to the CFO is the thread the attacker opened, you have no route.

Run it once, out loud, in this room. Two people, one plays the urgent CFO. The other executes steps 2, 3 and 4 of the flow above. It will feel absurd and slightly rude. That feeling is the actual attack surface, and rehearsing it once is what makes it survivable.

Write the sentence that makes it safe to be rude. Something like: "Anyone in this company may refuse any urgent request until they have called back, and nobody will be criticised for it." Without that sentence in writing from someone senior, your protocol dies the first time it is used on a genuinely impatient executive.

The design decision worth naming. Every step of this drill is deliberately media-blind. Nothing in it asks anyone to judge whether a face or a voice is real, because the evidence says people cannot do that, and a control built on a coin flip is not a control.

A pre-agreed verification phrase exists, and it is not findable onlineIC3: "Create a secret word or phrase ... to verify their identity."
Everyone who can move money knows to hang up and dial backAnd has a number that did not come from the message itself.
No payment above a stated threshold can be authorised on one channel aloneVideo counts as one channel. Arup shows why.
Refusing an urgent executive request is explicitly protected in writingOtherwise the protocol fails exactly when it is needed.
There is a route to report an attempt that failed, and someone reads itLastPass only has a Grade A case because the employee reported it.
Your executives know their public video and photos are the raw materialWPP was built from a public photo and YouTube footage.
Homework

Before session 6

35 min
  1. Grade three claims you already believe. Take any deck, article or vendor page about deepfake fraud and pick three claims. For each one, find the original source. Write A, B or Refuse next to it, and the one-line reason. Expect at least one to collapse.
  2. Chase the federal figure. Search for a specific FBI dollar figure for deepfake losses and try to find the document. Note how many pages cite it and how many link to a primary source. This exercise is more persuasive than being told.
  3. Count your CEO's minutes. How much public audio of your most senior person exists: interviews, webinars, earnings calls, conference video. Compare it to three seconds. Write down the number.
  4. Draft the phrase and the protection sentence. Both of them, in writing, one paragraph total. Name who holds the phrase and who signs the sentence.
  5. Find your callback list. If an urgent request came in from your CFO right now, on a channel you do not normally use, what number would you dial to check. If the answer takes more than thirty seconds to find, that is your first action item.
Check yourself

Three questions

Clear all three and session 5 gets stamped in your passport.

1 · You are writing a board paper. Which of these sentences is safe to put in it as stated?

C is Grade A: the victim organisation on the record plus independent police confirmation. A is Grade B wearing a Grade A costume, since Ferrari declined to comment and the account rests on unnamed sources reported by Bloomberg. B is unsourced, because neither IC3 nor FinCEN publishes a deepfake loss total at all.

2 · Why does this session argue that LastPass is a more valuable case than Arup, despite nothing being lost?

Arup tells you what a loss looks like. LastPass shows the defence working from the inside, names the two markers a human actually used, and comes from the company itself on the day, with the outcome stated as "There was no impact to our company." It carries an instruction rather than a warning.

3 · Graphika found that no identified Spamouflage video passed 300 views, while Synthesia subscriptions "start at $30 per month". What follows?

Hold both halves. Thirty dollars a month is a genuinely low barrier, and the campaign still failed. But the conclusion is narrow: those avatars were traceable because they were stock faces shared with paying customers, and a better-funded operation removes that tell. The finding describes that operation, in that period, with those avatars.

Sources covered

What this session teaches, and where it came from

✓Arup, January 2024Grade A. Arup on the record that it notified police and that fake voices and images were used, plus Hong Kong Police confirming the case as the first bogus video conference call of its kind
✓LastPass, 10 April 2024Grade A. Self-disclosed by LastPass the same day, quoted verbatim, including the two markers the employee noticed and "There was no impact to our company"
✓WPP, May 2024Grade A. CEO Mark Read to staff and a WPP spokesperson, both on the record, on an attack built from a public photo, a voice clone and YouTube footage
◐Ferrari, the book questionGrade B by design, and taught as the grading lesson. Reported by Bloomberg from unnamed sources. Ferrari declined to comment, so the page never says "Ferrari confirmed"
✓Graphika, "Deepfake It Till You Make It", February 2023Grade A. Wolf News, the Synthesia stock avatars "Anna" and "Jason", the $30 per month price point and the under-300-views finding, all quoted rather than paraphrased
✓IC3 Alert I-120324-PSA, 3 December 2024Quoted for its two recommendations only: the pre-agreed secret word or phrase, and hanging up and calling a researched number directly
◐No agency dollar figure appears on this page, deliberatelyIC3 Alert I-120324-PSA and FinCEN FIN-2024-Alert004 both exist, and neither publishes a loss total or an incident count for deepfake fraud. FinCEN states only that relevant SARs have "increased". Any circulating federal figure is unsourced, and the absence is taught as the lesson
◐The asymmetry figuresMicrosoft's VALL-E on 3 seconds of enrolled audio, set against the $6 million federal forfeiture in the New Hampshire robocall case. Both are cited as reported specifics rather than as measured loss data, and neither is an aggregate
◐Disclosure law, platform policy and provenanceNot covered here. Session 6 turns all of this into the policy your organisation has to write

Every fact on this page, and its verification tier, is recorded in the course's source map., including a dated list of what is deliberately not stated.

Session 5 cheat sheet · pin this

The grade travels with the claimA means the victim or a named authority on the record. B means say "reported by" out loud every time. No origin means refuse.
Arup is the loss, LastPass is the lessonHK$200 million across 15 transfers to 5 accounts. Versus two markers noticed and "no impact to our company".
Never write "Ferrari confirmed"Reported by Bloomberg, unnamed sources, company declined to comment. The book question is still a great control.
There is no federal loss figureIC3 and FinCEN both published alerts and neither published a number. FinCEN says only that SARs "increased".
$30 a month bought a state-aligned newsroomAnd it earned under 300 views per video. Production was never the hard part of an influence operation.
Three seconds is enoughSo the control goes on the request, never on the face. Hang up, dial a number you already had, ask for the phrase.