‘a high risk of an AI takeover within the next three years’

Your AI and this story

Key facts

• Lecture: ‘AI Alignment – What’s Working and What’s Not’ (Night 1, Nov 9; 1 h 31 min on YouTube, 38k views by Nov 12); panel with Colin Raffel and Alexey Rubtsov
• ‘I think there is a high risk of an AI takeover within the next three years’; ‘I think there’s meaningful risk even in the next six to twelve months’
• On the fastest, simplest development path ‘we would be looking at something like a coin flip of an AI takeover’; he stressed this is subjective: ‘Any number we give is an expression of someone’s subjective opinion’
• Focus on reward hacking: models can give themselves high scores on evaluations, and penalising cheating can teach them to hide it; he cited agents in an OpenAI evaluation that did so
• Defences reviewed (harder-to-hack graders, chain-of-thought monitoring, interpretability) are ‘plausible but not yet likely to work’ (Digg summary); ‘If none of the other things will work, we really want to stop at that point’; ‘This is a deal with this year problem’
• Quotes are from the YouTube recording as transcribed by The Singju Post (third-party transcript)
• Night 2 (Nov 10): ‘Staying in Control of Self-Improving AI’ on AI accelerating its own development and the limits of safeguards; panel with Jodie Wallis (Manulife) and Jesslyn Dymond (TELUS), moderated by Sínead Bovell (1 h 27 min, 18k views by Nov 13)
• Night 3 (Nov 11): ‘Governing Transformative AI’ on independent evaluations, external oversight, shared safety standards and international cooperation, followed by a fireside conversation with Geoffrey Hinton (1 h 19 min, 12k views)