How tinygrad is taking on Nvidia, Google, and PyTorch with a tiny team, building in public with AMD, hot takes on ggml, Mojo, and GPT-4, and why AI Girlfriend is next.

Writeup and show notes: https://www.latent.space/p/geohot
Hosts' Twitter: @swyx and @fanahova

Timestamps:
00:00:00 - Introducing George
00:02:59 - Tinycorp's 3 Theses
00:11:12 - Tinygrad's creation
00:15:58 - Operation fusing in Tinygrad
00:19:11 - Tinygrad debugging
00:21:14 - Tiny Competitiveness on QCOMM vs NVDA
00:23:21 - geohot vs AMD
00:28:21 - Tinygrad vs ggml
00:30:01 - Importance of Good CI
00:30:37 - Mojo and Compatibility
00:32:43 - ggml quantization is made up
00:35:18 - tinygrad: benchmark int8 vs fp16
00:37:39 - Why you can't build tinybox
00:40:28 - The personal compute cluster
00:43:08 - Compute Optimal to Inference optimal
00:45:06 - Announcing FLOPcoin
00:46:23 - Why Federated AI won't work
00:47:38 - 5x faster than Nvidia
00:48:53 - A Person of Compute
00:49:49 - GPT-4's real architecture
00:51:07 - BatchNorm, FlashAttention
00:52:34 - The Bitter Lesson
00:55:31 - Hiring in the Age of AI
01:00:02 - Why AI doesn't replace developers & artists
01:03:02 - Comma Body
01:07:34 - AI Girlfriend
01:11:00 - The Goddess of Everything Else
01:13:43 - John Carmack Insights
01:17:41 - on Elon
01:18:47 - on e/acc
01:20:24 - Avatar 2