2024.08.10
A couple thoughts: I can really see the parallels between the "similar image morphing" of the Activation Atlases and what happens in my head in terms of dreams (or on certain drugs) - that feeling that semi-random noise is being fed through a system that looks for interpretations and meaning, and that meaning layer pushes back down into the visual.
I'm still astonished that it works so well. I mean it's just jaw dropping. Yes, it's not clear when or if it can truly figure out something new - in some ways all it's novelty is the "art of the remix" but what it uses to make new things is absolutely a form of embedded intelligence, albeit one that might not be modeling the world in the way it would need to take the next steps of "true" intelligence.
The video argues that it's the orders-of-magnitude increase in compute (ugh I hate that term as a noun) that unlocked the potential of an already known AI technique; that runs against some idea I've heard that we didn't need as much computation as we have on hand in order to apply these techniques... it would be interesting to see, knowing what we know now, what's the best the old hardware could have done? (Similar to people who make homebrew for old video game systems, pushing boundaries now that optimizations and tricks are more widely known)
Also I played with ChatGPT's "conversation" mode. Good lord is that uncanny.
There's an old saying AI, that compression is intelligence.
