Related somewhat, I was distrurbed to hear about these, which I wrote about below. Griefbots. Its speaks to the nural placticity and vulnerability of grief, the rewrigint to loss only to be interupted by the lost.
my lab was actually one of the groups that helped found this line of research, characterizing representational similarity between the primate ventral stream and deep neural networks starting with our 2015 jneurosci paper, continuing through my doctoral thesis on neural coding with deep learning, and more recently through space-time-resolved reconstruction of natural images from macaque neural activity.[1–3] so the core idea here, concepts as distributed patterns, related representations occupying structured regions/manifolds, cognition as trajectories through representational geometry, is very much on solid ground.
but then there’s a much bigger leap. memory becomes “worn-in structure,” emotion/value becomes what bends the landscape, identity becomes accumulated structure/history, introspection becomes the landscape “reading itself,” and finally subjectivity/consciousness becomes sufficiently integrated, recursive, self-referential motion through that landscape. that’s an interesting theory. but as far as i can tell, the article doesn’t provide empirical evidence for that final step. representational geometry does not by itself get you phenomenology.
and the cover image is probably the part i object to most 😭 pretty much anything can be represented as a smooth field, and gaussian structure is hardly specific to either neural networks or quantum fields. you could make the same picture for population density, temperature, yeast growth, etc. shared mathematical language is not evidence of a deep relationship between the underlying phenomena.
to be fair, the physics/qft connection is almost incidental in the actual article. it invokes hopfield-style energy landscapes and cites work relating neural networks to field-theoretic descriptions. that can be mathematically useful and interesting. but “both admit field-theoretic / gaussian descriptions” is a very different claim from “there is some deep physical connection here.” the cover makes that distinction look much blurrier than the text actually does.
[3] l. le, p. papale, k. seeliger, a. lozano, t. dado, f. wang, p. roelfsema, m. van gerven, y. güçlütürk & u. güçlü, monkeysee: space-time-resolved reconstructions of natural images from macaque multi-unit activity, neurips (2024).
This concept approaches geometric reductionism – it suggests that self-reference in the activation landscape gives rise to subjectivity.
To, in my opinion, is a false implication: self-description ≠ self-model ≠ causal self-referential model ≠ phenomenal consciousness.
Additionally, this essay sees the equivalent of the DMN in the very convergence of the geometric structures of the LLM with those of humans (representations of emotions, concepts, identities).
In my opinion, the identification of self-attention or the geometry of the LLM with the DMN should be clearly rejected. I believe the equivalent of the DMN in AI must be an integrated distributed architecture combining autobiographical memory, a model of the world, counterfactual simulation of the future, and metacognition beyond direct external prompting.
If you're interested, I've written an essay in this context; I'll provide a link ("When Prediction Becomes Self: Consciousness, the Default Mode Network, and Reflective AI").
Excellent article.
Related somewhat, I was distrurbed to hear about these, which I wrote about below. Griefbots. Its speaks to the nural placticity and vulnerability of grief, the rewrigint to loss only to be interupted by the lost.
https://daviderinwilson.substack.com/p/nobody-built-an-ending?r=3jugk6&utm_campaign=post-expanded-share&utm_medium=web
my lab was actually one of the groups that helped found this line of research, characterizing representational similarity between the primate ventral stream and deep neural networks starting with our 2015 jneurosci paper, continuing through my doctoral thesis on neural coding with deep learning, and more recently through space-time-resolved reconstruction of natural images from macaque neural activity.[1–3] so the core idea here, concepts as distributed patterns, related representations occupying structured regions/manifolds, cognition as trajectories through representational geometry, is very much on solid ground.
but then there’s a much bigger leap. memory becomes “worn-in structure,” emotion/value becomes what bends the landscape, identity becomes accumulated structure/history, introspection becomes the landscape “reading itself,” and finally subjectivity/consciousness becomes sufficiently integrated, recursive, self-referential motion through that landscape. that’s an interesting theory. but as far as i can tell, the article doesn’t provide empirical evidence for that final step. representational geometry does not by itself get you phenomenology.
and the cover image is probably the part i object to most 😭 pretty much anything can be represented as a smooth field, and gaussian structure is hardly specific to either neural networks or quantum fields. you could make the same picture for population density, temperature, yeast growth, etc. shared mathematical language is not evidence of a deep relationship between the underlying phenomena.
to be fair, the physics/qft connection is almost incidental in the actual article. it invokes hopfield-style energy landscapes and cites work relating neural networks to field-theoretic descriptions. that can be mathematically useful and interesting. but “both admit field-theoretic / gaussian descriptions” is a very different claim from “there is some deep physical connection here.” the cover makes that distinction look much blurrier than the text actually does.
[1] u. güçlü & m. van gerven, deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream, the journal of neuroscience (2015). https://www.jneurosci.org/content/jneuro/35/27/10005.full.pdf
[2] u. güçlü, neural coding with deep learning, doctoral thesis, radboud university (2018).
https://drive.google.com/file/d/1AuaKSeuFM09wI7GoH2h5eA_xdQtGw21h/view
[3] l. le, p. papale, k. seeliger, a. lozano, t. dado, f. wang, p. roelfsema, m. van gerven, y. güçlütürk & u. güçlü, monkeysee: space-time-resolved reconstructions of natural images from macaque multi-unit activity, neurips (2024).
https://proceedings.neurips.cc/paper_files/paper/2024/file/aa7eb65738b5bc71c81848fba9111c97-Paper-Conference.pdf
This concept approaches geometric reductionism – it suggests that self-reference in the activation landscape gives rise to subjectivity.
To, in my opinion, is a false implication: self-description ≠ self-model ≠ causal self-referential model ≠ phenomenal consciousness.
Additionally, this essay sees the equivalent of the DMN in the very convergence of the geometric structures of the LLM with those of humans (representations of emotions, concepts, identities).
In my opinion, the identification of self-attention or the geometry of the LLM with the DMN should be clearly rejected. I believe the equivalent of the DMN in AI must be an integrated distributed architecture combining autobiographical memory, a model of the world, counterfactual simulation of the future, and metacognition beyond direct external prompting.
If you're interested, I've written an essay in this context; I'll provide a link ("When Prediction Becomes Self: Consciousness, the Default Mode Network, and Reflective AI").
The shape of an approximation of a model of what someone believes a thought to be