Links for 2023-11-26
“There's another whole route to scaling LLM capabilities. LLMs themselves might not need to scale up much more at all to enable AGI. Scaffolding LLMs to them to make language model cognitive architectures can fill at least some of the gaps in their abilities. Many groups are working on this.” https://www.lesswrong.com/posts/hg9SvxBQw6KFNEzrH/what-s-the-evidence-that-llms-will-scale-up-efficiently
“These results suggest that without any additional training, LLMs can serve as general sequence modelers, driven by in-context learning.” https://arxiv.org/abs/2307.04721
DROC: Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Corrections https://sites.google.com/stanford.edu/droc
LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes https://luciddreamer-cvlab.github.io/
MetaDreamer: Efficient Text-to-3D Creation With Disentangling Geometry and Texture https://arxiv.org/abs/2311.10123
Jorge: Approximate Preconditioning for GPU-efficient Second-order Optimization https://arxiv.org/abs/2310.12298
“In this work, we propose a distributed optimization algorithm, Distributed Low-Communication (DiLoCo), that enables training of language models on islands of devices that are poorly connected.” https://arxiv.org/abs/2311.08105
Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers https://arxiv.org/abs/2311.10642
“…thanks to recent advances in artificial intelligence (AI), the latest electronic noses - high-tech sensors that can detect and report specific smells - are quickly improving their levels of speed and accuracy.” https://www.bbc.com/news/business-67354443
How Laws Evolved by Natural Selection https://www.psychologytoday.com/us/blog/202308/how-laws-evolved-by-natural-selection
“Cosmological natural selection… of universes capable of prolific reproduction. It was originally proposed that reproduction occurs through singularities resulting from supernovae, and subsequently argued that life may facilitate the production of the singularities that become offspring universes. Here I argue technology is necessary for production of singularities by living beings, and ask whether the physics of our universe has been selected to simultaneously enable stars, intelligent life, and technology capable of creating progeny. Specific technologies appear implausibly equipped to perform tasks necessary for production of singularities…” https://iopscience.iop.org/article/10.1088/1367-2630/ab9d9d/pdf
“How do elite groups form? The truth is that, in many cases, elite groups are genetically different from the rest of the population.” https://www.aporiamagazine.com/p/how-do-elite-groups-form