Links for 2024-07-30

AI:

1. Claim: Continuous Learning Model (CLM) by Topology — “The CLM is a new model that remembers interactions, learns skills autonomously, and thinks in its free time, just like humans.” You can try it yourself here: https://topologychat.com/

2. Baidu presents an end-to-end self-reasoning framework to improve the reliability and traceability of RAG systems. https://arxiv.org/abs/2407.19813

3. MindSearch is an open-sourced AI search engine framework, with comparable performance with Perplexity.ai Pro. Deploy your own Perplexity.ai style search engine! https://mindsearch.netlify.app/

4. AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic https://garymarcus.substack.com/p/alphaproof-alphageometry-chatgp

5. “AI existential risk probabilities are too unreliable to inform policy” https://www.aisnakeoil.com/p/ai-existential-risk-probabilities

6. Small Molecule Optimization with Large Language Models https://arxiv.org/abs/2407.18897

7. “Introducing Meta Segment Anything Model 2 (SAM 2) — the first unified model for real-time, promptable object segmentation in images & videos. SAM 2 is available today under Apache 2.0 so that anyone can use it to build their own experiences.” https://ai.meta.com/blog/segment-anything-2/

8. Constrained-CoT: Constraining the reasoning of LLaMA2-70b to 100 words improves the accuracy from 36.01% (CoT) to 41.07% (CCoT) on GSM8K. https://arxiv.org/abs/2407.19825

9. Theia: Distilling Diverse Vision Foundation Models for Robot Learning http://theia.theaiinstitute.com/

10. “SearchGPT has the ‘best shot at changing the search paradigm as we’ve known it for 25 years’” https://www.tomsguide.com/ai/chatgpt/searchgpt-has-the-best-shot-at-changing-the-search-paradigm-as-weve-known-it-for-25-years

11. How This Brain Implant Is Using ChatGPT https://www.cnet.com/tech/computing/how-this-brain-implant-is-using-chatgpt/

12. "The Virtue of Complexity in Return Prediction", Kelly et al 2023 (large models can be profitable even with negative R^2) https://onlinelibrary.wiley.com/doi/full/10.1111/jofi.13298

13. "A Visual Guide to Quantization: Demystifying the Compression of Large Language Models", Maarten Grootendorst 2024 https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-quantization

Registering a prediction: I predict that within two years (by July 2026) we'll see an AI system beat all humans at the IMO, obtaining the top score. Alongside this, I would wager we'll see the same thing - an AI system beating all humans in a known-hard competition - in another scientific domain outside of mathematics. If both of those things occur, I believe that will present strong evidence that AI may successfully automate large chunks of scientific research before the end of the decade.

— Jack Clark, co-founder of Anthropic

Compute and algorithmic improvement:

1. Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget — “They trained BERT and outperformed the original using a single GPU in a single day. Their best model used around 1e19 FLOP compared to around 6e20 FLOP for the original BERT. So they are ~60x more compute-efficient after 4 years, consistent with roughly doubling algorithmic efficiency every 8 months.” https://arxiv.org/abs/2407.15811

2. Cambridge researchers show how to use distributed training to make a 1.3bn parameter LLM https://arxiv.org/abs/2405.10853

3. OpenAI will spend $3 billion this year on training new models https://nypost.com/2024/07/25/business/openai-may-lose-5b-this-year-alone-on-chatgpt-costs-report/ [“that implies at least one model trained on more data than a reasonable snapshot of the internet would contain” https://x.com/teortaxesTex/status/1817634603483439175]

4. 100 Petaflop AI Chip and 100 Zettaflop AI Training Data Centers in 2027 https://www.nextbigfuture.com/2024/07/100-petaflop-ai-chip-and-100-zettaflop-ai-training-data-centers-in-2027.html

5. Apple Foundational Models, both on device and server, were fully trained on Google TPU clusters. https://machinelearning.apple.com/research/apple-intelligence-foundation-language-models

6. Just four companies are hoarding tens of billions of dollars worth of Nvidia GPU chips https://sherwood.news/tech/companies-hoarding-nvidia-gpu-chips-meta-tesla/

Space:

1. The discovery of a possible sign of life in Venus’ clouds sparked controversy. Now, scientists say they have more proof https://edition.cnn.com/2024/07/29/science/venus-gases-phosphine-ammonia/index.html

2. “plants found on Earth could even survive the harsh conditions of the Red Planet. One such planet, a type of moss found in arid locales like Tibet & Antarctica, survived rigorous testing, including deep freezing and high radiation" https://www.popularmechanics.com/space/moon-mars/a61668800/moss-from-earth-can-survive-on-mars/

Miscellaneous:

1. Why controlling for variables is insufficient — On the pervasiveness of residual confounding in the social sciences, how to think about it, and what to do https://inquisitivebird.xyz/p/why-controlling-for-variables-is

2. How a Mind-Controlling Parasite Could Deliver Medicine to the Brain https://singularityhub.com/2024/07/29/how-a-mind-controlling-parasite-could-deliver-medicine-to-the-brain/

Ukraine:

1. “My team has been working on a comprehensive update on that area, but the Russians have progressed so quickly multiple times that we had to postpone the report to include the latest updates, redo the maps, and add new details.” https://x.com/Tatarigami_UA/status/1817939292905156746