Links for 2024-07-11
AI:
“Harmonic is continuing to make progress toward mathematical superintelligence” https://www.harmonic.fun/news
AI Math Olympiad winner is now Open Source! Introducing NuminaMath-7B-TIR, the small but mighty model that won the first progress prize of the AI Math Olympiad 🥇! https://huggingface.co/AI-MO/NuminaMath-7B-TIR (Demo: https://huggingface.co/spaces/AI-MO/math-olympiad-solver)
“Can Transformers extrapolate from short training sequences to long ones? Our new work shows that they display surprising “length generalization” capabilities on many algorithmic tasks: addition, multiplication, and even in-context simulation of SGD!” https://arxiv.org/abs/2407.03310
Mixture of A Million Experts — “This paper introduces PEER (parameter efficient expert retrieval), a novel layer design that utilizes the product key technique for sparse retrieval from a vast pool of tiny experts (over a million).” https://arxiv.org/abs/2407.04153
Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence https://arxiv.org/abs/2407.07061
“Meet Salesforce Einstein “Tiny Giant.” Our 1B parameter model xLAM-1B is now the best micro model for function calling, outperforming models 7x its size, including GPT-3.5 & Claude. On-device agentic AI is here.” https://apigen-pipeline.github.io/
Microsoft has unveiled a new method called MInference that can reduce LLM processing time by up to 90% for inputs of one million tokens (equivalent to about 700 pages of text) while maintaining accuracy. https://github.com/microsoft/MInference
Reasoning in LLMs: A Geometric Perspective — “We demonstrate through theoretical analysis and toy examples that a higher intrinsic dimension implies a greater expressive capacity of the LLM.” https://arxiv.org/abs/2407.02678
Google presents On scalable oversight with weak LLMs judging strong LLMs https://www.lesswrong.com/posts/Qn3ZDf9WAqGuAjWQe/on-scalable-oversight-with-weak-llms-judging-strong-llms
Towards shutdownable agents via stochastic choice https://www.lesswrong.com/posts/dzvnAGDPsisMY8h7b/towards-shutdownable-agents-via-stochastic-choice
Pantheon Interface: 1. A human user “thinks out loud” by typing out their thoughts one at a time. This leaves a text trace of their stream of thought. 2. AI characters (called daemons) read this trace, and interact with the user by responding asynchronously with comments and questions. https://www.lesswrong.com/posts/JHsfMWtwxBGGTmb8A/pantheon-interface
Distilling System 2 into System 1 https://arxiv.org/abs/2407.06023
MIT researchers introduce generative AI for databases https://news.mit.edu/2024/mit-researchers-introduce-generative-ai-databases-0708
Learning to (Learn at Test Time): RNNs with Expressive Hidden States https://arxiv.org/abs/2407.04620 (See also: Revisiting Dynamic Evaluation: Online Adaptation for Large Language Models https://arxiv.org/abs/2403.01518)
Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision https://orrzohar.github.io/projects/video-star/
How good are LLMs at figuring out what they are and what we are doing to them? It varies a lot. Some ingenious tests to check for it. https://situational-awareness-dataset.org/
Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps https://arxiv.org/abs/2407.07071
How Google Project Zero got 20x improvements on having models exploit buffer overflows and memory corruption https://googleprojectzero.blogspot.com/2024/06/project-naptime.html
Amazon is hiring execs from Adept onto the company’s 'AGI Autonomy' team https://www.adept.ai/blog/adept-update
Building an AI assistant that listens and sees the world (Step by step tutorial) https://www.youtube.com/watch?v=zVttVCQvACQ
Tech Industry Wants to Lock Up Nuclear Power for AI https://www.wsj.com/business/energy-oil/tech-industry-wants-to-lock-up-nuclear-power-for-ai-6cb75316 [no paywall: https://archive.is/4EXD6]
“But now things are understood “industrially,” so to speak. Companies spend huge amounts of money on training runs, and feel secure doing so, because they know that you get out what you put in, without surprises in either direction.” https://nostalgebraist.tumblr.com/post/741247180226052096/i-dont-think-youre-drawing-the-right-lesson-from
According to the Society of Authors, generative AI has already caused 26% of illustrators and 36% of translators to lose work. https://www2.societyofauthors.org/2024/04/11/soa-survey-reveals-a-third-of-translators-and-quarter-of-illustrators-losing-work-to-ai/
The Chinese government is going all-in on autonomous vehicles https://www.technologyreview.com/2024/07/10/1094811/chinese-government-policy-autonomous-vehicles/ [no paywall: https://archive.is/ph0q9]
AI War:
Destroying Russian Tanks Is Just The Start For U.S. AI Drone Autopilot — “The guidance system provides optical lock-on: the operator identifies the target and flags it for the autopilot while the drone is well outside jamming range. Then it can carry on through the ‘jamming bubble’ even if the operator loses contact…This means the two biggest causes of a miss—pilot error and jamming—can be eliminated.” https://www.forbes.com/sites/davidhambling/2024/07/10/destroying-russian-tanks-is-just-the-start-for-us-ai-drone-autopilot/
He created Oculus headsets as a teenager. Now he makes AI weapons for Ukraine https://www.npr.org/2024/07/09/nx-s1-4985981/oculus-ai-weapons-ukraine-palmer-luckey
AI Education:
Free book: Understanding Deep Learning https://udlbook.github.io/udlbook/
The Illustrated Transformer — A visual and intuitive guide to understanding how transformers work in machine learning. https://jalammar.github.io/illustrated-transformer/
The Illustrated AlphaFold — “Do you want to know how AlphaFold3 works? It has one of the most intimidating transformer-based architectures, so to make it approachable, we made a visual walkthrough.” https://elanapearl.github.io/blog/2024/the-illustrated-alphafold/
Biotech:
Why haven't biologists cured cancer? It's not because they're not good enough at math. Slow feedback loops. In the medical field, accelerating progress will involve streamlining clinical trials both in terms of time and costs. https://www.writingruxandrabio.com/p/why-havent-biologists-cured-cancer
Inside the Laboratory for Extraordinary Microbes: “Significant advances in science today do not often come from solitary geniuses, as they often did before World War II and the modern era of multi-billion dollar government programs. Much more commonly, progress stems from the collective efforts of large teams with aligned missions.” https://press.asimov.com/articles/cultivarium
Gene Drives Shown to Work in Wild Plants. They Could Wipe Out Weeds. https://singularityhub.com/2024/07/08/gene-drives-shown-to-work-in-wild-plants-they-could-wipe-out-weeds/
Computer Science:
To understand what quantum computers can do — and what they can’t — avoid falling for overly simple explanations. https://www.quantamagazine.org/why-is-quantum-computing-so-hard-to-explain-20210608/
The Zombie Misconception of Theoretical Computer Science https://scottaaronson.blog/?p=8106
A Trustworthy, Free (Libre), Linux Capable, Self-Hosting 64bit RISC-V Computer https://x.com/karpathy/status/1811097021539045582 (Project page: https://www.contrib.andrew.cmu.edu/~somlo/BTCP/)
Astronomy:
Astronomers find surprising ice world in the habitable zone with JWST data https://news.umich.edu/astronomers-find-surprising-ice-world-in-the-habitable-zone-with-jwst-data/
Primon gas: a theoretical gas where there's one kind of particle for each prime number, and the energy of the prime p is log(p). The partition function of this gas is the Riemann zeta function. https://mathstodon.xyz/@johncarlosbaez/112762809456139367
Miscellaneous:
The Material So Classified We Forgot How to Make It https://youtu.be/Y6tqlf31YTc
The evidence is mounting: humans were responsible for the extinction of large mammals https://nat.au.dk/en/about-the-faculty/news/show/artikel/beviserne-hober-sig-op-mennesket-stod-bag-udryddelsen-af-store-pattedyr
“Despite its small brain size, H. naledi shared some aspects of human brain organization, suggesting that innovations in brain structure were ancestral within the genus Homo.” https://www.pnas.org/doi/10.1073/pnas.1720842115
Politics:
“Read the story of a decade-long propaganda campaign by the Forrest Gump of the internet—a Wikipedia admin who was once Yudkowsky’s strongest soldier—set against the backdrop of the collapse of the semi-unified Internet ethos of the ‘90s and ‘00s” https://www.tracingwoodgrains.com/p/reliable-sources-how-wikipedia-admin
History is written by the losers https://scholars-stage.org/history-is-written-by-the-losers/
Ukraine:
Satellite images of a Ukrainian drone hit on a Russian ammo depot https://x.com/bradyafr/status/1810286460081250538
“Just look at the amount of destroyed Russian equipment.” https://x.com/GloOouD/status/1810324134079185024
“Quite unique footage from a Russian base of the 1307th regiment after it was hit by the Ukrainian army.” https://x.com/wartranslated/status/1810221337509536141
Russian infantry vs drones https://x.com/GloOouD/status/1810266064606966065
Ukrainian drone strikes are reported on a substation in the Rostov region and an oil depot in the city of Kalach-on-Don, Volgograd region. https://x.com/NOELreports/status/1810536741519921600
“Ukrainian loitering munition strikes on Russian TOR and BUK air defence systems, as well as Msta-S self propelled howitzer. Plus other strikes on different targets.” https://x.com/bayraktar_1love/status/1811023732191711468
“Recent Ukrainian drone attack also targeted Astrakhan region of Russia, targeting objects on the territory of the 4th State Central Interspecific Test Site, also known as Kapustin Yar.” https://x.com/bayraktar_1love/status/1810770587100577847
“Russia has been able to advance unusually quickly in the Niu York-Toretsk direction, which has been a mostly static direction since 2022.” https://x.com/emilkastehelmi/status/1810038700803539292
“Ukraine’s manpower, fortifications, and ammunition situation is steadily improving. Russian forces are advancing in Donetsk, and likely to make further gains, but they have not been able to exploit the Kharkiv offensive into a major breakthrough.” https://x.com/KofmanMichael/status/1811079176822435851
Russians executed 2 Ukrainian soldiers who surrendered in Zaporizhzhia region. https://x.com/GloOouD/status/1811028587316699271
Ukraine Hospital Attack:
A detailed analysis of all the evidence relating to the munition used on the Kyiv children's hospital attack, clearly pointing to a Russian Kh-101 cruise missile https://www.bellingcat.com/news/2024/07/09/russian-missile-identified-in-kyiv-childrens-hospital-attack/
“More compellingly, the explosive payload of AIM-120 (18 kg) is insufficient compared to the hundreds of kilograms typical of Kh-101 missiles, evident in the observed blast radius and damage pattern with burn marks.” https://euromaidanpress.com/2024/07/09/russia-struck-kyiv-children-hospital-with-kh-101-missile-osint-analysis-indicates/
“Here we unambiguously show that the missile was highly likely a RU Kh-101 missile descending for attack normally.” https://x.com/Dmojavensis/status/1810611899933024579
“If you zoom in and look closely, you'll see atmospheric distortions indicating the exhaust stream coming from the missile's rear section. This is typical for cruise missiles like the Kh-101, as cruise missiles are continuously propelled until impact.” https://x.com/FRHoffmann1/status/1810686983506735555
Biden announced that the U.S., Germany, the Netherlands, Romania, and Italy will provide five additional air defense systems to Ukraine. In the coming months, the U.S. will also supply Ukraine with dozens of additional tactical air defense systems, including NASAMS, HAWK, IRIS T-SLM, IRIS T-SLS, and Gepard. https://www.whitehouse.gov/briefing-room/statements-releases/2024/07/09/joint-statement-on-strengthening-ukraines-air-defenses-by-u-s-president-joseph-r-biden-dutch-prime-minister-dick-schoof-german-chancellor-olaf-scholz-italian-prime-minister-giorgia-melon/