Links for 2024-01-02
AI:
Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws https://arxiv.org/abs/2401.00448
Agent Attention: On the Integration of Softmax and Linear Attention — “…when applied to Stable Diffusion, our agent attention accelerates generation and substantially enhances image generation quality without any additional training.” https://arxiv.org/abs/2312.08874
“Experimentally, our method can prune rows and columns from a range of OPT models and Llamav2-7B by 20%-30%, with a negligible loss in performance, and achieve state-of-the-art results in unstructured and semi-structured pruning of large language models.” https://arxiv.org/abs/2312.17244
City-on-Web: Real-time Neural Rendering of Large-scale Scenes on the Web https://ustc3dv.github.io/City-on-Web/
“LLMs are good at prediction, but that means they're also good at identifying unpredictable and out-of-distribution things. I expect many people are going to do something like inverse-prediction on lots of data types to identify odd things.” https://twitter.com/jackclarkSF/status/1740787848712958298
This reminds me of Elon Musk's 2014 prediction that something seriously dangerous would happen within 10 years. It was wrong, but it pointed in the right direction. The insider information he gained from exposure to groups like DeepMind allowed him to sense that AI progress would accelerate dramatically. Similarly, as Baidu's former chief scientist, Andrew Ng knew that AI progress would continue to accelerate. 5 years later, it has yet to drive massive global GDP growth, but most people could have learned something from his prediction, namely that the importance of AI will massively increase and massively drive the stocks of companies like Nvidia.
Similarly, a bunch of roboticists are now predicting that robots are coming. Maybe it won't happen as fast as they predict (~3 years), but we'll probably see a lot of progress.
Science and Technology:
2023: A Review of the Year in Neuroscience https://medium.com/the-spike/2023-a-review-of-the-year-in-neuroscience-ece56b1dcc89 [https://archive.is/wip/rbGEq]
A new RNA targeting CRISPR enzyme paired with a convolutional neural network (CNN) model predicting highly efficient guide RNA sequences for transcriptome engineering https://arcinstitute.org/news/blog/arc-tool-portal
Will 2024 be very hot? Should we be worried? https://www.lesswrong.com/posts/kiACM88mT3tMx5TGx/will-2024-be-very-hot-should-we-be-worried
"It's not just time zones and leap seconds. SI seconds on Earth are slower because of relativity, so there are time standards for space stuff (TCB, TGC) that use faster SI seconds than UTC/Unix time. T2 - T1 = [God doesn't know and the Devil isn't telling.]" https://xkcd.com/2867/
Ukraine:
Russia is still suffering huge losses every day: The 2nd mechanized battalion of the 30th mechanized brigade shows the scale of destruction of both Russian heavy equipment and infantry personnel. https://twitter.com/GloOouD/status/1741927579123101795
The 1st Tank Brigade "Severia" of ZSU destroying 4 russian APC/IFV (last one was BTR) in Avdiivka direction. https://twitter.com/TheDeadDistrict/status/1741762440948257231
Drone operators of the Ukrainian 14th Regiment of UAC supported the destruction of a Russian Buk-M1-2 air defense system. https://twitter.com/Tendar/status/1742081878901891250
“The Ukrainian SBU released a video of one of their USVs. It shows a spectacular new development. Apparently, Ukrainian USVs are now capable to fire missiles. The video shows an attack against Russian Navy vessels at the port of Russian-occupied Sevastopol.” http://www.hisutton.com/Ukraine-Sea-Baby-Rockets.html
"We were 70 kilometres away from the Admiral Makarov frigate. Everyone was on edge, as we were going to attack it. And then, our communication was cut off. Elon Musk switched off Starlink, which we used to control the vessels." https://www.pravda.com.ua/eng/news/2024/01/1/7435389/
The Government of Norway has decided to allow direct sales of weapons and defense products from the Norwegian defense industry to the defense authorities of Ukraine. https://www.regjeringen.no/en/aktuelt/direct-sales-of-weapons-and-defense-related-products-to-ukraine-permitted-from-1-january-2024/id3020487/
"Finland and its people continue to strongly support Ukraine. Our decisions at the national level, such as increasing production capacity for heavy ammunition, allow us to support Ukraine in the long term," Prime Minister of Finland Petteri Orpo said. https://www.eurointegration.com.ua/rus/news/2024/01/1/7176540/