Links for 2023-02-27
“...we show how trained Transformers implement gradient descent in their forward pass...we identify how Transformers surpass plain gradient descent by an iterative curvature correction and learn linear models on deep data representations to solve non-linear regression tasks...” https://threadreaderapp.com/thread/1603659524006625284.html
“I used Langchain to create an app that summarizes papers in ~100 lines of code, which shows that it’s pretty easy to use Langchain as a general LLM programming framework.” https://lancemartin.notion.site/lancemartin/Langchain-for-paper-summarization-d4ad122ea9a64c0eb1f981e743d6c419
“There are many things to do with an always-on 1-milliwatt machine-learning chip, but few spark the imagination quite like watching it play Doom.” https://spectrum.ieee.org/syntiant-chip-plays-doom
“In ‘Unnatural Instructions’, Meta AI researchers propose a new method to generate natural language instructions, allowing us to scale up to 240K diverse instructions & train models that rival performance of contemporary instruction-tuned models.” https://arxiv.org/abs/2212.09689
BioGPT: generative pre-trained transformer for biomedical text generation and mining https://academic.oup.com/bib/article-abstract/23/6/bbac409/6713511
Due to recent EU regulations, non-invasive brain stimulation is now in the same regulatory class as invasive brain stimulation despite their vastly different safety profiles. https://twitter.com/Jake_Toth_/status/1613473374595858433
“In 2020 alone, researchers spent 15,000 YEARS worth of time reviewing articles. In the US, the time cost for reviews is equal to $1.5B, in China it is $600M, in the UK is $400M.” https://researchintegrityjournal.biomedcentral.com/articles/10.1186/s41073-021-00118-2
Atomic clocks use the hyperfine transitions between energy states of electrons, with 10^-18 accuracy (1 second per 30 billion years). A 'nuclear clock' that uses transitions in atomic nuclei (like in isomer Th-229m) would be 10x more accurate. https://iopscience.iop.org/article/10.1088/1361-6455/ab29b8
NIH software assembles complete genome sequences on-demand https://www.genome.gov/news/news-release/nih-software-assembles-complete-genome-sequences-on-demand
“Contradictions within economic theory. All well known but still important and, I think, not taken as seriously as they should be.” https://statmodeling.stat.columbia.edu/2023/02/18/contradictions-within-economic-theory-all-well-known-but-still-important-and-i-think-not-taken-as-seriously-as-they-should-be/
Eurasia on the Eve of the Arab Conquests https://michaelbonner.substack.com/p/eurasia-on-the-eve-of-the-arab-conquests
“Female researchers in math, psychology and economics are 3-15 times more likely to be elected as members of the US National Academy of Sciences or American Academy of Arts and Sciences than male counterparts with similar publication and citation records.” https://www.nature.com/articles/d41586-023-00501-7
Even without hardware advances, this would not be an implausible prediction because of algorithmic advances:
1. With today's algorithms, computers would have beat the world chess champion already in 1994 on a contemporary desk computer https://www.lesswrong.com/posts/75dnjiD8kv2khe9eQ/measuring-hardware-overhang
2. Algorithmic progress has yielded more gains than classical hardware efficiency: https://openai.com/blog/ai-and-efficiency/
3. While improvements in hardware accounted for an approximate 1,000 fold increase in calculation speed over a 15-year time-span, improvements in algorithms accounted for an over 43,000 fold increase: https://www.johndcook.com/blog/2015/12/08/algorithms-vs-moores-law/
The Singularity Is Near 🚀
Mathematics is not a deductive science – that’s a cliché. When you try to prove a theorem, you don’t just list the hypotheses, and then start to reason. What you do is trial-and-error, experimentation, and guess work.
— Paul Halmos
Definitely one of the most baffling poll results I have ever seen:
Given that another U.S. department has concluded that the Covid pandemic most likely arose from a laboratory leak, it’s time to repost a quick reminder of how anyone could have ever taken the COVID-19 lab leak theory seriously:
1. Of all places, the outbreak happened in the vicinity of a BSL-4 laboratory that has among the world's largest collections of this type of virus and did the most research about them.
2. Laboratory biosecurity incidents are common, and there are confirmed historic examples of actual lab leaks causing infectious disease outbreaks.
3. Scientists have been and still are working to make various viruses more lethal and transmissible.
4. The domain experts debunking the COVID-19 lab leak theory were directly connected to the Wuhan Institute of Virology.
5. Official organizations repeatedly and extensively damaged their reputation as reliable sources. For example, it took the WHO two years to even acknowledge that COVID is airborne.
6. The more costly it is to dispute X the less confident we can rationally be about X. And it is very costly for domain experts to take the lab leak theory seriously. This means that the majority of people who take it seriously will be nonconformists who likely hold other beliefs outside the Overton window. Trying to inform yourself about the possibility of an actual conspiracy makes it therefore necessary to accept some noise and engage with actual conspiracy theorists.