Haikal Ryclova-Pribadi

We've shared before that pseudocode is now an important part of VCE Maths Methods. And (spoiler alert!), we've shared the answer in a post below. In this video, Toan Huynh shares this year's question and a step-by-step solution. Enjoy! https://lnkd.in/g3TN_hF2

10

https://lnkd.in/epNxY4fb I've been thinking about the way that large language models seem to be able to generalise to unseen problems lately and I've been thinking that the spectral bias of neural networks is probably a key component of that. Between this and the double descent problem, it's becoming increasingly clear that massively overparameterized neural networks are biased towards simple and smooth solutions to a problem. Often, the simplest representation of a domain is also the 'correct' one. Consider a dataset that consists exclusively of uniformly sized black dots against a white background. The most compact encoding that contains all the information for this dataset is just the x and y co-ordinates of the dot. These values also happen to be the real generator of the dataset. It’d be reasonable to expect a beta VAE to converge on this representation based on spectral bias and it would be the most generic encoding. Is a similar thing happening in large language models? When presented with conversations and papers about scientific matters where the next word depends on what is true, perhaps the simplest policy is to understand the topic and to model reality first - language second.

3

Seems like a fascinating book. The review itself is written very succinctly. Learnt that laws of conservation ( momentum, energy, ) in physics is mathematically a result of symmetry. https://lnkd.in/gKTH3-2p

1

Proprietary datasets and the combination of AI and experimental data are the key to build successful companies.

1

Here's a good read - an article I recently wrote about epochs and how to tune them: Tuning Epochs: Because Your Model's Been Lazy Long Enough #machinelearning #epoch

2

Exciting discovery in Physics and Data Sciences! 🚀 I came across a fascinating paper discussing the integration of Zipf’s law into algorithmic predictions. This law, dating back to linguist George Kingsley Zipf in 1935, highlights word frequency following an inverse exponential distribution with a long tail. The paper takes an innovative approach by suggesting that embracing Zipf’s law could enhance symbolic regression algorithms. By filtering out complex and unphysical expressions, researchers believe this method could potentially unveil new laws of physics. This probabilistic approach aims to streamline the identification of novel physics laws amidst the vast possibilities produced by AI technologies like large language models and machine learning. Check out the full paper here: [Link to the paper](https://lnkd.in/extbfwTa) #Physics #DataScience #Innovation #AI #AlgorithmicPredictions #ailearn #learnitall #msadvocate

3

An increase of 1,600 contributors to open source projects in the UK, in the last 6 months. This figure reflects those whose code contributions have been accepted (commits). This increase of 1600 in 6 months reflects against the growth of 1,700 in the previous 12 months. The data indicates a strong increase across all levels of commits to open source projects from the UK, although, at the top level of a committer, who has had 1,000 plus commits accepted, the growth rate is unsurprisingly significantly smaller, but still with relatively good growth. Read the recent report to understand the OpenUK ask on Skills, to build more contributors to open source projects from the UK through OpenUK's Community Inclusion Program. https://lnkd.in/eUvbNwyk Thanks to GitHub and Runa Capital for their support on this data. To get involved mail us at mailto: admin@openuk.uk #theopenmanifesto #openuk #opensourcesoftware #opensource

2

"...embeddings enable machine learning models to find similar objects." Read more of Gregory Hodum's article on embeddings https://lnkd.in/eFs6hhAP #ai #softwaredevelopment #embeddings #machinelearning

9

Excited to share our latest work, where we demonstrate the first application of SAEs in a real-world use case that outperforms traditional methods!

10

2 Comments

Ex-OpenAI Superalignment employee Leopold Aschenbrenner published a lengthy paper this week about how he thinks we could get to AGI (Artificial General Intelligence) by 2027. THAT IS SOON. It’s a long and complicated document but basically posits that between increases in compute, algorithmic improvements and a few ‘mysterious’ other things, we could actually speed this up. The implications are pretty significant overall and it’s worth diving into what he has to say even if you don’t agree with it. #openAI #technology #chatGPT #research #machinelearning #ai

2

A great read on the performance and financials of LLM fine-tuning. Really hoping the outcome is a tool that helps overcome the biggest elephant in the room, Mr. Cost! #llm #finetuning #finance #computaioncosts

2

1 Comment

This article by Pere Martra explores a structured pruning technique for state-of-the-art models, that uses a GLU architecture, enabling the creation of smaller and more efficient large language models. #Llama #LLM

5

I am pleased to share that our paper on Jaynes machine is now available online via Open Access: https://lnkd.in/e64_Jgja John Hopfield and Geoffrey Hinton, who are receiving the Physics Nobel next month, looked to physics for inspiration to model neural networks. They developed the Hopfield network and the Boltzmann machine by using the Ising model and energy minimization as their guiding principles. I went in a completely different direction, looked to game theory, and identified potential games as the starting point. In our framework, the neurons and the connections compete in an error-minimization game to increase their effective utilities during training and reach an arbitrage equilibrium. The outcome is a lognormal distribution of weights and neuronal iota. Our paper shows that this is empirically observed in real-life deep neural nets such as the LLMs. The Hopfield model and the Boltzmann machine cannot predict the distribution of connection weights as it is beyond their capabilities. We show that the Hopfield network and the Boltzmann machine are special cases of the Jaynes machine. One can think of the former two as the "ideal gas law" version of neural networks, whereas ours is the "van der Waals equation". Just like the real-world gases are not ideal and behave more like van der Waals gases, the real-world deep neural networks such as VGGnet, BERT, and LLAMA are not Boltzmann machines, but are more like Jaynes machines.

132

14 Comments

Finally - achieving and surpassing the benchmarks that have long been held to mark a tipping point in quantum computing. Today's announcement, taken together with other disclosure in the past few weeks, extends our leadership in the quantum computing sector and brings reliable quantum computing out from the lab into the real world. This achievement should be one that all of us in the sector can enjoy and appreciate. I look forward to building on these developments as we get to the point where it becomes impossible for classical computers to simulate quantum processors. A few weeks ago we demonstrated our ability to scale our quantum computers. Then we disclosed stunning results with Microsoft on logical qubit development. And today we provide details of our world leading quantum volume numbers (exponentially better than literally anyone else) and for the first time, 2 qubit gate fidelities above '3 nines' across all our zones in our fully connected quantum processor. I could not be prouder of my colleagues. There is so much more to come as we continue to set the high bar in the industry and lead by example in exemplifying excellence. https://lnkd.in/esMQfmNt

227

17 Comments

pleias releases world’s first LLMs trained on exclusively open data Training large language models required copyrighted data until it did not. Today we release Pleias 1.0 models, a family of fully open small language models and their RAG-tuned counterparts. Pleias 1.0 models come in three sizes: 350m, 1.2b and 3b parameters. Our models are the first to be fully EU AI Act compliant, thus establishing a clear precedent that truly ethical and socially acceptable AI is possible. What Makes pleias 1.0 Special Our models are: *  trained exclusively on open data, meaning data that are either non-copyrighted or are published under a permissible license * multilingual, offering strong support for multiple European languages *  safe, showing the lowest results on the toxicity benchmark *  frugal (<3b) yet performant for key tasks, such as knowledge retrieval *  able to run efficiently on consumer-grade hardware locally Specialised pretraining This release is not just about open and ethical data. We’re publishing two specialized models for knowledge retrieval with strong performance: Pleias-Nano (based on Pleias-1b) and Pleias-Pico (based on Pleias-350m) are competitive with the best existing models (Llama, Qwen) ten times their size. They're like the espresso shots of the model world: small, strong, and they get the job done without making a fuss. The small size of our Pico and Nano models makes it possible to create fully local RAG applications running on CPU. The models are an integral part of our scientific assistant scholasticAI we are demoing today for Mozilla Builders. Not only open source but open science models The Pleias 1.0 are not just open weights but open science models. The models were trained on Nanotron, the open pretraining library from Hugging Face and we’re releasing our code variant and configurations. Along with Common Corpus we are going to release our training data recipes. The 3B model was trained with support from Etalab - Direction interministérielle du numérique (DINUM) on the GENCI Jean Zay H100 NVIDIA Eviden partition, as part of the Grand Challenge aligned with the European strategy for establishing AI factories through the EuroHPC Joint Undertaking (EuroHPC JU) aimed at supporting startups and providing open source models to the community. We developed our Nano (1.2B) model with the support and expertise from TractoAI, a serverless AI platform for running data and compute-intensive workloads at scale. Use our models The whole family of models is available for use through HuggingFace. We release the Pleias 1.0 models under a permissive Apache 2.0 license, meaning that models are available for use, distribution, and modification for any purpose. In the end, this is about more than just models – it's about showing that the "impossible" might just be a matter of perspective. And maybe, just maybe, the ethical path isn't as boring as everyone thought. https://lnkd.in/eZieYVpQ

266

50 Comments