BLOOM is here: here’s what makes it different from GPT-3

Training is over. The largest open source language model to date is here. The Big Science Language Open-science Open-access Multilingual, better known as BLOOM, after several months of work (in which our NLP engineer Manuel Romero has participated a little bit) it is available to everyone who wants to try it. You’ll find all the key components, in this model, that many believe represents a drastic change in the world of artificial intelligence. BLOOM is here: here’s what makes it different from GPT-3.

What is BLOOM?

Although we would like it to, this model has nothing to do with Orlando Bloom. BLOOM is a Big Science project born with the aim of creating the most powerful language model in the world. No, ‘one of the most important’. It is THE model.

Some describe it as the most important model of the last decade, as a turning point in the world of artificial intelligence. GPT-3 is the most powerful, but this one has a big difference: BLOOM is accessible to everyone.

Large language models are hard to come by because not all organizations have the ability to train such a model. We’ll give you some data and you will be amazed.

How is it different to GPT-3?

We have already talked about GPT-3 on occasion. But as a quick reminder, here is the definition:

GPT-3 is a language model developed by OpenAI (an initiative promoted by Elon Musk) that learns from existing text and can provide different ways of finishing a sentence (similar to predictive text). Once trained, it can save you a lot of time, providing linguistic richness and variability along with perfect grammar. In addition, it is capable of answering a wide variety of questions.

GPT-3 is a language model developed by OpenAI

So, how does it do it?

• It uses 175 billion parameters.
• It has been trained with 500 billion words.
• Its reading comprehension is superior to that of the average human.

Impressive figures. But very few have access to it. In this BLOOM changes everything. In addition, at the architectural level it is totally different. We tell you why it is different from the OpenAI model.

BLOOM:  training that lead around the world

The training started on March 11, 2022. But in fact, the preparations of the corpus and the datasets started much earlier. A model with these characteristics is not achieved overnight. 4 months later, here we have it. And it hasn’t been easy:

• 384 graphic cards of 80 gigabytes each on the Jean Zay supercomputer in France.
• BLOOM has 176 billion parameters, one billion more than GPT-3.
• 70 layers – 112 attention heads per layers – hidden dimensionality of 14336 – 2048 tokens sequence length.
• ALiBi positional embeddings – GeLU activation function.

The training has been open to everyone and we have been able to follow it. BLOOM has been trained in various languages ​​(English, Spanish, Italian…) and even programming codes. Every resource is available and documented.

We have tried BLOOM and these are our thoughts

At Narrativa we have tried BLOOM and we have been impressed. Its abilities are extraordinary. In fact, without giving the model many examples (what is known as few shots), it is able to answer questions and translate without problems. Even without giving it any examples (zero shots) it is able to grammatically correct a text.

Our NLP engineer Manuel Romero has done several tests and you can see the results in this Twitter thread. If you want to try it yourself, here you have the link.

Big Science

As they explain on their blog, Big Science is an open collaboration promoted by HuggingFace, GENCI and IDRIS. This research workshop brings together academic, industry, and independent researchers from many affiliations and whose research interests span many research fields in AI, NLP, social science, legal, ethics, and public policy.

Thanks to initiatives like this, artificial intelligence is accessible to everyone, not just to few. The goal of democratizing AI is something that we share in Narrativa, and that is why you can enjoy the models that we have open in our Hugging Face profile.

About Narrativa

Narrativa is an internationally recognized content services company that uses its proprietary artificial intelligence and machine learning platforms to build and deploy digital content solutions for enterprises. Its technology suite, consisting of data extraction, data analysis, natural language processing (NLP) and natural language generation (NLG) tools, all seamlessly work together to power a lineup of smart content creation, automated business intelligence reporting and process optimization products for a variety of industries.

Contact us to learn more about our solutions!