Skip to content
Eight Google Authors Posted the Transformer Paper on 12 June 2017, and Its Big Model Scored 28.4 BLEU in English to German After 3.5 Days on Eight GPUs
AI & Technology

Eight Google Authors Posted the Transformer Paper on 12 June 2017, and Its Big Model Scored 28.4 BLEU in English to German After 3.5 Days on Eight GPUs

Illustration by tuput

English

The 2017 paper dropped word-by-word reading and let every word in a sentence look at every other word at once. Its big model beat the best earlier English to German results by more than 2 BLEU, and Llama 3, Gemini 1.0, DeepSeek-V3 and Sarvam 105B are all described by their makers as Transformers.

· · 11 min read

On 12 June 2017 eight researchers posted a paper called “Attention Is All You Need” to the arXiv preprint server. It described the Transformer, a network “based solely on attention mechanisms, dispensing with recurrence and convolutions entirely”, in the words of its abstract. The paper’s big model scored 28.4 BLEU, a standard translation score where higher is better, on the WMT 2014 English to German test. The abstract says that beat the best earlier results, ensembles included, by over 2 BLEU, and section 6.1 says the training took 3.5 days on eight Nvidia P100 GPUs.

The authors were Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser and Illia Polosukhin, listed under Google Brain, Google Research and the University of Toronto. The paper carries a footnote saying the listing order is random and that all eight contributed equally. It was presented at the Conference on Neural Information Processing Systems, then called NIPS, in Long Beach, California, in 2017 (the proceedings page lists it in volume 30). How a large language model is trained covers what happens after the architecture is chosen. This article is about the architecture.

Why reading one word at a time was a bottleneck

Before 2017 the leading translation systems used recurrent neural networks. The paper describes how they work: a recurrent network goes through the words in order and produces a running summary at each position, computed from the previous summary and the current word. Because each step needs the one before it, the paper says, the design “precludes parallelization within training examples”. The steps for one sentence could not be run in parallel.

Google’s blog post by Jakob Uszkoreit, dated 31 August 2017, gives a concrete case. To decide whether “bank” in “I arrived at the bank after crossing the…” means a riverbank or a lender, the network needs the last word of the sentence, “river” or “road”. A recurrent network, the post says, can only settle it after reading every word between “bank” and “river” step by step, and the more steps a decision needs, the harder it is for the network to learn it. The post adds that GPUs and TPUs, the chips used for training, are good at parallel work and poor at sequential work.

Convolutional networks, the other main design, are less sequential. The paper notes that in the models called ByteNet and ConvS2S the number of steps needed to connect two distant words still grows with the distance between them, linearly for ConvS2S and logarithmically for ByteNet.

Attention existed before the Transformer

The idea of attention came first and was added to recurrent networks. On 1 September 2014 Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio submitted a paper, later an oral presentation at ICLR 2015, that took on a weakness of the standard design. The usual encoder-decoder squeezed a whole source sentence into one fixed-length vector before the decoder began translating. The authors conjectured that this was a bottleneck, and let the model “soft-search” the source sentence for the positions that matter to each word it writes. They report translation quality comparable to the best phrase-based system of the time on English to French.

A year later Minh-Thang Luong, Hieu Pham and Christopher Manning at Stanford tested two kinds of attention on English-German translation. Their abstract reports a gain of 5.0 BLEU over comparable systems without attention, and a combined model that scored 25.9 BLEU on the WMT 2015 English to German task.

The Transformer paper states how attention was usually used by then: “in all but a few cases” it sat alongside a recurrent network. What the eight authors proposed was to keep attention and remove the recurrent part. In the paper’s contribution footnote, Uszkoreit is credited with proposing the replacement of recurrent networks by self-attention.

How a word looks at the rest of the sentence

Self-attention is attention applied within one sentence. Google’s blog post explains it with the same “bank” example. To compute a better representation of “bank”, the Transformer compares that word with every other word in the sentence. Each comparison produces an attention score. The scores decide how much each other word contributes, so “river” can receive a high score. The model then takes a weighted average of all the words’ representations and passes it through a small feed-forward network to produce the new representation. Because every word can see every other word directly, the post says, the model can attend to “river” and make the decision in a single step.

The paper names the parts of the comparison. An attention function maps a query and a set of key-value pairs to an output, and the output is a weighted sum of the values. The weight on each value comes from comparing the query with that value’s key. One way to read this is that each word asks a question (the query), every word in the sentence offers a label (the key) and some content (the value), and the best-matching labels contribute the most content. The paper’s version multiplies the query by the keys, divides by the square root of the key size, and turns the results into weights that add up to one.

The division is there for a reason the authors state as a suspicion: with large keys the products grow large, which pushes the weighting function into a region where it learns very slowly. The decoder, which writes the translation one word at a time, also hides later positions from itself so that a prediction can depend only on words already known.

Google’s post gives a second example, a pair of sentences in which “it” refers to the animal in one and to the street in the other. Translating into French depends on the gender of the noun, and the post says “animal” and “street” differ in gender there. In Google’s account the Transformer translated both sentences correctly where the Google Translate model then in use did not, and a chart of one attention head showed the attention paid to each noun shifting with the context. That is Google’s own demonstration on one pair of sentences, not an independent test.

Eight heads, and a trick for word order

Averaging all the words into one blend loses detail, so the paper runs several attention operations side by side, which it calls multi-head attention. The base model uses eight heads. Each works on a 64-number slice of the 512-number representation, and the results are joined at the end, so the total cost stays close to one head working on the full width.

Table 3 of the paper tests the choice on a development set (the 2013 news test set rather than the 2014 one). A single head scored 0.9 BLEU below the best setting, and quality also fell when there were too many heads. The authors add in their discussion of self-attention that individual heads appear to learn different jobs and that many seem related to the grammar and meaning of the sentences. That is an observation from inspecting attention charts, not a proof.

A network with no recurrence has no sense of word order, since all the words are looked at together. The authors fixed this by adding position information to each word’s representation, using sine and cosine waves of different wavelengths. They also tried learning the positions instead, and the two versions gave nearly identical results: 25.7 BLEU against 25.8 for the base model on the development set. They kept the sine version because it might handle sentences longer than any seen in training.

This is one piece of the 2017 design that later models replaced. Sarvam’s March 2026 blog post lists rotary positional embeddings among its models’ features, and Meta’s Llama 3 paper says the company raised the base frequency of its rotary embeddings to 500,000 to support longer contexts.

Why it trained faster

Table 1 of the paper compares the layer types. A self-attention layer connects any two positions in a constant number of sequential operations, where a recurrent layer needs a number that grows with the sentence length. The paper’s base model trained for 100,000 steps, about 12 hours, on one machine with eight Nvidia P100 GPUs. The big model took 300,000 steps, 3.5 days. Google’s blog says training sped up by up to an order of magnitude, which is Google’s own claim from its experiments.

The model was small by later standards. Table 3 lists 65 million parameters, the adjustable numbers inside a network, for the base model and 213 million for the big one. Meta’s Llama 3 flagship, described seven years later, has 405 billion, about 1,900 times as many.

Self-attention has a cost of its own. Comparing every word with every other word means the work per layer grows with the square of the sentence length, and the paper says self-attention is cheaper than a recurrent layer only when the sentence is shorter than the representation width. The authors suggested limiting attention to a neighbourhood of nearby words for very long inputs, as work for the future. Meta’s paper gives the same reason for training on long texts late: self-attention compute grows quadratically with sequence length, so Llama 3 reached its 128,000-token context in the final stages of pretraining.

The numbers, and the one that differs

The paper’s headline results are on the newstest2014 sets from the WMT 2014 translation task. For English to German, Table 2 shows the big model at 28.4 BLEU against 26.36 for the best earlier entry, an ensemble of ConvS2S models. For English to French the abstract and Table 2 give 41.8 BLEU for a single model, while a sentence in section 6.1 of the same paper says 41.0. The table and abstract agree with each other, and the text differs, so the figure is quoted here as the abstract has it, with the discrepancy noted.

The authors also tried the model on a different task, English constituency parsing, which labels the grammatical structure of a sentence. A four-layer Transformer scored 91.3 F1 trained on the Wall Street Journal data alone and 92.7 with extra semi-supervised data. The paper says this beat all earlier reported models except one, the Recurrent Neural Network Grammar. Google’s post says the same network, with little adaptation, outperformed all but one of the earlier approaches.

The results are the authors’ own, measured against published numbers. The paper’s code was released in the tensor2tensor library, and Harvard NLP’s Annotated Transformer reimplements the model line by line as a runnable notebook.

What stayed and what changed in Llama, Gemini, DeepSeek and Sarvam

The paper’s conclusion says the authors planned to extend the Transformer beyond text, to images, audio and video. Google’s 2017 post said that work had already begun. By October 2018 Google’s BERT paper called its model a multi-layer bidirectional Transformer encoder, and described OpenAI’s GPT as a left-to-right Transformer in which each token attends only to earlier ones. The Hugging Face course groups the models that followed into GPT-like, BERT-like and sequence-to-sequence families.

The makers of today’s large models describe their own work in the same terms. Meta’s paper says Llama 3 “uses a standard, dense Transformer architecture” and that its gains come mainly from data quality, diversity and scale. Meta chose a dense model over a mixture of experts for training stability. Its small changes include grouped query attention, with 8 key-value heads, to shrink the memory the model keeps while writing. Google’s Gemini report says Gemini models “build on top of Transformer decoders” with improvements for stable training and for Google’s TPU chips, and names multi-query attention among them.

DeepSeek’s V3 report says its basic architecture remains within the Transformer framework, with multi-head latent attention, which Sarvam describes as a compressed form of attention that cuts memory use, and a mixture-of-experts feed-forward layer. The model has 671 billion parameters in total, with 37 billion used for each token. Sarvam AI describes its 30-billion and 105-billion-parameter models as a mixture-of-experts Transformer backbone with 128 experts. The 30B uses grouped query attention and the 105B uses multi-head latent attention. Sarvam says both were trained in India on compute provided under the IndiaAI mission and released under the Apache 2.0 licence, as covered in the IndiaAI Mission article and the piece on India building AI in its own languages. Translation, the task the Transformer was built for, is also the subject of the Bhashini and AI4Bharat article, which covers open translation models for the 22 scheduled Indian languages.

In every one of these, the self-attention layer from 2017 is still the core, and the changes are in how keys and values are stored, how positions are encoded and what replaces the single feed-forward layer. The 2017 big model had 6 layers and 16 heads; Meta’s paper gives Llama 3 405B 126 layers and 128 attention heads.

Share
Copied!

Sources & further reading

  1. Vaswani et al.: Attention Is All You Need, arXiv:1706.03762 (submitted 12 June 2017, abstract page)
  2. Vaswani et al.: Attention Is All You Need, full paper as presented at NIPS 2017 (PDF)
  3. NeurIPS Proceedings: Attention is All you Need (Advances in Neural Information Processing Systems 30, 2017)
  4. Google Research blog, Jakob Uszkoreit: Transformer, A Novel Neural Network Architecture for Language Understanding (31 August 2017)
  5. Bahdanau, Cho and Bengio: Neural Machine Translation by Jointly Learning to Align and Translate, arXiv:1409.0473 (submitted 1 September 2014, ICLR 2015)
  6. Bahdanau, Cho and Bengio: Neural Machine Translation by Jointly Learning to Align and Translate (full paper, PDF)
  7. Luong, Pham and Manning, Stanford: Effective Approaches to Attention-based Neural Machine Translation, arXiv:1508.04025 (2015)
  8. Devlin et al., Google AI Language: BERT, Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv:1810.04805
  9. Gemini Team, Google: Gemini, A Family of Highly Capable Multimodal Models, arXiv:2312.11805
  10. Llama Team, AI @ Meta: The Llama 3 Herd of Models, arXiv:2407.21783 (23 July 2024)
  11. DeepSeek-AI: DeepSeek-V3 Technical Report, arXiv:2412.19437
  12. Sarvam AI: Open-Sourcing Sarvam 30B and 105B (6 March 2026)
  13. Harvard NLP and collaborators: The Annotated Transformer (line-by-line implementation of the paper)
  14. Hugging Face LLM Course: How do Transformers work?

Researched and written with the help of AI tools and edited for accuracy. Provided for general information and discussion only, not professional advice. See our editorial standards and disclaimer. Spotted an error? Tell us.

#transformer#attention is all you need#self-attention#neural machine translation#large language models#google research#sarvam ai#deep learning

Enjoyed this? Get the next one.

One good read at a time, straight to your inbox. No spam, unsubscribe anytime.

More in AI & Technology
OpenAI's 2020 Scaling Laws Paper Found Model Loss Falls as a Power Law Over More Than Seven Orders of Magnitude, and DeepMind's 2022 Chinchilla Paper Showed a 70 Billion Parameter Model Could Beat a 280 Billion One on the Same Compute
A model's size, its training text and its compute bill can be traded against each other. Here is what each paper measured, the numbers each printed, one replication attempt, and what a 2026 safety report says about how far the rules reach.
Meta Trained Llama 3 405B on 15.6 Trillion Tokens and 30.84 Million GPU Hours, and OpenAI's Human Raters Preferred a 1.3 Billion Parameter Tuned Model to the 175 Billion Parameter GPT-3
Five papers and one model card lay out each stage of building a chatbot's underlying model. Here are the stages in order, with the parameters, tokens, GPU hours and test results the labs themselves published.
Sarvam's 105-Billion-Parameter Model Computes With 10.3 Billion of Them for Each Token, Using the Mixture-of-Experts Design That Runs From a 1991 Paper to Mixtral and DeepSeek
A small router sends every word to a few of 128 sub-networks, so most of the model sits idle for any one word. Here is where the design came from, how Mixtral, DeepSeek and Sarvam use it, and what it does not save.
← all articles