pip install transformers [I've removed this output cell for brevity]. If there are two sentences A and B, BERT … ! However, I would rather go with @Palak's solution below – …

The library also includes task-specific classes for token classification, question answering, next sentence prediciton, etc. However, it is also important to understand how different sentences making up a text are related as well; for this, BERT is trained on another NLP task: Next Sentence Prediction (NSP). The idea with “Next Sentence Prediction” is to detect whether two sentences are coherent when placed one after another or not. You can only mask a word and ask BERT to predict it given the rest of the sentence (both to the left and to the right of the masked word). Next Sentence Prediction (NSP): In order to train a model that understands sentence relationships, we pre-train for a next sentence prediction task.
BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling. An additional objective was to predict the next sentence. To make BERT better at handling relationships between multiple sentences, the pre-training process also included an additional task: given two sentences (A and B), is B likely to be the sentence that follows A? 10 comments Comments. Using these pre-built classes simplifies the process of modifying BERT for your purposes. Copy link Quote reply pierrerappolt commented Jan 17, 2019 • edited Hi, are the final output weights available which were used to predict next sentence during your pre-training on Wiki (the feed forward layer weights and biases on the CLS token)? Let’s look at examples of these tasks: Masked LM (MLM) The idea here is “simple”: Randomly mask out 15% of the words in the input — replacing them with a [MASK] token — run the entire sequence through the BERT attention based encoder and then predict only the masked words, based on the context provided by the other non …

BERT is pre-trained on a next sentence prediction task, so I would think the [CLS] token already encodes the sentence. Then the encoder of BERT is made to predict the masked words. For this, consecutive sentences from the training data are used as a positive example. BERT’s clever language modeling task masks 15% of words in the input and asks the model to predict the missing word. Here two sentences selected from the corpus are both tokenized, separated from one another by a special Separation token, and fed as a single intput sequence into BERT. For a negative example, some sentence is taken and a random sentence from another document is placed next to it. Next sentence prediction: Model takes 2 sentences and predict if the second sentence comes after the other. BERT is trained on a masked language modeling task and therefore you cannot "predict the next word".