Workflow from Scientific Research

The model is a BERT architecture with 12 transformer blocks (in purple), which use multihead attention and a feed forward layer with normalisation steps (LayerNorm) in between. The model is embedding the tokens and is trained with cross-entropy loss to predict the masked token and updates the embedding while training. The output is probabilities of the masked token identity.

DOI