\nWhen talking about textual data, a language model is a mathematical model that assigns a probability to a sequence of words. The most common type of language model is the Bag of Words model (BoW). Bag of words is a representation of text where each word is represented by a number. This can be done by assigning a unique number to each word, or by using a technique called \u201chashing.\u201d. For instance, the number of times a certain word appears in a document is called term frequency, which is part of the BoW language model [1].<\/p>\n
Types of language models<\/strong> What is the purpose of language models?<\/strong> To illustrate a real-world example of using language models, let\u2019s look at Gmail\u2019s spam filtering text classification problem. Machine learning researchers are constantly looking for ways to improve the performance of language models. One area of research is to develop new algorithms that can better learn from data, as we explained in the spam filtering problem. Another area of research is to develop new ways to represent data that can be more easily learned by language models.<\/p>\n BERT as a neural network language model<\/strong> Now that we briefly described BERT, we will illustrate its usage on the problem of detecting toxicity from comments on Reddit. The dataset that we will use was part of a study that investigates the toxic behavior of users on Reddit [5]. The dataset (can be found here<\/a>) consists of comments from the subreddit r\/AskReddit, where each comment can be either non-toxic, slightly-toxic, or highly-toxic. So, the dataset is a great candidate for testing our understanding of language models by using and fine-tuning a BERT neural network language model.<\/p>\n
\nLanguage models can be either rule-based, statistical, or based on neural networks:<\/p>\n\n
\nLanguage models are used in NLP and ML tasks such as speech recognition, question answering, document or text classification, and machine translation.<\/p>\n
\nThe spam filtering problem is the problem of designing a system that can automatically detect spam emails and flag them for the user, as in the figure below. This is a difficult problem because spam emails are often very similar to regular emails, and it can be hard to design a system that can accurately distinguish between them. There are many different approaches to this problem, but one common approach is to use a supervised machine learning algorithm with a language model to learn from a dataset of labeled emails (i.e., emails that have been manually labeled as spam or not spam). The algorithm can then be used to classify new emails as spam or not spam [3].<\/p>\n![\"\"](\"https:\/\/acua.qcri.org\/blog\/wp-content\/uploads\/2022\/11\/document_classification.webp\")
\nA popular type of neural network-based language models is BERT, which was
\ncreated in 2017 by Google and fine-tuned on the Wikipedia corpus that contains 2.5 billion words. BERT is designed to help computers understand the meaning of ambiguous language in text by using surrounding text to establish context, just like how humans try to use surrounding terms in sentences to identify the meaning of unknown words.
\nAs a language model, BERT was pre-trained using text from Wikipedia and can be fine-tuned with datasets to solve any type of classification problems like question answering.
\nHowever, as a neural network, Bidirectional Encoder Representations from Transformers (BERT) is a machine learning framework for NLP that uses deep learning algorithms to better understand the meaning of ambiguous language [4]. BERT is a type of transformer neural network that is pre-trained on text from Wikipedia and can be fine-tuned with any type of dataset to suit user\u2019s needs, such as performing sentiment analysis, or question answering, as seen in the figure below.<\/p>\n![\"\"](\"https:\/\/acua.qcri.org\/blog\/wp-content\/uploads\/2022\/11\/BERT-model-example.png\")