![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:2000\/1*wWg2hy2Tum8rHuy-Dmfl_g.jpeg\")
<\/figure>In part 1<\/a>, I introduced the field of Natural Language Processing (NLP) and the deep learning movement that\u2019s powered it. I also walked you through 3 critical concepts in NLP: text embeddings<\/strong> (vector representations of strings), machine translation<\/strong> (using neural networks to translate languages), and dialogue & conversation<\/strong> (tech that can hold conversations with humans in real time). In part 2, I\u2019ll cover 4 other important NLP techniques that you should pay attention to in order to keep up with the fast growing pace of this research field.<\/p>\n Human communication isn\u2019t just words and their explicit meanings. Instead, it\u2019s nuanced and complex. You can tell based on the way a friend asks you a question whether they\u2019re bored, angry, or curious. You can tell based on word choice and punctuation whether a customer is getting furious, even in a completely text-based conversation.<\/p>\n You can read an Amazon review for a product and understand whether the reviewer liked or disliked it even if they never directly said so. For computers to truly understand the way humans communicate every day, they need to understand more than the objective definitions of words; they need to understand our sentiments, what we really mean. Sentiment analysis<\/a> is this process of interpreting the meaning of larger text units (entities, descriptive terms, facts, arguments, stories) by the semantic composition of smaller elements.<\/p>\n The traditional approach to sentiment analysis is to treat a sentence as a bag-of-words and to consult a curated list of \u201cpositive\u201d and \u201cnegative\u201d words to determine the sentiment of that particular sentence. This would require hand-designed features to capture the sentiment, which is extremely time-consuming and unscalable.<\/p>\n The modern deep learning approach for sentiment analysis can be used for morphology, syntax, and logical semantics, of which the most effective one is Recursive Neural Networks<\/a>. As the name implies, the main assumption for Recursive Neural Net development is such that recursion is a natural way for describing language. Recursion is useful in disambiguation, helpful for some tasks to refer to specific phrases, and works extremely well for tasks that use a grammatical tree structure.<\/p>\n Recursive Neural Networks are perfect for settings that have a nested hierarchy and an intrinsic recursive structure. If we think about a sentence, doesn\u2019t this have such a structure? Take the sentence \u201cA big crowd violently attacks the unarmed police.\u201d First, we break apart the sentence into its respective Noun Phrase and Verb Phrase \u2014 \u201cA big crowd\u201d and \u201cviolently attacks the unarmed police.\u201d But there\u2019s a noun phrase within that verb phrase, right? \u201cviolently attacks\u201d and \u201cunarmed police.\u201d Seems pretty recursive to me.<\/p>\n The syntactic rules of language are highly recursive. So we take advantage of that recursive structure with a model that respects it! Another added benefit of modeling sentences with RNN\u2019s is that we can now input sentences of arbitrary length, which was a huge head scratcher for using Neural Nets in NLP, with very clever tricks to make the sentence\u2019s input vector to be of equal size, despite the length of the sentences not being equal.<\/p>\n The Standard RNN<\/a> is the most basic version of a Recursive Neural Network. It has a max-margin structure prediction architecture that can successfully recover such structure both in complex scene images as well as sentences. It\u2019s used to provide a competitive syntactic parser for natural language sentences from the Penn Treebank<\/a>. For your reference, the Penn Treebank is the 1st large-scale treebank dataset composed of 2,499 stories from a three year Wall Street Journal (WSJ) collection of 98,732 stories for syntactic annotation. Additionally, it outperforms alternative approaches for semantic scene segmentation, annotation, and classification.<\/p>\n However, the standard RNN captures neither the full syntactic nor semantic richness of linguistic phrases. The Syntactically Untied RNN<\/a>, otherwise known as Compositional Vector Grammar (CVG), is a major upgrade that addresses this issue. It uses a syntactically untied recursive neural network that learns syntactic-semantic and compositional vector representations. The model is fast to train and implemented as efficiently as the standard RNN. It learns a soft notion of head words and improves performance on the types of ambiguities that require semantic information.<\/p>\n Another evolution is the Matrix-Vector RNN<\/a>, which is capable of capturing the compositional meaning of even much longer phrases. The model assigns a vector and a matrix to every node in a parse tree: the vector captures the inherent meaning of the constituent, while the matrix captures how it changes the meaning of neighboring words or phrases. This matrix-vector RNN can learn the meaning of operators in propositional logic and natural language.<\/p>\n As a result, the model obtains state of the art performance on three different experiments:<\/p>\n The most powerful RNN model for sentiment analysis developed thus far is Recursive Neural Tensor Network<\/a>, which has a tree structure with a neural net at each node. This model can be used for boundary segmentation to determine which word groups are positive and which are negative. The same applies to sentences as a whole. When trained on the Sentiment Treebank<\/a>, this model outperformed all previous methods on several metrics by more than 5%. Currently, it\u2019s the only model that can accurately capture the effects of negation and its scope at various tree levels for both positive and negative phrases.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n The latest in deep learning \u2014 from a source you can trust. Sign up for a weekly dive into all things deep learning<\/a>, curated by experts working in the field.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n The idea of a Question Answering (QA) system<\/strong> is to extract information, directly from documents, conversations, online searches, and elsewhere, that will meet a user\u2019s information needs. Rather than make the user read through an entire document, a QA system prefers to give a short and concise answer. Nowadays, a QA system can combine very easily with other NLP systems like chatbots, and some QA systems even go beyond the search of text documents and can extract information from a collection of pictures.<\/p>\n In fact, most of the NLP problems can be considered as a question answering problem. The paradigm is simple: we issue a query, and the machine provides a response. By reading through a document, or a set of instructions, an intelligent system should be able to answer a wide variety of questions. So naturally, we\u2019d like to design a model that can be used for general QA.<\/p>\n A powerful deep learning architecture, known as dynamic memory network<\/a>(DMN), has been developed and optimized specifically for QA problems. Given a training set of input sequences (knowledge) and questions, it can form episodic memories, and use them to generate relevant answers. The architecture has the following components:<\/p>\n DMN not only did extremely well for QA tasks, but also outperformed other architectures for sentiment analysis and part-of-speech tagging. Since its inception, there have been major improvements to Dynamic Memory Networks to further improve their accuracy on question answering tasks, including:<\/p>\n It\u2019s very difficult for human beings to manually summarize large documents of text. Text summarization is the problem in NLP of creating short, accurate, and fluent summaries for source documents. It\u2019s become an important and timely tool for assisting and interpreting text information in today\u2019s fast-growing information age. With push notifications and article digests gaining more and more traction, the task of generating intelligent and accurate summaries for long pieces of text has been growing every day.<\/p>\n Automatic summarization of text works by first calculating the word frequencies for the entire text document. Then, the 100 most common words are stored and sorted. Each sentence is then scored based on how many high frequency words it contains, with higher frequency words being worth more. Finally, the top X sentences are taken and sorted based on their position in the original text.<\/p>\n By keeping things simple and for a general purpose, the automatic text summarization algorithm is able to function in a variety of situations that other implementations might struggle with, such as documents containing foreign languages or unique word associations that aren\u2019t found in standard english language corpuses.<\/p>\n There are two fundamental approaches to text summarization: extractive <\/strong>and abstractive<\/strong>. The former extracts words and word phrases from the original text to create a summary. The latter learns an internal language representation to generate more human-like summaries, paraphrasing the intent of the original text.<\/p>\n The methods in extractive summarization<\/strong><\/a> work by selecting a subset. This is done by extracting the phrases or sentences from the actual article to form a summary. LexRank<\/strong><\/mark> and <\/mark>TextRank<\/strong><\/mark> are well known extractive summarizations.<\/mark>Both of them use a variation of the Google PageRank algorithm.<\/p>\n Models for abstractive summarization<\/strong> fall under the larger umbrella of deep learning. There have been certain breakthroughs in text summarization using deep learning. Below are some of the most notable published results by some of the biggest companies in the field of NLP:<\/p>\n Attention Mechanisms in Neural Networks are loosely based on the visual attention mechanism found in humans. Human visual attention is well-studied and while there exist different models, all of them essentially come down to being able to focus on a certain region of an image with \u201chigh resolution\u201d while perceiving the surrounding image in \u201clow resolution,\u201d and then adjusting the focal point over time.<\/p>\n Imagine you\u2019re reading a whole essay: instead of going through each word or character sequentially, you subconsciously focus on a few sentences of highest information density and filter out the rest. Your attention effectively captures contextual information in a hierarchical manner, such that it\u2019s sufficient for decision making while reducing overheads. Attention Mechanisms in Neural Networks are loosely based on the visual attention mechanism found in humans.<\/p>\n So why is this important? Models such as LSTM<\/a> and GRU rely on reading a complete sentence and compressing all the information into a fixed-length vector. This requires sophisticated feature engineering<\/a> based on the statistical properties of text. A sentence with hundreds of words represented by several words will surely lead to information loss, inadequate translation, etc.<\/p>\n With an attention mechanism, we no longer try encode the full-surge sentence into a fixed-length vector. Rather, we allow the decoder to attend to different parts of the source sentence at each step of the output generation. We let the model learn what to attend to based on the input sentence and what it has produced so far.<\/p>\n According to the image above from Effective Approaches to Attention-Based Neural Machine Translation<\/a>, blue represents encoder and red represents decoder, so we can see that the context vector takes all cells\u2019 outputs as input to compute the probability distribution of source language words for each single word the decoder wants to generate. By utilizing this mechanism, it\u2019s possible for the decoder to capture global information rather than solely to infer based on one hidden state.<\/p>\n Besides Machine Translation, the attention model works on a variety of other NLP tasks. In Show, Attend and Tell: Neural Image Caption Generation with Visual Attention<\/a>, the authors apply attention mechanisms to the problem of generating image descriptions. They use a Convolutional Neural Network<\/a> to encode the image, and a Recurrent Neural Network with attention mechanisms to generate a description. By visualizing the attention weights, they interpret what the model is looking at while generating a word:<\/p>\n In Grammar as a Foreign Language<\/a>, the authors use a Recurrent Neural Network with attention mechanism to generate sentence-parsed trees. The visualized attention matrix gives insight into how the network generates those trees:<\/p>\n In Teaching Machines to Read and Comprehend<\/a>, the authors use a Recurrent Neural Network to read a text, read a question, and then produce an answer. By visualizing the attention matrix, they show where the network looks while trying to find the answer to the question:<\/p>\n Attention does come at a cost, however. We need to calculate an attention value for each combination of input and output word. If you have a 100-word input sequence and generate a 100-word output sequence, that would be 10,000 attention values. If you do character-level computations and deal with sequences consisting of hundreds of tokens, the above mechanisms can become prohibitively expensive.<\/p>\n It should be noted that in each of the 7 NLP techniques I have discussed over these 2 posts, researchers have had to deal with a variety of obstacles: limits of the algorithms, scalability of the models, vague understanding of the human language. . .The good news is that the development of this field seems like a giant open-source project: researchers keep building better models to solve the existing problems and sharing their results with the community. Here are the major obstacles in NLP that have been resolved thanks to recent academic research progress:<\/p>\n So there you go! I showed you a basic rundown of the major natural language processing techniques that can help a computer extract, analyze, and understand useful information from a single text or sequence of texts.<\/p>\n From machine translation that connects humans across cultures, to conversational chatbots that help with customer service; from sentiment analysis that deeply understands a human\u2019s mood, to attention mechanisms that can mimic our visual attention, the field of NLP is too expansive to cover completely, so I\u2019d encourage you to explore it further, whether through online courses, blog tutorials, or research papers.<\/p>\n I\u2019d highly recommend Stanford\u2019s CS 224<\/a> for starters, as you\u2019ll learn to implement, train, debug, visualize, and invent your own neural network models for NLP tasks. As a bonus, you can get all the lecture slides, assignment guidelines, and source code from my GitHub repo<\/strong><\/a>. I hope it\u2019ll guide you in the quest of changing how we\u2019ll communicate in the future!<\/p>\n If you enjoyed this piece, I\u2019d love it if you hit the clap button<\/em> \ud83d\udc4f so others might stumble upon it. You can find my own code on<\/em> GitHub<\/em><\/a>, and more of my writing and projects at<\/em> https:\/\/jameskle.com\/<\/em><\/a>. You can also follow me on <\/em>Twitter<\/em><\/a>, email me directly or <\/em>find me on LinkedIn<\/em><\/a>. <\/em>Sign up for my newsletter<\/em><\/a> to receive my latest thoughts on data science, machine learning, and artificial intelligence right at your inbox!<\/em><\/p>\nTechnique 4: Sentiment Analysis<\/h1>\n
<\/figure>
<\/figure>
<\/figure>
\n
<\/figure>
Technique 5: Question Answering<\/h1>\n
<\/figure>
\n
<\/figure>
\n
Technique 6: Text Summarization<\/h1>\n
<\/figure>
\n
<\/figure>
\n
Technique 7: Attention Mechanism<\/h1>\n
<\/figure>
<\/figure>
<\/figure>
<\/figure>
Natural Language Processing Obstacles<\/h1>\n
\n
Conclusion<\/h1>\n