![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:2500\/1*9s0eyPQqGOhQW5fxmM90Zw.jpeg\")
<\/figure>Introduction<\/strong><\/h1>\nSpeech recognition is invading our lives. It\u2019s built into our phones (Siri), our game consoles (Kinect), our smartwatches (Apple Watch), and even our homes (Amazon Echo). But speech recognition has been around for decades, so why is it just now hitting the mainstream?<\/p>\n
The reason is that deep learning finally made speech recognition accurate enough to be useful outside of carefully-controlled environments. In this blog post, we\u2019ll learn how to perform speech recognition with 3 different implementations of popular deep learning frameworks.<\/p>\n
Note<\/strong>: The content of this blog post comes from Navdeep Jaitly\u2019s lecture at Stanford<\/a>. I\u2019d highly recommend watching his talk for the full details.<\/p><\/blockquote>\nSpeech Recognition \u2014 The Classic Way<\/strong><\/h1>\nIn the era of OK Google<\/em>, I might not really need to define ASR, but here\u2019s a basic description: Say you have a person or an audio source saying something textual, and you have a bunch of microphones that are receiving the audio signals. You can get these signals from one or many devices, and then pass them into an ASR system \u2014 whose job it is to infer the original source transcript that the person spoke or that the device played.<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*LnORfDgmydjugnbWws5-Pw.png\")
<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\nSo why is ASR important?<\/h2>\n
Firstly, it\u2019s a very natural interface for human communication. You don\u2019t need a mouse or a keyboard, so it\u2019s obviously a good way to interact with machines. You don\u2019t even really need to learn new techniques because most people learn to speak as a function of natural development. It\u2019s a very natural interface for talking with simple devices such as cars, handheld phones, and chatbots.<\/p>\n
So how is this done classically?<\/h2>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*6NVTrAW3UdsILOcJaxyGcA.png\")
<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\nAs observed above, the classic way of building a speech recognition system is to build a generative model of language. On the rightmost side, you produce a certain sequence of words from language models. And then for each word, you have a pronunciation model that says how this particular word is spoken. Typically it\u2019s written out as the sequence of phonemes \u2014 which are basic units of sound, but for our vocabulary, we\u2019ll just say a sequence of tokens \u2014 which represent a cluster of things that have been defined by linguistics experts.<\/p>\n
Then, the pronunciation models are fed into an acoustic model, which basically defines how does a given token sounds. These acoustic models are now used to describe the data itself. Here the data would be x, which is the sequence of frames of audio features from x1 to xT. Typically, these features are something that signal processing experts have defined (such as the frequency components of the audio waveforms that are captured).<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*GEhB6B2nzBICGqgs__DeDQ.png\")
<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\nEach of these different components in this pipeline uses a different statistical model:<\/p>\n
\n- In the past, language models were typically N-gram models, which worked very well for simple problems with limited speech input data. They are essentially tables describing the probabilities of token sequences.<\/li>\n
- The pronunciation models were simple lookup tables with probabilities associated with pronunciations. These tables would be very large tables of different pronunciations.<\/li>\n
- Acoustic models are built using Gaussian Mixture Models<\/a> with very specific architectures associated with them.<\/li>\n
- The speech processing was pre-defined.<\/li>\n<\/ul>\n
Once we have this kind of model built, we can perform the recognition by doing the inference on the data received. So you get a waveform, you compute the features for it (X) and do a search for Y that gives the highest probabilities of X.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n\n\nThe Neural Network Invasion<\/strong><\/h1>\nOver time, researchers started noticing that each of these components could work more effectively if we used neural networks.<\/p>\n
\n- Instead of the N-gram language models, we can build neural language models and feed them into a speech recognition system to restore things that were produced by a first path speech recognition system.<\/li>\n
- Looking into the pronunciation models, we can figure out how to do pronunciation for a new sequence of characters that we\u2019ve never seen before using a neural network.<\/li>\n
- For acoustic models, we can build deep neural networks (such as LSTM-based models) to get much better classification accuracy scores of the features for the current frame.<\/li>\n
- Interestingly enough, even the speech pre-processing steps were found to be replaceable with convolutional neural networks on raw speech signals.<\/li>\n<\/ul>\n
\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*tUBIQuzq1JcSSPyA_ESqOQ.png\")
<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\nHowever, there\u2019s still a problem. There are neural networks in each component, but they\u2019re trained independently with different objectives. Because of that, the errors in one component may not behave well with the errors in another component. So that\u2019s the basic motivation for devising a process where you can train the entire model as one big component itself.<\/p>\n
These so-called end-to-end models<\/strong> encompass more and more components in the pipeline discussed above. The 2 most popular ones are (1) Connectionist Temporal Classification<\/strong><\/mark> (CTC), which is in wide usage these days at Baidu and Google, but it requires a lot of training; and (2) Sequence-To-Sequence<\/strong> (Seq-2-Seq), which doesn\u2019t require manual customization.<\/p>\nThe basic motivation is that we want to do end-to-end speech recognition. We are given the audio X \u2014 which is a sequence of frames from x1 to xT, and the corresponding output text Y \u2014 which is a sequence of y1 to yL. Y is just a text sequence (transcript) and X is the audio processed spectrogram. We want to perform speech recognition by learning a probabilistic model p(Y|X): starting with the data and predicting the target sequences themselves.<\/p>\n
1 \u2014 Connectionist Temporal Classification<\/strong><\/h1>\nThe first of these models is called Connectionist Temporal Classification (CTC) ([1], [2], [3]). X is a sequence of data frames with length T: x1, x2, \u2026, xT, and Y is the output tokens of length L: y1, y2, \u2026, yL. Because of the way the model is constructed, we require T to be greater than L.<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*i70AUnRnHSauVRIJ1afWDg.png\")
<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\nThis model has a very specific structure that makes it suitable for speech:<\/p>\n
\n- You get the spectrogram at the bottom (X). You feed it into a bi-directional recurrent neural network, and as a result, the arrow pointing at any time step depends on the entirety of the input data. As such, it can compute a fairly complicated function of the entire data X.<\/li>\n
- This model, at the top, has softmax functions at every timeframe corresponding to the input. The softmax function is applied to a vocabulary with a particular length that you\u2019re interested in. In this case, you have the lowercase letters a to z and some punctuation symbols. So the vocabulary for CTC would be all that and an extra token called a blank token.<\/li>\n
- Each frame of the prediction is basically producing a log probability for a different token class at that time step. In the case above, a score s(k, t) is the log probability of category k at time step t given the data X.<\/li>\n<\/ul>\n
In a CTC model, if you look at just the softmax functions that are produced by the recurring neural network over the entire time step, you\u2019ll be able to find the probability of the transcript through these individual softmax functions over time.<\/p>\n
Let\u2019s take a look at an example (below). The CTC model can represent all these paths through the entire space of softmax functions and look at only the symbols that correspond to each of the time steps.<\/p>\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:416\/1*3d8our9f89mxbYf8e1Po2g.png\")
<\/figure><\/picture><\/div>\n<\/figure>\nAs seen on the left, the CTC model will go through 2 C symbols, then through a blank symbol, then produce 2 A symbols, then produce another blank symbol, then transition to a T symbol, and then finally produce a blank symbol again.<\/p>\n
So when you go through these paths with the constraint, you can only transition between the same phoneme from one step to the next. Therefore, you\u2019ll end up with different ways of representing an output sequence.<\/p>\n
For the example above, we have cc <b> aa <b> t <b> <\/strong>or cc <b> <b> a <b> t <b><\/strong> or cccc <b> aaaa <b> tttt <b>.<\/strong> Given these constraints, it turns out that even though there\u2019s an exponential number of paths by which you can produce the same output symbol, you can actually do it correctly using a dynamic programming algorithm. Because of dynamic programming, it\u2019s possible to compute both the log probability p(Y|X) and its gradient exactly. This gradient can be backpropagated to a neural network whose parameters can then be adjusted by your favorite optimizer!<\/p>\nBelow are some results for CTC, which show how the model functions on given audio. A raw waveform is aligned at the bottom, and the corresponding predictions are outputted at the top. You can see that it produces the symbol H at the beginning. At a certain point, it gets a very high probability, which means that the model is confident that it hears the sound corresponding to H.<\/p>\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:662\/1*35HAf2Agf8AT7pQ2MlZzbw.png\")
<\/figure><\/picture><\/div>\n<\/figure>\nHowever, there are some drawbacks to CTC language models. They often misspell words and struggle with grammar. So if you had some way to figure out how to rank the different paths produced from the model and re-rank them just by the language model, the results should be much better.<\/p>\n
Google actually fixed these problems by integrating a language model as part of the CTC model itself during training. That\u2019s the kind of production model currently being deployed with OK Google<\/em>.<\/p>\n2 \u2014 Sequence-To-Sequence<\/strong><\/h1>\nAn alternative approach to speech processing is the sequence-to-sequence model that makes next-step predictions. Let\u2019s say that you\u2019re given some data X and that you need to produce some symbols y1 to y{i}. The model predicts the probability of the next symbol of y{i+1}. The goal here is to basically learn a very good model for p.<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*EjsrlveQH-K3pNO4O8M_ag.png\")
<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\nWith the model architecture (left), you have a neural network (which is the decoder in a sequence-to-sequence model) that looks at the entire input (which is the encoder). It feeds in the path symbols that are produced as a recurrent neural network, and then you predict the next token itself as the output.<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*Vywp5idCLQb3iMhH2VSbcw.png\")
<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\nSo this model does speech recognition with the sequence-to-sequence framework. In translation, the X would be the source language. In the speech domain, the X would be a huge sequence of audio that\u2019s now encoded with a recurrent neural network.<\/p>\n
What it needs to function is the ability to look at different parts of temporal space, because the input is really long. Intuitively, translation results get worse as the source sentence becomes longer. That\u2019s because it\u2019s really difficult for the model to look in the right place. Turns out, that problem is aggravated a lot more with audio streams that are much longer. Therefore, you would need to implement an attention mechanism if you want to make this model work at all.<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*LfeKh4OPR-Jp4GVrpdEQ6w.png\")
<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\nSeen in the example on the left, you\u2019re trying to produce the 1st character C. You create an attention vector that essentially looks at different parts of the input time steps and produces the next chapter (which is A) after changing the attention.<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*b6SKvhvGLRUq4Su78NHSXQ.png\")
<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\nIf you keep doing this over the entire input stream, then you\u2019re moving forward attention just learned by the model itself. Seen here, it produces the output sequence \u201ccancel, cancel, cancel.\u201d<\/p>\n
The Listen, Attend, and Spell [4] model is the canonical model for the seq-2-seq category. Let\u2019s look at the diagram below taken from the paper:<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*LRPyLHYjoX_gk8dCjN2hQg.png\")
<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\n\n- In the Listener architecture, you have an encoder structure. For every time step of the input, it produces a vector representation that encodes the input and is represented as h_t at time step t.<\/li>\n
- In the Speller architecture, you have a decoder architecture. You generate the next character c_t at every time step t.<\/li>\n
- The LAS model uses a hierarchical encoder to replace the traditional recurrent neural network. Instead of processing one frame for every time step, it collapses neighboring frames as you feed into the next layer. Because of that, it reduces the number of time steps to be processed, thus making the processing faster.<\/li>\n<\/ul>\n
So what are the limitations of this model?<\/p>\n
\n- One of the big limitations preventing its use in an online system is that the output produced is being conditioned on the entire input. That means if you\u2019re going to put the model in a real-world speech recognition system, you\u2019d have to first wait for the entire audio to be received before outputting the symbol.<\/li>\n
- Another limitation is that the attention model itself is a computational bottleneck since every output token pays attention to every input time step. This makes it harder and slower for the model to do its learning.<\/li>\n
- Further, as the input is received and becomes longer, the word error rate goes down.<\/li>\n<\/ul>\n
3 \u2014 Online Sequence-to-Sequence<\/strong><\/h1>\nOnline sequence-to-sequence models are designed to overcome the limits of sequence-to-sequence models\u2014you don\u2019t want to wait for the entire input sequence to arrive, and you also want to avoid using the attention model itself over the entire sequence. Essentially, the intention is to produce the outputs as the inputs arrive. It has to solve the following problem: is the model ready to produce an output now that it\u2019s received this much input?<\/p>\n
The most notable online seq-2-seq model is called a Neural Transducer<\/strong> [5]. If you take the input as it comes in, and every so often at a regular interval, you can run a seq-2-seq model on what\u2019s been received in the last block. As seen in the architecture below, the encoder’s attention (instead of looking at the entire input) will focus only on a little block. The transducer will produce the output symbols.<\/p>\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:610\/1*h7IAUR8txM9hkJanypAdQA.png\")
<\/figure><\/picture><\/div>\n<\/figure>\nThe nice thing about the neural transducer is that it maintains causality. More specifically, the model preserves the disadvantage of a seq-2-seq model. It also introduces an alignment problem: in essence, what you want to know is that you have to produce some symbols as outputs, but you don\u2019t know which chunk these symbols should be aligned to.<\/p>\n
You can actually make this model better by incorporating convolutional neural networks, which are borrowed from computer vision. The paper [6] uses CNNs to do the encoder side in speech architecture.<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*MV_vtzZHZzvt-thxxiTDkg.png\")
<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\nYou take the traditional model for the pyramid as seen to the left, and instead of building the pyramid by simply stacking 2 things together, you can put a fancy architecture on top when you do the stacking. More specifically, as seen below, you can stack them as feature maps and put a CNN on the top. For the speech recognition problem, the frequency bands and the timestamps of the features that you look at will correspond to a natural substructure of the input data. The convolutional architecture essentially looks at that substructure.<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:586\/1*AH3Vm5bX937DlMTOm7v4lQ.png\")
<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\nConclusion<\/strong><\/h1>\nYou should now generally be up to speed on the 3 most common deep learning-based frameworks for performing automatic speech recognition in a variety of contexts. The papers that I\u2019ve referenced below will help you get into the nitty-gritty technical details of how they work if you\u2019re inclined to do that.<\/p>\n
References<\/strong><\/h2>\n[1] Graves, Alex, and Navdeep Jaitly. \u201cTowards End-To-End Speech Recognition with Recurrent Neural Networks<\/a>.\u201d ICML. Vol. 14. 2014.<\/p>\n[2] Amodei, Dario, et al. \u201cDeep speech 2: End-to-end speech recognition in english and mandarin<\/a>.\u201d arXiv preprint arXiv:1512.02595 (2015).<\/p>\n[3] H. Sak, A. Senior, K. Rao, O. Irsoy, A. Graves, F. Beaufays, and J. Schalkwyk, \u201cLearning acoustic frame labeling for speech recognition with recurrent neural networks<\/a>,\u201d in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2015.<\/p>\n[4] W. Chan, N. Jaitly, Q. Le, and O. Vinyals. \u201cListen, Attend, and Spell<\/a>,\u201d in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2015.<\/p>\n[5] N. Jaitly, D. Sussillo, Q. Le, O. Vinyals, I. Sutskever, and S. Bengio. \u201cA Neural Transducer<\/a>,\u201d arXiv preprint arXiv:1511.04868 (2016).<\/p>\n
Note<\/strong>: The content of this blog post comes from Navdeep Jaitly\u2019s lecture at Stanford<\/a>. I\u2019d highly recommend watching his talk for the full details.<\/p><\/blockquote>\n In the era of OK Google<\/em>, I might not really need to define ASR, but here\u2019s a basic description: Say you have a person or an audio source saying something textual, and you have a bunch of microphones that are receiving the audio signals. You can get these signals from one or many devices, and then pass them into an ASR system \u2014 whose job it is to infer the original source transcript that the person spoke or that the device played.<\/p>\n Firstly, it\u2019s a very natural interface for human communication. You don\u2019t need a mouse or a keyboard, so it\u2019s obviously a good way to interact with machines. You don\u2019t even really need to learn new techniques because most people learn to speak as a function of natural development. It\u2019s a very natural interface for talking with simple devices such as cars, handheld phones, and chatbots.<\/p>\n As observed above, the classic way of building a speech recognition system is to build a generative model of language. On the rightmost side, you produce a certain sequence of words from language models. And then for each word, you have a pronunciation model that says how this particular word is spoken. Typically it\u2019s written out as the sequence of phonemes \u2014 which are basic units of sound, but for our vocabulary, we\u2019ll just say a sequence of tokens \u2014 which represent a cluster of things that have been defined by linguistics experts.<\/p>\n Then, the pronunciation models are fed into an acoustic model, which basically defines how does a given token sounds. These acoustic models are now used to describe the data itself. Here the data would be x, which is the sequence of frames of audio features from x1 to xT. Typically, these features are something that signal processing experts have defined (such as the frequency components of the audio waveforms that are captured).<\/p>\n Each of these different components in this pipeline uses a different statistical model:<\/p>\n Once we have this kind of model built, we can perform the recognition by doing the inference on the data received. So you get a waveform, you compute the features for it (X) and do a search for Y that gives the highest probabilities of X.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n Over time, researchers started noticing that each of these components could work more effectively if we used neural networks.<\/p>\n However, there\u2019s still a problem. There are neural networks in each component, but they\u2019re trained independently with different objectives. Because of that, the errors in one component may not behave well with the errors in another component. So that\u2019s the basic motivation for devising a process where you can train the entire model as one big component itself.<\/p>\n These so-called end-to-end models<\/strong> encompass more and more components in the pipeline discussed above. The 2 most popular ones are (1) Connectionist Temporal Classification<\/strong><\/mark> (CTC), which is in wide usage these days at Baidu and Google, but it requires a lot of training; and (2) Sequence-To-Sequence<\/strong> (Seq-2-Seq), which doesn\u2019t require manual customization.<\/p>\n The basic motivation is that we want to do end-to-end speech recognition. We are given the audio X \u2014 which is a sequence of frames from x1 to xT, and the corresponding output text Y \u2014 which is a sequence of y1 to yL. Y is just a text sequence (transcript) and X is the audio processed spectrogram. We want to perform speech recognition by learning a probabilistic model p(Y|X): starting with the data and predicting the target sequences themselves.<\/p>\n The first of these models is called Connectionist Temporal Classification (CTC) ([1], [2], [3]). X is a sequence of data frames with length T: x1, x2, \u2026, xT, and Y is the output tokens of length L: y1, y2, \u2026, yL. Because of the way the model is constructed, we require T to be greater than L.<\/p>\n This model has a very specific structure that makes it suitable for speech:<\/p>\n In a CTC model, if you look at just the softmax functions that are produced by the recurring neural network over the entire time step, you\u2019ll be able to find the probability of the transcript through these individual softmax functions over time.<\/p>\n Let\u2019s take a look at an example (below). The CTC model can represent all these paths through the entire space of softmax functions and look at only the symbols that correspond to each of the time steps.<\/p>\n As seen on the left, the CTC model will go through 2 C symbols, then through a blank symbol, then produce 2 A symbols, then produce another blank symbol, then transition to a T symbol, and then finally produce a blank symbol again.<\/p>\n So when you go through these paths with the constraint, you can only transition between the same phoneme from one step to the next. Therefore, you\u2019ll end up with different ways of representing an output sequence.<\/p>\n For the example above, we have cc <b> aa <b> t <b> <\/strong>or cc <b> <b> a <b> t <b><\/strong> or cccc <b> aaaa <b> tttt <b>.<\/strong> Given these constraints, it turns out that even though there\u2019s an exponential number of paths by which you can produce the same output symbol, you can actually do it correctly using a dynamic programming algorithm. Because of dynamic programming, it\u2019s possible to compute both the log probability p(Y|X) and its gradient exactly. This gradient can be backpropagated to a neural network whose parameters can then be adjusted by your favorite optimizer!<\/p>\n Below are some results for CTC, which show how the model functions on given audio. A raw waveform is aligned at the bottom, and the corresponding predictions are outputted at the top. You can see that it produces the symbol H at the beginning. At a certain point, it gets a very high probability, which means that the model is confident that it hears the sound corresponding to H.<\/p>\n However, there are some drawbacks to CTC language models. They often misspell words and struggle with grammar. So if you had some way to figure out how to rank the different paths produced from the model and re-rank them just by the language model, the results should be much better.<\/p>\n Google actually fixed these problems by integrating a language model as part of the CTC model itself during training. That\u2019s the kind of production model currently being deployed with OK Google<\/em>.<\/p>\n An alternative approach to speech processing is the sequence-to-sequence model that makes next-step predictions. Let\u2019s say that you\u2019re given some data X and that you need to produce some symbols y1 to y{i}. The model predicts the probability of the next symbol of y{i+1}. The goal here is to basically learn a very good model for p.<\/p>\n With the model architecture (left), you have a neural network (which is the decoder in a sequence-to-sequence model) that looks at the entire input (which is the encoder). It feeds in the path symbols that are produced as a recurrent neural network, and then you predict the next token itself as the output.<\/p>\n So this model does speech recognition with the sequence-to-sequence framework. In translation, the X would be the source language. In the speech domain, the X would be a huge sequence of audio that\u2019s now encoded with a recurrent neural network.<\/p>\n What it needs to function is the ability to look at different parts of temporal space, because the input is really long. Intuitively, translation results get worse as the source sentence becomes longer. That\u2019s because it\u2019s really difficult for the model to look in the right place. Turns out, that problem is aggravated a lot more with audio streams that are much longer. Therefore, you would need to implement an attention mechanism if you want to make this model work at all.<\/p>\n Seen in the example on the left, you\u2019re trying to produce the 1st character C. You create an attention vector that essentially looks at different parts of the input time steps and produces the next chapter (which is A) after changing the attention.<\/p>\n If you keep doing this over the entire input stream, then you\u2019re moving forward attention just learned by the model itself. Seen here, it produces the output sequence \u201ccancel, cancel, cancel.\u201d<\/p>\n The Listen, Attend, and Spell [4] model is the canonical model for the seq-2-seq category. Let\u2019s look at the diagram below taken from the paper:<\/p>\n So what are the limitations of this model?<\/p>\n Online sequence-to-sequence models are designed to overcome the limits of sequence-to-sequence models\u2014you don\u2019t want to wait for the entire input sequence to arrive, and you also want to avoid using the attention model itself over the entire sequence. Essentially, the intention is to produce the outputs as the inputs arrive. It has to solve the following problem: is the model ready to produce an output now that it\u2019s received this much input?<\/p>\n The most notable online seq-2-seq model is called a Neural Transducer<\/strong> [5]. If you take the input as it comes in, and every so often at a regular interval, you can run a seq-2-seq model on what\u2019s been received in the last block. As seen in the architecture below, the encoder’s attention (instead of looking at the entire input) will focus only on a little block. The transducer will produce the output symbols.<\/p>\n The nice thing about the neural transducer is that it maintains causality. More specifically, the model preserves the disadvantage of a seq-2-seq model. It also introduces an alignment problem: in essence, what you want to know is that you have to produce some symbols as outputs, but you don\u2019t know which chunk these symbols should be aligned to.<\/p>\n You can actually make this model better by incorporating convolutional neural networks, which are borrowed from computer vision. The paper [6] uses CNNs to do the encoder side in speech architecture.<\/p>\n You take the traditional model for the pyramid as seen to the left, and instead of building the pyramid by simply stacking 2 things together, you can put a fancy architecture on top when you do the stacking. More specifically, as seen below, you can stack them as feature maps and put a CNN on the top. For the speech recognition problem, the frequency bands and the timestamps of the features that you look at will correspond to a natural substructure of the input data. The convolutional architecture essentially looks at that substructure.<\/p>\n You should now generally be up to speed on the 3 most common deep learning-based frameworks for performing automatic speech recognition in a variety of contexts. The papers that I\u2019ve referenced below will help you get into the nitty-gritty technical details of how they work if you\u2019re inclined to do that.<\/p>\n [1] Graves, Alex, and Navdeep Jaitly. \u201cTowards End-To-End Speech Recognition with Recurrent Neural Networks<\/a>.\u201d ICML. Vol. 14. 2014.<\/p>\n [2] Amodei, Dario, et al. \u201cDeep speech 2: End-to-end speech recognition in english and mandarin<\/a>.\u201d arXiv preprint arXiv:1512.02595 (2015).<\/p>\n [3] H. Sak, A. Senior, K. Rao, O. Irsoy, A. Graves, F. Beaufays, and J. Schalkwyk, \u201cLearning acoustic frame labeling for speech recognition with recurrent neural networks<\/a>,\u201d in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2015.<\/p>\n [4] W. Chan, N. Jaitly, Q. Le, and O. Vinyals. \u201cListen, Attend, and Spell<\/a>,\u201d in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2015.<\/p>\n [5] N. Jaitly, D. Sussillo, Q. Le, O. Vinyals, I. Sutskever, and S. Bengio. \u201cA Neural Transducer<\/a>,\u201d arXiv preprint arXiv:1511.04868 (2016).<\/p>\nSpeech Recognition \u2014 The Classic Way<\/strong><\/h1>\n
<\/figure>So why is ASR important?<\/h2>\n
So how is this done classically?<\/h2>\n
<\/figure><\/figure>\n
The Neural Network Invasion<\/strong><\/h1>\n
\n
<\/figure>1 \u2014 Connectionist Temporal Classification<\/strong><\/h1>\n
<\/figure>\n
<\/figure><\/figure>2 \u2014 Sequence-To-Sequence<\/strong><\/h1>\n
<\/figure><\/figure><\/figure><\/figure><\/figure>\n
\n
3 \u2014 Online Sequence-to-Sequence<\/strong><\/h1>\n
<\/figure><\/figure><\/figure>Conclusion<\/strong><\/h1>\n
References<\/strong><\/h2>\n