The Curious Programmer

What’s all this fuss about the AI AlphaGo’s recent victory against the masters?

While it’s seemed like AI had hit a dead-end as much as a decade ago, if you’re like many of us sci-fi enthusiasts and have always wanted an AI best friend, the recent victory of AlphaGo has brought us much closer than you may have thought was possible.

AI is Finally Moving Forward

We’re not surprised if you haven’t been following the recent developments in AI all that closely because, for the most part, it’s seemed like nothing exciting has happened for quite a long time. Sci-fi dreams about computer powered best friends aside, AI for the general public has come to mean reasonably responsive and well-programmed computer assistance rather than independent thinking machines. Concepts like ‘smart’ chatbots somehow seem to pull us further from the Star Trek or Heinlinian dream of fully sentient and intuitive computers while many products and services that claim to integrate AI seem to be nothing more than a fast way to analyze large amounts of data. In fact, the last time most of us heard something hopeful about AI was when Deep Blue beat the world Chess champion, but what ever came of that AI? Surely it hasn’t used that incredible logical power to take over the world or begin making friends, so what do we even care?

Not All AIs are Equal

The answer lies in the fact that there many forms of Artificial Intelligence and most of them are limited by the tasks they were made to perform. That’s what makes AlphaGo so special, because while it was designed, named, and trained to play Go against the masters, its potential functionalities go well beyond the realm of board games unlike most of its AI contemporaries.

While practical applications for specifically built AI are growing, the tradition of training your AI programming skills on classic strategy games has existed since the 1950s when a computer was programmed to play and was able to win a game of tic-tac-toe. Since then a large variety of games and custom built AIs have been tested against each other to the great entertainment of experts in the field and curious nerds like us who care about that sort of thing. The real difference is not what they’re programmed for but how they are programmed to start with and, in fact, this is also what most profoundly distinguishes Alpha Go from its older-generation relative, the Chess champion Deep Blue.

Chess is a Closed Game

You may not know this, but there is a standard way to program an AI to play a board game known as the Search Tree in which the computer analyzes all the pieces and spaces in a game and determines which move during its turn is most likely to result in victory. However, for games with a limited total number of moves and responses, you don’t even have to spend too much time on programming good judgment, all you need is a complete understanding of the game. That said, consider how long people have been playing, analyzing, and writing down their analysis for chess.

Every possible arrangement of the limited and highly specialized pieces on the board has been replicated and studied in-depth. Do you know what they found? There is a finite number of possible piece arrangements on the board and each one of these finite arrangements has a finite number of moves that can be made and each of these moves can then be judged as a good or bad idea. In other words, you can contain every possible chess move and the best move for each board arrangement into a single database. That’s right, the quick and dirty way to make a chess “AI” doesn’t even require any thought, simply a database containing a complete knowledge of the game. Therefore AIs were always destined to master chess because it can simply store everything there is to know and reference it at will.

So how did Deep Blue win back in the 90s? You can breath easy knowing that the famous AI did not use the database method but instead relied on a parallel system designed to run a complex tree search. At each point in the game it would analyze the board and run an assessment on the possible moves it detected and which could move it closer toward a win. Defeating the world chess champion was a huge victory for Deep Blue by more than just capturing a king piece. It indicated that the AI’s board assessment program could be faster and smarter than a human strategy expert, but it was not what most of us sci-fi enthusiasts would think of as the beginning of independent computer thought. The only thing Deep Blue can do is play chess and because chess is a finite game, Deep Blue never needs to get smarter.

Go is Not a Closed Game

People have been trying to define Go for thousands of years. With computer analysis in hand, they have tried to discover if it is a finite game, like chess, and it simply cannot be done. With a near-infinite number of pieces available to each player and the complexities of the game itself, there are too many possibilities, board arrangements, and good or bad placement choices for any reasonable purpose-built program to handle. While you can make a program that plays go, until AlphaGo computer opponents only ever reached an intermediate level of capability and trying to fill a database with all the possible board arrangements and possible moves might well catch your servers on fire.

AlphaGo Learns to Play

It is for this reason that many people, Go masters included, were certain that a computer could never learn to beat the human champions of the game and for this reason that DeepMind decided to try. Why has AlphaGo succeeded where other AIs were judged to not even have a chance? The difference was that DeepMind decided to try something new in the world of games vs AIs: Machine learning and neural networks instead of custom built search trees. AlphaGo doesn’t just judge the board, it learns from its mistakes. Like a go expert who has been playing since their early childhood, they ran AlphaGo through thousands of games against itself and it learned from every one of those games how to be a better player, improve its strategy, and it never gets bored, frustrated, or tired during practice.

AlphaGo Teaches the Masters

Two years ago, DeepMind felt that AlphaGo was ready to start playing against expert human opponents and invited the European Go champion Mr. Fan Hui to a closed-door five-game test. To their surprise and delight, it won every single game and became the first computer program to defeat a professional go player. They then set it against the legendary winner of 18 world titles, Mr. Lee Sedol in Seoul in which it won 4–1 and earned a 9-dan professional ranking, the highest certification available. If this wasn’t awesome enough, during these games AlphaGo dazzled the audience and its opponent with creative winning moves, one of which effectively overturned hundreds of years of cumulative go wisdom.

DeepBlue Was Columbus Discovering America And AlphaGo Is The Moon Landing

Any computer scientist or programmer will admit that DeepBlue achieved something incredible when it beat Kasparov. But the amazing feat was in the computational power that DeepBlue had. It did not learn to play chess. It was programmed to search through thousands of chess games and evaluate the best move it had. Once DeepBlue had won the game and proven its strength, it was packed away and it has not been seen since. Everyone knew that its only purpose was to play chess and its programming could not be applied to much of anything else. AlphaGo, on the other hand, took the idea of computational power and added human reasoning or intuition — this combination makes it incredibly applicable to countless purposes.

Computer Scientists Versus Chess Masters

Another very unique aspect of how AlphaGo was created versus how DeepBlue was created is who the experts relied on. With DeepBlue, the computer scientists heavily relied on Chess experts, professionals, and masters to help the program have as many chess games programmed into it as possible. And the thing is, even after DeepBlue had strutted its stuff, it did not change much for the world of Chess. Chess players did not learn anything from it. With AlphaGo, however, the computer scientists simply used lots and lots of games from a myriad of players, who were all at different levels of Go knowledge and experience. And unlike when DeepBlue was unveiled, when AlphaGo was first shown to the world, Go players paid attention. They saw that AlphaGo was playing in innovative ways. It has taught them to think and play more creatively.

AlphaGo’s Intuitive Factor

It is easy to say that AlphaGo has intuition, which DeepBlue was missing. It is much more difficult to explain where that intuition comes from. To put it simply, it built on DeepBlue’s search and optimize idea. The DeepMind team programmed AlphaGo with 150,000 Go games that had been played by good players. It would then search through those games to base its next move on probability. To take AlphaGo to the next level, though, DeepMind used a neural network, or machine learning, so that through self-play and play against humans it could slowly make millions of tiny adjustments, allowing it to obtain something as close to intuition as possible.

And it is this intuition factor of recognizing good patterns and learning them that will have a much deeper impact on artificial intelligence. In the world of art, this type of artificial intelligence will expose a neural network to a specific artistic style, it will then show the network an image, and the network will replicate that image in the artistic style it was shown. In the world of language, the same neural networks are being used to recognize natural language. In the world of games, these networks are employed to improve video game experiences. And the list of future possibilities for expanding the impact of neural networks, machine learning, and artificial intelligence to provide the ability of intuition to computers is growing by the day — Think healthcare, smartphone assistants, and robotics. In fact, UK’s National Health Service has already signed a deal with DeepMind.

It Was Not Supposed To Be This Easy

Go is a game that has been around for 3000 years. It is widely accepted as the most challenging strategy game that exists. Individuals, especially in countries like South Korea and China, are sent to private school specifically to learn how to play the game at an expert level. It takes years of playing for several hours every day to master the game. In other words, even though it has simple rules, it is not a simple game to excel at. And due to its complexity, and how long it had taken computer scientists to create a machine that could win at Chess, experts estimated that a machine that could effectively play Go would only be created in about 10 years.

Surprise! Deep Mind managed to create a machine that could master the game, without being programmed with explicit rules and without being taught by a professional Go player. AlphaGo mainly played against itself and learned from this self-play. At its core, it learned like a human learns, by looking at the board, evaluating the options, making moves, and learning from mistakes — it just did it a lot faster than any human can.

This is extremely exciting because, at its core, what it means is that computer scientists have had all the tools they needed to do this for years. Neural networks have been known about and discussed since the middle of the last century. All it really took was simply getting creative with them, applying them in new ways. AlphaGo beating the world’s best Go player proves that AI has the potential to do anything. It can learn anything and understand anything, and from that learning and understanding it can accomplish what humans can accomplish in a much shorter period of time.

You’re probably wondering what this all means. The good news is that we’re much closer to the dream of an AI best friend than most of us would have dared to imagine a few years ago. Let it sink in for a moment: AlphaGo can learn the most complex, intuition and creativity based logic game known to man and it didn’t do so through a finite database or search trees alone. It learned from practice and experience, just like we do, and the ability to create amazing new solutions to ancient puzzles suggests a realm of digital creativity never before fathomed.

AlphaGo is not like other game playing AIs that have come before it. It is the future of intelligent and intuitive machines, one that we plan to turn toward more than just board games. From practical applications to that friend you’ve been hoping for, AlphaGo is sure to be the first of a new generation of self-learning intuitive AIs that go above and beyond the limited calculating capacities of its older siblings and contemporaries. If you love AI like I do, keep your eyes open for new practical applications for very real artificial intelligence popping up in places you may not have even imagined. The AI winter is over.

Java isn’t just a language; it’s an ecosystem. You can write code for the JVM without writing any Java. This gives you the option of using a more modern language. Some of the shortcomings of Java are obvious. It makes you write a lot of boilerplate code. It supports functional programming only as an afterthought; the lambda feature is a kludge. The NullPointerException is every Java programmer’s bane.

In 2004, a group led by Martin Odersky released an updated version of the language, called Scala (“scalable language”). It added features such as objects for everything, functions as assignable data, type inference, and pattern matching. It compiles to Java bytecodes and can be mixed with Java code.

Another language aimed at the same goals is Kotlin, released by JetBrains in 2012. It built on people’s experience with Scala. A common complaint with Scala is slow compilation time, and Kotlin offers compile speeds comparable to Java. It’s recently gotten a big boost from Google, which has declared it a first-class language for Android development.

If some features of Java constantly annoy you, you’ll find things to like in both languages. If you’re annoyed enough to make the jump, which way should you go? Should you choose the maturity of Scala or the freshness of Kotlin? There are benefits to each.

Solving problems different ways

Kotlin and Scala, like Java, are statically typed. Whatever type a variable starts out as, it will keep it for its whole life. But both of them save you some of the effort of declaring every variable. You can implicitly declare a type with an initializer. In either language you can write

var count = 1

That makes count an integer. Notice that no semicolon is required. The difference between the languages is that Scala goes much further in allowing implicit conversions. If you use x.transmogrify(), and x belongs to a class which doesn’t have a transmogrify function, that isn’t necessarily an error. You can create an implicit class which has a transmogrify method, and the compiler will figure out, without making you do any casting, whether it can step in to do the job.

Kotlin’s creators found this a little too free-wheeling. It lets you define extension methods on a class, adding custom functionality. You can do this even on standard data types. (Remember, everything is an object, so every data type is a class. Boxing of simple data types is no longer needed.)

Null values are a huge headache in Java. Scala helps to relieve this in a couple of ways. First, variables must be initialized. You can initialize them to the default value (var a:Int = _), which is often null, but at least it makes you aware you’re doing it. Second, the Option class helps in guaranteeing null-safety in parameters and returned values. It’s one of the more complicated features of the language to understand, but it gives you a lot of control.

Kotlin gets right to the point. By default, it doesn’t allow variables to have the value null. You can declare a variable to be nullable if you really need to, by putting a question mark after the type. If you’ve worked with Swift, this approach will sound familiar. If you use nullable values, the compiler does extensive checking to make sure you aren’t putting them at risk of a NullPointerException and will give you a compile-time error if you are.

Java makes you use a regular class or an enum if you just want to package some data together in an object. Scala and Kotlin offer some better options. Scala gives you the case class, which a specialized class for data objects. It automatically defines accessor functions (why doesn’t Java just do that?). Instances are compared by structure rather than reference. A copy function is automatically provided to do a shallow copy.

Kotlin’s data class does pretty much the same thing. The main difference is that Scala has a powerful pattern matching feature which Kotlin didn’t pick up. A match statement is like a bionically enhanced case statement. Patterns can check not only literal values but types, lists, and ranges. Scala can do matching on all kinds of objects, but the feature is especially powerful with case classes.

Scala provides strong support for XML. You can put XML directly into Scala code and assign it to an XML object. This creates complications, since a <operator that isn’t followed by a space may be read as the start of an XML expression. Kotlin uses the more traditional approach of classes to handle XML objects.

Type classes are a feature of Scala that doesn’t have an equivalent in Kotlin. A type class defines a set of operations which member classes must support. This isn’t like subclassing in Java; a type class can be added to types that already exist. It lets the developer create new kinds of polymorphism with existing types. Extension functions in Kotlin aren’t the same thing, but they let you add common ground to different types, so they address some of the same needs.

The feel of the language

OK, there are differences between the languages, but they aim at more or less the same thing. You can learn either one. Are there bigger, more philosophical reasons for choosing one or the other?

To some people, the difference is that Scala is more aimed at exploring new ideas, and Kotlin is more focused on getting results. Kotlin’s emphasis on fast compilation and its removal of some of Scala’s more esoteric features reflect this. Scala just lets you do lots of things. The operator name ?:+ appears to be legal, and maybe there’s a reason you’d want to use it. Kotlin is more restrictive. Some would say saner.

If you love functional programming, Scala has more of its features than Kotlin. Type classes are a functional programming feature. As another example, Scala supports currying and partial application, which are ways to break down functions that take multiple arguments. This provides additional flexibility in using argument lists. Kotlin provides ways to do the same things, but they might not be as mathematically elegant.

People who have learned Scala thoroughly love it. It takes more effort, but it lets developers do things they can’t do in Kotlin. Kotlin adherents often find that much flexibility more confusing than useful.

Practical considerations

Sometimes the realities of what you’re trying to do are the main factor. You need to pick the language that will let you do the job, even if you don’t like it quite as much. If you ‘re going to do Android development, Kotlin is the only choice. Android doesn’t use Oracle’s JVM, so you can’t use any old JVM-compatible language. Kotlin has the tools for compiling, debugging, and running software on Android. It’s built into Android Studio, starting with version 3.0.

Outside Android, Kotlin’s options are more limited. Are you committed to Eclipse for your IDE? You can use it to work with both languages, up to a point. The Scala IDE for Eclipse is more mature than the Kotlin plugin, which is a bit painful to set up. Some users have reported trouble getting the latter to work. The situation for NetBeans is similar. With Kotlin’s growing popularity, the situation may be more equal in a year or two. If you like working from the command line, the IDE situation isn’t an issue, and Kotlin has all the necessary tools.

Kotlin is still maturing, but many Java people find adopting it is an easier transition than Scala is. The one that works best for your needs will depend on your personal style and your practical aims. Look at both carefully before making a decision.

If you enjoyed this article please share it! This is the biggest compliment you can give a writer! Thank you!

Artificial intelligence (AI) is a lot like life’s relationships. Sometimes what you put into it is pretty straightforward, leading to the output or outcome that you wanted. Other times, let’s just say, the process gets a bit more convoluted and sometimes the outcome isn’t exactly what you envisioned. In other words, you may input the same into both relationships, but different paths lead you to different results. Nevertheless, both are learning processes. In the AI world, this is called supervised and unsupervised deep learning–and like most relationships, the shortest distance between what you input to what you get as output isn’t always the proverbial straight line.

What is Deep Learning?

Before we delve into what supervised and unsupervised deep learning is, you should know that deep learning evolved from a process called machine learning. Machine learning employs an algorithm, or set of rules, that creates output without specific programming. Think about how social networking mines data from your posts. For instance, you go out to eat with your friends at your favorite sushi place and share facts online about your experience–what you loved, found distasteful, photos, would you return–once you input these into your social network, an algorithm picks up tidbits about your input to extract patterns about what you like, don’t like, even what you look like based upon your pictures. The algorithm may discover that you are around 23 years old, eat out at this particular type of restaurant twice a month with your friends and like California rolls over eel sushi. It then sends you ads based upon that data. Machine learning iteratively gleans information about input despite not being told how to do so or where to look for that information.

Deep learning kicks it up a notch. It takes your input, finds that it can either categorize it without issue (supervised) or clusters unlabeled information, attempting to categorize it so that it makes sense (unsupervised), before taking that input and creating some sort of viable output. It’s a layered architecture making sense of data that can be quite abstract from one layer to another. That’s how deep learning emulates the multi-faceted complexity of the human brain–its neural pathways processing copious amounts of information that doesn’t make sense until it does (or not).

Supervised Deep Learning: The KISS Pathway That Leads To The Expected

What happens when your supervisor’s hanging over your shoulder at work? Like most, it drives you batty, so you tend to take the path of least resistance to find the most non-challenging way to get your job done quickly and still meet the expectations of your supervisor, right? Let’s say that particular supervisor trained you to process credit applications. Said supervisor knows what’s in those applications and that the expected outcome of any application is approval or not approval. You learned from your training set how to function in the best way to get to the desired outcome, i.e. the results that your supervisor needs. Supervised deep learning is like that. We humans tend to process in a specific hierarchy: we take in life’s input and based on our experiences (training), we organize that input so that our prior knowledge can make sense of it, process it to some expected outcome. Supervised deep learning belongs to that Keep-It-Simple-Stupid (KISS) pathway, that literal path of least resistance leading to some fulfilled expectation.

Supervised deep learning is well suited for decision-making: take our credit card example for instance. The bank takes your application and runs it against its categories of risk before taking action for or against approving you. Here’s the procedural gist:

Application is input from customer

The bank inputs data from application into the algorithm

The algorithm notices from past applications that data follows certain pathways (modeling)

For example: marital status–single, married, divorced, widowed all have a yes or no answer

The algorithm takes that application data, the yes or no answers, as determined by the bank and follows its flowchart (pathway rules)

Data flows through that pathway as the algorithm decides which of the primary categories of approved and not approved the data belongs in

The expected decision of approved or not approved is rendered

The customer is approved and is a happy camper or is not approved and wonders how to fix his credit score (had to throw that in).

Supervised deep learning is more than your typical lights on, lights off binary function. The algorithm classifies criteria into the bank’s expectation of risk, processing that risk into one of two decisions. This method of classification is known as binomial classification (two choices) or multi-class (more than two choices).

Unsupervised Deep Learning: An Exploratory Journey To Figuring Out the Unknown

If supervised deep learning is a path to expected output, unsupervised deep learning takes that same input and attempts to make sense of it before eschewing some output. Let’s take a trip to the art museum with your best friend as an example. You both become captivated with a painting of a rose. One of you sees it rather literally, the other sees it figuratively. To you, a rose is a just a rose and you want to move on to the Van Gogh exhibit. To your friend that rose is yellow when it should be red and your friend cannot figure out why the painting denotes friendship and not love. There’s no Van Gogh until there’s ready to go–and that’s not happening until your friend muses about that rose and why her current relationship is hanging on that museum wall.

Unsupervised deep learning has no target, no expectation from the input. It relies on exploring layers of possibilities to get to some conclusion. While you can move on to the Van Gogh exhibit, your friend struggles to figure out how to classify all the many pathways friendship and love can take someone from convolution to happy life and how one can learn from their mistakes.

Decision Time: If You Knew Then What You Know Now

Humans are subjectively sentient creatures with decision-making processes that cater more to the unexpected (unsupervised deep learning) than to the expected (supervised deep learning). Computers don’t have the human factor. They don’t have experiences. They just have data sets, functions, and “thinking” based on layers of pooling information together in either ordered or non-ordered ways.

As neural nets and AI become more complex, so do the deep learning algorithms. You can choose among supervised, unsupervised or a combo-pack of deep learning to tackle anything from credit approvals to the complexities of mind-boggling, robotic data sets. Remember the social networking example? When you uploaded images, something called Convolutional Neural Networks (CNN) picked out traits before it came to the conclusion that you around 23, pooling together relevant data: restaurant, friends laughing, friends frowning, facial recognition, background recognition. Combine and categorize those subgroups and your image spoke volumes about who you are and how you live. Imagine what they’d unpack from what you say on an uploaded video (Recurrent NN)? Yet, sometimes life has to unfold unsupervised by knowns, reconstructing (autoencoding) the data-driven universe while self-organizing maps translate often nebulous data patterns into two-dimensions (think topographic maps) that allow you further muse as to why that rose by any other name is just backpropagation (wink).

Artificial intelligence has been in the background for decades, kicking up dust in the distance, but never quite arriving. Well that era is over. In 2017, AI has broken through the dust cloud and arrived in a big way. But why? What’s the big deal all of a sudden? And what do recurrent neural networks have to do with it? Well, a lot, actually. Thanks to an ingenious form of short-term memory that is unheard of in conventional neural networks, today’s recurrent neural networks (RNNs) have been proving themselves as powerful predictive engines. When it comes to certain sequential machine learning tasks, such as speech recognition, RNNs are reaching levels of predictive accuracy, time and time again, that no other algorithm can match. However, the first generation of RNNs, back in the day, were not so hot. They suffered from a serious setback in their error-tweaking process that held up their progress for decades. Finally, a major breakthrough came in the late 90s that led to a new generation of far more accurate RNNs. Building on that breakthrough for nearly twenty years, developers refined and perfected their new RNNs until all-star apps such as Google Voice Search and Apple’s Siri started snatching them up to power key processes. Now recurrent networks are showing up everywhere, and are helping to ignite the AI renaissance that’s unfolding right now.

Neural Networks That Cling to the Past

Most artificial neural networks, such as feedforward neural networks, have no memory of the input they received just one moment ago. For example, if you provide a feedforward neural network with the sequence of letters “WISDOM,” when it gets to “D,” it has already forgotten that it just read “S.” That’s a big problem. No matter how hard you train it, it will always struggle to guess the most likely next character: “O.” This makes it a rather crappy candidate for certain tasks, such as speech recognition, that greatly benefit from the capacity to predict what’s coming next. Recurrent networks, on the other hand, do remember what they’ve just encountered, and at a remarkably sophisticated level.

Let’s take the example of the input “WISDOM” again and apply it to a recurrent network. The unit, or artificial neuron, of the RNN, upon receiving the “D” also takes as its input the character it received one moment ago, the “S.” In other words, it adds the immediate past to the present. This gives it the advantage of a limited short-term memory that, along with its training, provides enough context for guessing what the next character is most likely to be: “O.”

Tweaking and Re-tweaking

If you like to get into the weeds, this is where you get excited. Otherwise, get ready for a rough patch. But hang in there, it’s worth it. Like all artificial neural networks, the units of an RNN assign a matrix of weights to their multiple inputs, then apply a function to those weights to determine a single output. However, recurrent networks apply weights not only to their present inputs, but also to their inputs from a moment ago. Then they adjust the weights assigned to their present and past inputs through a process that involves two key concepts that you’ll definitely want to know if you really want to get into AI: gradient descent and backpropogation through time (BPTT).

Gradient Descent

One of the most famous algorithms in machine learning is known as gradient descent. Its primary virtue is its remarkable capacity to sidestep the dreaded “curse of dimensionality.” This issue plagues systems, such as neural networks, with far too many variables to make a brute-force calculation of their optimal values possible. Gradient descent, however, breaks the curse of dimensionality by zooming in on the local low-point, or local minimum, of the multi-dimensional error or cost function. This helps the system determine the tweaked value, or weight, to assign to each of the units in the network, bringing accuracy back in line.

Backpropogation Through Time

The RNN trains its units by adjusting their weights following a slight modification of a feedback process known as backpropogation. Okay, this is a weird concept. But if you’re into AI, you’ll learn to love it. The process of backpropogation works its way back, layer by layer, from the network’s final output, tweaking the weights of each unit, or artificial neuron, according to the unit’s calculated portion of the total output error. Got it? If so, get ready for one more layer of complexity. Recurrent neural networks use a heavier version of this process known as backpropogation through time (BPTT). This version extends the tweaking process to include the weight of the T-1 input values responsible for each unit’s memory of the prior moment.

Yikes: The Vanishing Gradient Problem

Despite enjoying some initial success with the help of gradient descent and BPTT, many artificial neural networks, including the first generation of RNNs, eventually ran out gas. Technically, they suffered a serious setback known as the vanishing gradient problem. Although the details fall way outside the scope of this sweeping overview, the basic idea is pretty straightforward. First, let’s look at the notion of a gradient. Like its simpler relative, the derivative, you can think of a gradient as a slope. In the context of training a deep neural network, the larger the gradient, the steeper the slope, the more quickly the system can roll downhill to the finish line and complete its training. But this is where developers ran into trouble — their slopes were too flat for fast training. This was particularly problematic in the first layers of their deep networks, which are the most critical when it comes to proper tweaking of memory units. Here the gradient values got so small, and their corresponding slopes so flat, that one could describe them as “vanishing,” thus the vanishing gradient problem. As the gradients got smaller and smaller, and thus flatter and flatter, the training times grew unbearably long. It was an error-correction nightmare without end.

The Big Breakthrough: Long Short-Term Memory

Finally, in the late 90s, a major breakthrough solved the vanishing descent problem and gave a second wind to recurrent network development. At the center of this new approach were units of long short-term memory (LSTM).

As weird as that sounds, the long and short of it is that LSTM made a world of difference in the field AI. These new units, or artificial neurons, like the standard short-term memory units of RNNs, remember their inputs from a moment ago. However, unlike standard RNN units, LSTMs can hang on to their memories, which have read/write properties akin to memory registers in a conventional computer. Yet LSTMs have analog, rather than digital, memory, making their functions differentiable. In other words, their curves are continuous and you can find the steepness of their slopes. So they are a good fit for the partial differential calculus involved in backpropogation and gradient descent.

Altogether, LSTMs can not only tweak their weights, but retain, delete, transform and otherwise control the inflow and outflow of their stored data according to the quirks of their training. Most importantly, LSTMs can cling to important error information for long enough to keep gradients relatively steep and thus training periods relatively short. This wipes out the vanishing gradient problem and greatly improves the accuracy of today’s LSTM-based recurrent networks. Thanks to this remarkable improvement in the RNN architecture, Google, Apple and many other leading companies, not to mention startups, are now using RNNs to power applications at the center of their businesses. In short, RNNs are suddenly a big deal.

What to Remember about RNNs

Let’s recap the highlights of these amazing memory machines. Recurrent neural networks, or RNNs, can remember their former inputs, which gives them a big edge over other artificial neural networks when it comes to sequential, context-sensitive tasks such as speech recognition. However, the first generation of RNNs hit the wall when it came to their capacity to correct for errors through the all-important twin processes of backpropogation and gradient descent. Known as the dreaded vanishing gradient problem, this stumbling block virtually halted progress in the field until 1997, when a major breakthrough introduced a vastly improved LSTM-based architecture to the field. The new approach, which effectively turned each unit in a recurrent network into an analogue computer, greatly increased accuracy and helped lead to the renaissance in AI we’re seeing all around us today.

If you have enjoyed this post, the biggest compliment you could give would be to share this with someone that you think would enjoy it!

If you would like to see more articles like this, click the subscribe button and never miss a post. Have a great day and never stop learning!