We’ve come to grips with the basics of CNNs, and so now we’d like to move onto RNNs. Recurrent neural networks are not feedforward; they’re allowed to have loops, so outputs from the previous computation(s) can be used as inputs to the present ones. This allows them to have memory, or to use past experiences to influence present decision-making.
One of the more popular variants these days of RNNs are so-called LSTM networks, which are somewhat more involved in that they possess the ability to forget if it’s deemed helpful. They’ve been used in many contexts, notably sentiment analysis and sequence generation. We’d like to use an LSTM-based architecture to generate a sequence of characters that reads like a Trump tweet, given the first 25 characters of a live Trump tweet as a seed.
As an example, Trump recently tweeted:
“I will be interviewed on @foxandfriends at 6:00 A.M. Enjoy!”
Our desired LSTM would take as its seed the first 25 characters of this tweet,
“I will be interviewed on”,
and construct an original tweet from this seed that ideally is indistinguishable from an actual Trump tweet. (Trump is a good target for this because he famously has the best words.) Our baseline is the LSTM architecture included with TFLearn, which, after training on a sample of 3000 Trump tweets from this summer, produces novel tweets like:
“I will be interviewed on @FoxNews will be many as the people on the dishonest to ver…”
Note that this architecture is given neither a dictionary of English words nor any information about English grammar; it learned them through training, and it does often make spelling mistakes and tend to ramble.
Our goal is to improve upon these tweets until they pass as actual Trump tweets, then set this bot up with a twitter account, and have it monitor Trump’s twitter account and post concurrently with the beginning of his tweet as a seed. This is inspired by the physics site arXiv vs snarXiv, where the user must guess the title of an actual physics paper, rather than the computer-generated name of a physics paper. If we succeed we could set up a Trump vs TrumpBot, or something similar.
The purpose of this series of blog posts is to determine how architecture affects quality of tweet, similar to our last series.