In this project, a Recurrent Neural Networks is used to generate a TV script for a scene at Moe’s Tavern using some of the data from Simpsons dataset of scripts from 27 seasons. Here is the code for this project which was a part of Udacity’s Deep Learning Nanodegree.

Get the Data

The data is already provided for you. You’ll be using a subset of the original dataset. It consists of only the scenes in Moe’s Tavern. This doesn’t include other versions of the tavern, like “Moe’s Cavern”, “Flaming Moe’s”, “Uncle Moe’s Family Feed-Bag”, etc..

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import helper

data_dir = './data/simpsons/moes_tavern_lines.txt'
text = helper.load_data(data_dir)
# Ignore notice, since we don't use it for analysing the data
text = text[81:]
# print(text)
Moe_Szyslak: (INTO PHONE) Moe's Tavern. Where the elite meet to drink.
Bart_Simpson: Eh, yeah, hello, is Mike there? Last name, Rotch.
Moe_Szyslak: (INTO PHONE) Hold on, I'll check. (TO BARFLIES) Mike Rotch. Mike Rotch. Hey, has anybody seen Mike Rotch, lately?
Moe_Szyslak: (INTO PHONE) Listen you little puke. One of these days I'm gonna catch you, and I'm gonna carve my name on your back with an ice pick.
Moe_Szyslak: What's the matter Homer? You're not your normal effervescent self.
Homer_Simpson: I got my problems, Moe. Give me another one.
Moe_Szyslak: Homer, hey, you should not drink to forget your problems.
Barney_Gumble: Yeah, you should only drink to enhance your social skills.

Explore the Data

Play around with view_sentence_range to view different parts of the data.

view_sentence_range = (0, 10)

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np

print('Dataset Stats')
print('Roughly the number of unique words: {}'.format(len({word: None for word in text.split()})))
scenes = text.split('\n\n')
print('Number of scenes: {}'.format(len(scenes)))
sentence_count_scene = [scene.count('\n') for scene in scenes]
print('Average number of sentences in each scene: {}'.format(np.average(sentence_count_scene)))

sentences = [sentence for scene in scenes for sentence in scene.split('\n')]
print('Number of lines: {}'.format(len(sentences)))
word_count_sentence = [len(sentence.split()) for sentence in sentences]
print('Average number of words in each line: {}'.format(np.average(word_count_sentence)))

print()
print('The sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))
Dataset Stats
Roughly the number of unique words: 11492
Number of scenes: 262
Average number of sentences in each scene: 15.248091603053435
Number of lines: 4257
Average number of words in each line: 11.50434578341555

The sentences 0 to 10:
Moe_Szyslak: (INTO PHONE) Moe's Tavern. Where the elite meet to drink.
Bart_Simpson: Eh, yeah, hello, is Mike there? Last name, Rotch.
Moe_Szyslak: (INTO PHONE) Hold on, I'll check. (TO BARFLIES) Mike Rotch. Mike Rotch. Hey, has anybody seen Mike Rotch, lately?
Moe_Szyslak: (INTO PHONE) Listen you little puke. One of these days I'm gonna catch you, and I'm gonna carve my name on your back with an ice pick.
Moe_Szyslak: What's the matter Homer? You're not your normal effervescent self.
Homer_Simpson: I got my problems, Moe. Give me another one.
Moe_Szyslak: Homer, hey, you should not drink to forget your problems.
Barney_Gumble: Yeah, you should only drink to enhance your social skills.

Implement Preprocessing Functions

The first thing to do to any dataset is preprocessing. Implement the following preprocessing functions below:

  • Lookup Table
  • Tokenize Punctuation

Lookup Table

To create a word embedding, you first need to transform the words to ids. In this function, create two dictionaries:

  • Dictionary to go from the words to an id, we’ll call vocab_to_int
  • Dictionary to go from the id to word, we’ll call int_to_vocab

Return these dictionaries in the following tuple (vocab_to_int, int_to_vocab)

import numpy as np
import problem_unittests as tests

def create_lookup_tables(text):
    """
    Create lookup tables for vocabulary
    :param text: The text of tv scripts split into words
    :return: A tuple of dicts (vocab_to_int, int_to_vocab)
    """
    # TODO: Implement Function
    # print(text[:10])
    unique_words = set(text)
    # print (unique_words)
    
    vocab_to_int = {}
    int_to_vocab = {}
    
    for i, word in enumerate(unique_words):
        vocab_to_int[word] = i
        int_to_vocab[i] = word
    
    
    return (vocab_to_int, int_to_vocab)


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_create_lookup_tables(create_lookup_tables)
Tests Passed

Tokenize Punctuation

We’ll be splitting the script into a word array using spaces as delimiters. However, punctuations like periods and exclamation marks make it hard for the neural network to distinguish between the word “bye” and “bye!”.

Implement the function token_lookup to return a dict that will be used to tokenize symbols like “!” into “||Exclamation_Mark||”. Create a dictionary for the following symbols where the symbol is the key and value is the token:

  • Period ( . )
  • Comma ( , )
  • Quotation Mark ( “ )
  • Semicolon ( ; )
  • Exclamation mark ( ! )
  • Question mark ( ? )
  • Left Parentheses ( ( )
  • Right Parentheses ( ) )
  • Dash ( – )
  • Return ( \n )
This dictionary will be used to token the symbols and add the delimiter (space) around it. This separates the symbols as it’s own word, making it easier for the neural network to predict on the next word. Make sure you don’t use a token that could be confused as a word. Instead of using the token “dash”, try using something like “   dash   ”.
def token_lookup():
    """
    Generate a dict to turn punctuation into a token.
    :return: Tokenize dictionary where the key is the punctuation and the value is the token
    """
    # TODO: Implement Function
    tokens_dict = {
        '.': '||period||',
        ',': '||comma||',
        '"': '||quotation_mark||',
        ';': '||semicolon||',
        '!': '||exclamation_mark||',
        '?': '||question_mark||',
        '(': '||left_parentheses||',
        ')': '||right_parentheses||',
        '--': '||dash||',
        '\n': '||return||'
    }
    return tokens_dict

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_tokenize(token_lookup)
Tests Passed

Preprocess all the data and save it

Running the code cell below will preprocess all the data and save it to file.

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(data_dir, token_lookup, create_lookup_tables)

Check Point

This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import helper
import numpy as np
import problem_unittests as tests

int_text, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()

Build the Neural Network

You’ll build the components necessary to build a RNN by implementing the following functions below:

  • get_inputs
  • get_init_cell
  • get_embed
  • build_rnn
  • build_nn
  • get_batches

Check the Version of TensorFlow and Access to GPU

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
from distutils.version import LooseVersion
import warnings
import tensorflow as tf

# Check TensorFlow Version
assert LooseVersion(tf.__version__) >= LooseVersion('1.3'), 'Please use TensorFlow version 1.3 or newer'
print('TensorFlow Version: {}'.format(tf.__version__))

# Check for a GPU
if not tf.test.gpu_device_name():
    warnings.warn('No GPU found. Please use a GPU to train your neural network.')
else:
    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))
TensorFlow Version: 1.3.0
Default GPU Device: /gpu:0

Input

Implement the get_inputs() function to create TF Placeholders for the Neural Network. It should create the following placeholders:

  • Input text placeholder named “input” using the TF Placeholder name parameter.
  • Targets placeholder
  • Learning Rate placeholder

Return the placeholders in the following tuple (Input, Targets, LearningRate)

import numpy as np
import problem_unittests as tests

def get_inputs():
    """
    Create TF Placeholders for input, targets, and learning rate.
    :return: Tuple (input, targets, learning rate)
    """
    # TODO: Implement Function
    Input = tf.placeholder(tf.int32, [None, None], name = "input")
    Targets = tf.placeholder(tf.int32, [None, None], name = "targets")
    LearningRate = tf.placeholder(tf.float32, name = "learning_rate")
    return (Input, Targets, LearningRate)


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_inputs(get_inputs)
Tests Passed

Build RNN Cell and Initialize

Stack one or more BasicLSTMCells in a MultiRNNCell.

  • The Rnn size should be set using rnn_size
  • Initalize Cell State using the MultiRNNCell’s zero_state() function
    • Apply the name “initial_state” to the initial state using tf.identity()

Return the cell and initial state in the following tuple (Cell, InitialState)

def get_init_cell(batch_size, rnn_size):
    """
    Create an RNN Cell and initialize it.
    :param batch_size: Size of batches
    :param rnn_size: Size of RNNs
    :return: Tuple (cell, initialize state)
    """
    # TODO: Implement Function
    lstm = tf.contrib.rnn.BasicLSTMCell(rnn_size)
    Cell = tf.contrib.rnn.MultiRNNCell([lstm] * 3)
    InitialState = Cell.zero_state(batch_size, tf.float32)
    InitialState = tf.identity(InitialState, name="initial_state")    
    return (Cell, InitialState)


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_init_cell(get_init_cell)
Tests Passed

Word Embedding

Apply embedding to input_data using TensorFlow. Return the embedded sequence.

def get_embed(input_data, vocab_size, embed_dim):
    """
    Create embedding for <input_data>.
    :param input_data: TF placeholder for text input.
    :param vocab_size: Number of words in vocabulary.
    :param embed_dim: Number of embedding dimensions
    :return: Embedded input.
    """
    # TODO: Implement Function
    # https://www.tensorflow.org/api_docs/python/tf/contrib/layers/embed_sequence
    return tf.contrib.layers.embed_sequence(input_data, vocab_size, embed_dim)


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_embed(get_embed)
Tests Passed

Build RNN

You created a RNN Cell in the get_init_cell() function. Time to use the cell to create a RNN.

Return the outputs and final_state state in the following tuple (Outputs, FinalState)

def build_rnn(cell, inputs):
    """
    Create a RNN using a RNN Cell
    :param cell: RNN Cell
    :param inputs: Input text data
    :return: Tuple (Outputs, Final State)
    """
    # TODO: Implement Function
    (Outputs, FinalState) = tf.nn.dynamic_rnn(cell, inputs, dtype=tf.float32)
    FinalState = tf.identity(FinalState, name="final_state")    
    return (Outputs, FinalState)


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_build_rnn(build_rnn)
Tests Passed

Build the Neural Network

Apply the functions you implemented above to:

  • Apply embedding to input_data using your get_embed(input_data, vocab_size, embed_dim) function.
  • Build RNN using cell and your build_rnn(cell, inputs) function.
  • Apply a fully connected layer with a linear activation and vocab_size as the number of outputs.

Return the logits and final state in the following tuple (Logits, FinalState)

def build_nn(cell, rnn_size, input_data, vocab_size, embed_dim):
    """
    Build part of the neural network
    :param cell: RNN cell
    :param rnn_size: Size of rnns
    :param input_data: Input data
    :param vocab_size: Vocabulary size
    :param embed_dim: Number of embedding dimensions
    :return: Tuple (Logits, FinalState)
    """
    # TODO: Implement Function
    embedded_input = get_embed(input_data, vocab_size, embed_dim)
    rnn_output, FinalState = build_rnn(cell, embedded_input)
    Logits = tf.contrib.layers.fully_connected(rnn_output, vocab_size, activation_fn=None)
    return (Logits, FinalState)


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_build_nn(build_nn)
Tests Passed

Batches

Implement get_batches to create batches of input and targets using int_text. The batches should be a Numpy array with the shape (number of batches, 2, batch size, sequence length). Each batch contains two elements:

  • The first element is a single batch of input with the shape [batch size, sequence length]
  • The second element is a single batch of targets with the shape [batch size, sequence length]

If you can’t fill the last batch with enough data, drop the last batch.

For example, get_batches([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], 3, 2) would return a Numpy array of the following:

[
  # First Batch
  [
    # Batch of Input
    [[ 1  2], [ 7  8], [13 14]]
    # Batch of targets
    [[ 2  3], [ 8  9], [14 15]]
  ]

  # Second Batch
  [
    # Batch of Input
    [[ 3  4], [ 9 10], [15 16]]
    # Batch of targets
    [[ 4  5], [10 11], [16 17]]
  ]

  # Third Batch
  [
    # Batch of Input
    [[ 5  6], [11 12], [17 18]]
    # Batch of targets
    [[ 6  7], [12 13], [18  1]]
  ]
]

Notice that the last target value in the last batch is the first input value of the first batch. In this case, 1. This is a common technique used when creating sequence batches, although it is rather unintuitive.

def get_batches(int_text, batch_size, seq_length):
    """
    Return batches of input and target
    :param int_text: Text with the words replaced by their ids
    :param batch_size: The size of batch
    :param seq_length: The length of sequence
    :return: Batches as a Numpy array
    """
    # TODO: Implement Function
    inputs_per_batch = batch_size * seq_length
    num_batches = len(int_text)//(inputs_per_batch)
    int_text = int_text[:num_batches*inputs_per_batch] # drop unused  
    int_text.append(int_text[0]) # to use first input value of first batch as last target value of the last batch

    # allocate memory with shape of batches
    batches = np.zeros([num_batches, 2, batch_size, seq_length], dtype=np.int32)

    # Add seq_length elements at a time to input and targets appropriately
    for i in range(0, len(int_text), seq_length):
        batch_no = (i // seq_length) % num_batches
        index_in_batch = i // (seq_length * num_batches)
        
        if (index_in_batch == batch_size):            
            break
        
        # input 
        batches[batch_no, 0, index_in_batch] = int_text[i : i+seq_length]

        # targets
        batches[batch_no, 1, index_in_batch] = int_text[i+1 : i+seq_length+1] # element next to input element
    return batches


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_batches(get_batches)
Tests Passed

Neural Network Training

Hyperparameters

Tune the following parameters:

  • Set num_epochs to the number of epochs.
  • Set batch_size to the batch size.
  • Set rnn_size to the size of the RNNs.
  • Set embed_dim to the size of the embedding.
  • Set seq_length to the length of sequence.
  • Set learning_rate to the learning rate.
  • Set show_every_n_batches to the number of batches the neural network should print progress.
# Number of Epochs
num_epochs = 300 # loss was still going down after 128 epochs but started increasing after 400
# Batch Size
batch_size = 128
# RNN Size
rnn_size = 256
# Embedding Dimension Size
embed_dim = 256
# Sequence Length
seq_length = 32
# Learning Rate
learning_rate = 0.01
# Show stats for every n number of batches
show_every_n_batches = 16

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
save_dir = './save'

Build the Graph

Build the graph using the neural network you implemented.

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
from tensorflow.contrib import seq2seq

train_graph = tf.Graph()
with train_graph.as_default():
    vocab_size = len(int_to_vocab)
    input_text, targets, lr = get_inputs()
    input_data_shape = tf.shape(input_text)
    cell, initial_state = get_init_cell(input_data_shape[0], rnn_size)
    logits, final_state = build_nn(cell, rnn_size, input_text, vocab_size, embed_dim)

    # Probabilities for generating words
    probs = tf.nn.softmax(logits, name='probs')

    # Loss function
    cost = seq2seq.sequence_loss(
        logits,
        targets,
        tf.ones([input_data_shape[0], input_data_shape[1]]))

    # Optimizer
    optimizer = tf.train.AdamOptimizer(lr)

    # Gradient Clipping
    gradients = optimizer.compute_gradients(cost)
    capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients if grad is not None]
    train_op = optimizer.apply_gradients(capped_gradients)

Train

Train the neural network on the preprocessed data. If you have a hard time getting a good loss, check the forums to see if anyone is having the same problem.

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
batches = get_batches(int_text, batch_size, seq_length)

with tf.Session(graph=train_graph) as sess:
    sess.run(tf.global_variables_initializer())

    for epoch_i in range(num_epochs):
        state = sess.run(initial_state, {input_text: batches[0][0]})

        for batch_i, (x, y) in enumerate(batches):
            feed = {
                input_text: x,
                targets: y,
                initial_state: state,
                lr: learning_rate}
            train_loss, state, _ = sess.run([cost, final_state, train_op], feed)

            # Show every <show_every_n_batches> batches
            if (epoch_i * len(batches) + batch_i) % show_every_n_batches == 0:
                print('Epoch {:>3} Batch {:>4}/{}   train_loss = {:.3f}'.format(
                    epoch_i,
                    batch_i,
                    len(batches),
                    train_loss))

    # Save Model
    saver = tf.train.Saver()
    saver.save(sess, save_dir)
    print('Model Trained and Saved')
Epoch   0 Batch    0/16   train_loss = 8.822
Epoch   1 Batch    0/16   train_loss = 6.426
Epoch   2 Batch    0/16   train_loss = 6.098
Epoch   3 Batch    0/16   train_loss = 6.074
Epoch   4 Batch    0/16   train_loss = 6.044
Epoch   5 Batch    0/16   train_loss = 6.026
Epoch   6 Batch    0/16   train_loss = 6.027
Epoch   7 Batch    0/16   train_loss = 6.010
Epoch   8 Batch    0/16   train_loss = 6.010
Epoch   9 Batch    0/16   train_loss = 5.997
Epoch  10 Batch    0/16   train_loss = 6.012
Epoch  11 Batch    0/16   train_loss = 5.989
Epoch  12 Batch    0/16   train_loss = 5.978
Epoch  13 Batch    0/16   train_loss = 5.980
Epoch  14 Batch    0/16   train_loss = 5.938
Epoch  15 Batch    0/16   train_loss = 5.910
Epoch  16 Batch    0/16   train_loss = 5.890
Epoch  17 Batch    0/16   train_loss = 5.867
Epoch  18 Batch    0/16   train_loss = 5.854
Epoch  19 Batch    0/16   train_loss = 5.784
Epoch  20 Batch    0/16   train_loss = 5.343
Epoch  21 Batch    0/16   train_loss = 5.027
Epoch  22 Batch    0/16   train_loss = 4.800
Epoch  23 Batch    0/16   train_loss = 4.605
Epoch  24 Batch    0/16   train_loss = 4.449
Epoch  25 Batch    0/16   train_loss = 4.327
Epoch  26 Batch    0/16   train_loss = 4.179
Epoch  27 Batch    0/16   train_loss = 4.064
Epoch  28 Batch    0/16   train_loss = 3.952
Epoch  29 Batch    0/16   train_loss = 3.842
Epoch  30 Batch    0/16   train_loss = 3.767
Epoch  31 Batch    0/16   train_loss = 3.766
Epoch  32 Batch    0/16   train_loss = 3.623
Epoch  33 Batch    0/16   train_loss = 3.474
Epoch  34 Batch    0/16   train_loss = 3.370
Epoch  35 Batch    0/16   train_loss = 3.328
Epoch  36 Batch    0/16   train_loss = 3.213
Epoch  37 Batch    0/16   train_loss = 3.185
Epoch  38 Batch    0/16   train_loss = 3.078
Epoch  39 Batch    0/16   train_loss = 2.988
Epoch  40 Batch    0/16   train_loss = 2.943
Epoch  41 Batch    0/16   train_loss = 2.858
Epoch  42 Batch    0/16   train_loss = 2.773
Epoch  43 Batch    0/16   train_loss = 2.738
Epoch  44 Batch    0/16   train_loss = 2.723
Epoch  45 Batch    0/16   train_loss = 2.734
Epoch  46 Batch    0/16   train_loss = 2.683
Epoch  47 Batch    0/16   train_loss = 2.592
Epoch  48 Batch    0/16   train_loss = 2.543
Epoch  49 Batch    0/16   train_loss = 2.474
Epoch  50 Batch    0/16   train_loss = 2.364
Epoch  51 Batch    0/16   train_loss = 2.271
Epoch  52 Batch    0/16   train_loss = 2.196
Epoch  53 Batch    0/16   train_loss = 2.148
Epoch  54 Batch    0/16   train_loss = 2.119
Epoch  55 Batch    0/16   train_loss = 2.098
Epoch  56 Batch    0/16   train_loss = 2.115
Epoch  57 Batch    0/16   train_loss = 2.040
Epoch  58 Batch    0/16   train_loss = 1.962
Epoch  59 Batch    0/16   train_loss = 1.930
Epoch  60 Batch    0/16   train_loss = 1.847
Epoch  61 Batch    0/16   train_loss = 1.756
Epoch  62 Batch    0/16   train_loss = 1.742
Epoch  63 Batch    0/16   train_loss = 1.694
Epoch  64 Batch    0/16   train_loss = 1.637
Epoch  65 Batch    0/16   train_loss = 1.597
Epoch  66 Batch    0/16   train_loss = 1.513
Epoch  67 Batch    0/16   train_loss = 1.441
Epoch  68 Batch    0/16   train_loss = 1.363
Epoch  69 Batch    0/16   train_loss = 1.348
Epoch  70 Batch    0/16   train_loss = 1.293
Epoch  71 Batch    0/16   train_loss = 1.288
Epoch  72 Batch    0/16   train_loss = 1.312
Epoch  73 Batch    0/16   train_loss = 1.286
Epoch  74 Batch    0/16   train_loss = 1.234
Epoch  75 Batch    0/16   train_loss = 1.210
Epoch  76 Batch    0/16   train_loss = 1.223
Epoch  77 Batch    0/16   train_loss = 1.164
Epoch  78 Batch    0/16   train_loss = 1.123
Epoch  79 Batch    0/16   train_loss = 1.069
Epoch  80 Batch    0/16   train_loss = 1.016
Epoch  81 Batch    0/16   train_loss = 0.991
Epoch  82 Batch    0/16   train_loss = 0.946
Epoch  83 Batch    0/16   train_loss = 0.899
Epoch  84 Batch    0/16   train_loss = 0.873
Epoch  85 Batch    0/16   train_loss = 0.875
Epoch  86 Batch    0/16   train_loss = 0.860
Epoch  87 Batch    0/16   train_loss = 0.804
Epoch  88 Batch    0/16   train_loss = 0.812
Epoch  89 Batch    0/16   train_loss = 0.817
Epoch  90 Batch    0/16   train_loss = 0.784
Epoch  91 Batch    0/16   train_loss = 0.741
Epoch  92 Batch    0/16   train_loss = 0.706
Epoch  93 Batch    0/16   train_loss = 0.698
Epoch  94 Batch    0/16   train_loss = 0.693
Epoch  95 Batch    0/16   train_loss = 0.667
Epoch  96 Batch    0/16   train_loss = 0.671
Epoch  97 Batch    0/16   train_loss = 0.647
Epoch  98 Batch    0/16   train_loss = 0.694
Epoch  99 Batch    0/16   train_loss = 0.708
Epoch 100 Batch    0/16   train_loss = 0.721
Epoch 101 Batch    0/16   train_loss = 0.756
Epoch 102 Batch    0/16   train_loss = 0.754
Epoch 103 Batch    0/16   train_loss = 0.739
Epoch 104 Batch    0/16   train_loss = 0.806
Epoch 105 Batch    0/16   train_loss = 0.909
Epoch 106 Batch    0/16   train_loss = 0.967
Epoch 107 Batch    0/16   train_loss = 0.987
Epoch 108 Batch    0/16   train_loss = 0.941
Epoch 109 Batch    0/16   train_loss = 0.856
Epoch 110 Batch    0/16   train_loss = 0.838
Epoch 111 Batch    0/16   train_loss = 0.815
Epoch 112 Batch    0/16   train_loss = 0.769
Epoch 113 Batch    0/16   train_loss = 0.719
Epoch 114 Batch    0/16   train_loss = 0.671
Epoch 115 Batch    0/16   train_loss = 0.653
Epoch 116 Batch    0/16   train_loss = 0.630
Epoch 117 Batch    0/16   train_loss = 0.637
Epoch 118 Batch    0/16   train_loss = 0.612
Epoch 119 Batch    0/16   train_loss = 0.546
Epoch 120 Batch    0/16   train_loss = 0.482
Epoch 121 Batch    0/16   train_loss = 0.478
Epoch 122 Batch    0/16   train_loss = 0.449
Epoch 123 Batch    0/16   train_loss = 0.442
Epoch 124 Batch    0/16   train_loss = 0.413
Epoch 125 Batch    0/16   train_loss = 0.376
Epoch 126 Batch    0/16   train_loss = 0.351
Epoch 127 Batch    0/16   train_loss = 0.344
Epoch 128 Batch    0/16   train_loss = 0.331
Epoch 129 Batch    0/16   train_loss = 0.298
Epoch 130 Batch    0/16   train_loss = 0.297
Epoch 131 Batch    0/16   train_loss = 0.263
Epoch 132 Batch    0/16   train_loss = 0.223
Epoch 133 Batch    0/16   train_loss = 0.227
Epoch 134 Batch    0/16   train_loss = 0.221
Epoch 135 Batch    0/16   train_loss = 0.205
Epoch 136 Batch    0/16   train_loss = 0.208
Epoch 137 Batch    0/16   train_loss = 0.190
Epoch 138 Batch    0/16   train_loss = 0.172
Epoch 139 Batch    0/16   train_loss = 0.171
Epoch 140 Batch    0/16   train_loss = 0.195
Epoch 141 Batch    0/16   train_loss = 0.172
Epoch 142 Batch    0/16   train_loss = 0.157
Epoch 143 Batch    0/16   train_loss = 0.152
Epoch 144 Batch    0/16   train_loss = 0.140
Epoch 145 Batch    0/16   train_loss = 0.133
Epoch 146 Batch    0/16   train_loss = 0.128
Epoch 147 Batch    0/16   train_loss = 0.124
Epoch 148 Batch    0/16   train_loss = 0.121
Epoch 149 Batch    0/16   train_loss = 0.117
Epoch 150 Batch    0/16   train_loss = 0.116
Epoch 151 Batch    0/16   train_loss = 0.114
Epoch 152 Batch    0/16   train_loss = 0.112
Epoch 153 Batch    0/16   train_loss = 0.111
Epoch 154 Batch    0/16   train_loss = 0.109
Epoch 155 Batch    0/16   train_loss = 0.108
Epoch 156 Batch    0/16   train_loss = 0.107
Epoch 157 Batch    0/16   train_loss = 0.106
Epoch 158 Batch    0/16   train_loss = 0.104
Epoch 159 Batch    0/16   train_loss = 0.104
Epoch 160 Batch    0/16   train_loss = 0.103
Epoch 161 Batch    0/16   train_loss = 0.102
Epoch 162 Batch    0/16   train_loss = 0.101
Epoch 163 Batch    0/16   train_loss = 0.101
Epoch 164 Batch    0/16   train_loss = 0.100
Epoch 165 Batch    0/16   train_loss = 0.099
Epoch 166 Batch    0/16   train_loss = 0.099
Epoch 167 Batch    0/16   train_loss = 0.098
Epoch 168 Batch    0/16   train_loss = 0.097
Epoch 169 Batch    0/16   train_loss = 0.097
Epoch 170 Batch    0/16   train_loss = 0.096
Epoch 171 Batch    0/16   train_loss = 0.096
Epoch 172 Batch    0/16   train_loss = 0.095
Epoch 173 Batch    0/16   train_loss = 0.095
Epoch 174 Batch    0/16   train_loss = 0.094
Epoch 175 Batch    0/16   train_loss = 0.094
Epoch 176 Batch    0/16   train_loss = 0.094
Epoch 177 Batch    0/16   train_loss = 0.093
Epoch 178 Batch    0/16   train_loss = 0.093
Epoch 179 Batch    0/16   train_loss = 0.092
Epoch 180 Batch    0/16   train_loss = 0.092
Epoch 181 Batch    0/16   train_loss = 0.092
Epoch 182 Batch    0/16   train_loss = 0.091
Epoch 183 Batch    0/16   train_loss = 0.091
Epoch 184 Batch    0/16   train_loss = 0.091
Epoch 185 Batch    0/16   train_loss = 0.091
Epoch 186 Batch    0/16   train_loss = 0.090
Epoch 187 Batch    0/16   train_loss = 0.090
Epoch 188 Batch    0/16   train_loss = 0.090
Epoch 189 Batch    0/16   train_loss = 0.090
Epoch 190 Batch    0/16   train_loss = 0.089
Epoch 191 Batch    0/16   train_loss = 0.089
Epoch 192 Batch    0/16   train_loss = 0.089
Epoch 193 Batch    0/16   train_loss = 0.089
Epoch 194 Batch    0/16   train_loss = 0.088
Epoch 195 Batch    0/16   train_loss = 0.088
Epoch 196 Batch    0/16   train_loss = 0.088
Epoch 197 Batch    0/16   train_loss = 0.088
Epoch 198 Batch    0/16   train_loss = 0.088
Epoch 199 Batch    0/16   train_loss = 0.087
Epoch 200 Batch    0/16   train_loss = 0.087
Epoch 201 Batch    0/16   train_loss = 0.087
Epoch 202 Batch    0/16   train_loss = 0.087
Epoch 203 Batch    0/16   train_loss = 0.087
Epoch 204 Batch    0/16   train_loss = 0.087
Epoch 205 Batch    0/16   train_loss = 0.086
Epoch 206 Batch    0/16   train_loss = 0.086
Epoch 207 Batch    0/16   train_loss = 0.086
Epoch 208 Batch    0/16   train_loss = 0.086
Epoch 209 Batch    0/16   train_loss = 0.086
Epoch 210 Batch    0/16   train_loss = 0.086
Epoch 211 Batch    0/16   train_loss = 0.086
Epoch 212 Batch    0/16   train_loss = 0.086
Epoch 213 Batch    0/16   train_loss = 0.085
Epoch 214 Batch    0/16   train_loss = 0.085
Epoch 215 Batch    0/16   train_loss = 0.085
Epoch 216 Batch    0/16   train_loss = 0.085
Epoch 217 Batch    0/16   train_loss = 0.085
Epoch 218 Batch    0/16   train_loss = 0.085
Epoch 219 Batch    0/16   train_loss = 0.085
Epoch 220 Batch    0/16   train_loss = 0.085
Epoch 221 Batch    0/16   train_loss = 0.085
Epoch 222 Batch    0/16   train_loss = 0.085
Epoch 223 Batch    0/16   train_loss = 0.084
Epoch 224 Batch    0/16   train_loss = 0.084
Epoch 225 Batch    0/16   train_loss = 0.084
Epoch 226 Batch    0/16   train_loss = 0.084
Epoch 227 Batch    0/16   train_loss = 0.084
Epoch 228 Batch    0/16   train_loss = 0.084
Epoch 229 Batch    0/16   train_loss = 0.085
Epoch 230 Batch    0/16   train_loss = 0.090
Epoch 231 Batch    0/16   train_loss = 0.100
Epoch 232 Batch    0/16   train_loss = 0.172
Epoch 233 Batch    0/16   train_loss = 0.723
Epoch 234 Batch    0/16   train_loss = 2.116
Epoch 235 Batch    0/16   train_loss = 2.848
Epoch 236 Batch    0/16   train_loss = 2.818
Epoch 237 Batch    0/16   train_loss = 2.644
Epoch 238 Batch    0/16   train_loss = 2.422
Epoch 239 Batch    0/16   train_loss = 2.131
Epoch 240 Batch    0/16   train_loss = 1.869
Epoch 241 Batch    0/16   train_loss = 1.657
Epoch 242 Batch    0/16   train_loss = 1.537
Epoch 243 Batch    0/16   train_loss = 1.358
Epoch 244 Batch    0/16   train_loss = 1.259
Epoch 245 Batch    0/16   train_loss = 1.167
Epoch 246 Batch    0/16   train_loss = 1.078
Epoch 247 Batch    0/16   train_loss = 0.996
Epoch 248 Batch    0/16   train_loss = 0.936
Epoch 249 Batch    0/16   train_loss = 0.884
Epoch 250 Batch    0/16   train_loss = 0.866
Epoch 251 Batch    0/16   train_loss = 0.817
Epoch 252 Batch    0/16   train_loss = 0.760
Epoch 253 Batch    0/16   train_loss = 0.720
Epoch 254 Batch    0/16   train_loss = 0.682
Epoch 255 Batch    0/16   train_loss = 0.643
Epoch 256 Batch    0/16   train_loss = 0.626
Epoch 257 Batch    0/16   train_loss = 0.596
Epoch 258 Batch    0/16   train_loss = 0.571
Epoch 259 Batch    0/16   train_loss = 0.537
Epoch 260 Batch    0/16   train_loss = 0.534
Epoch 261 Batch    0/16   train_loss = 0.517
Epoch 262 Batch    0/16   train_loss = 0.501
Epoch 263 Batch    0/16   train_loss = 0.478
Epoch 264 Batch    0/16   train_loss = 0.487
Epoch 265 Batch    0/16   train_loss = 0.510
Epoch 266 Batch    0/16   train_loss = 0.488
Epoch 267 Batch    0/16   train_loss = 0.463
Epoch 268 Batch    0/16   train_loss = 0.494
Epoch 269 Batch    0/16   train_loss = 0.479
Epoch 270 Batch    0/16   train_loss = 0.485
Epoch 271 Batch    0/16   train_loss = 0.474
Epoch 272 Batch    0/16   train_loss = 0.469
Epoch 273 Batch    0/16   train_loss = 0.463
Epoch 274 Batch    0/16   train_loss = 0.455
Epoch 275 Batch    0/16   train_loss = 0.429
Epoch 276 Batch    0/16   train_loss = 0.435
Epoch 277 Batch    0/16   train_loss = 0.471
Epoch 278 Batch    0/16   train_loss = 0.460
Epoch 279 Batch    0/16   train_loss = 0.483
Epoch 280 Batch    0/16   train_loss = 0.448
Epoch 281 Batch    0/16   train_loss = 0.395
Epoch 282 Batch    0/16   train_loss = 0.375
Epoch 283 Batch    0/16   train_loss = 0.377
Epoch 284 Batch    0/16   train_loss = 0.367
Epoch 285 Batch    0/16   train_loss = 0.350
Epoch 286 Batch    0/16   train_loss = 0.317
Epoch 287 Batch    0/16   train_loss = 0.296
Epoch 288 Batch    0/16   train_loss = 0.269
Epoch 289 Batch    0/16   train_loss = 0.260
Epoch 290 Batch    0/16   train_loss = 0.249
Epoch 291 Batch    0/16   train_loss = 0.217
Epoch 292 Batch    0/16   train_loss = 0.224
Epoch 293 Batch    0/16   train_loss = 0.201
Epoch 294 Batch    0/16   train_loss = 0.198
Epoch 295 Batch    0/16   train_loss = 0.222
Epoch 296 Batch    0/16   train_loss = 0.240
Epoch 297 Batch    0/16   train_loss = 0.265
Epoch 298 Batch    0/16   train_loss = 0.266
Epoch 299 Batch    0/16   train_loss = 0.249
Model Trained and Saved

Save Parameters

Save seq_length and save_dir for generating a new TV script.

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Save parameters for checkpoint
helper.save_params((seq_length, save_dir))

Checkpoint

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import tensorflow as tf
import numpy as np
import helper
import problem_unittests as tests

_, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()
seq_length, load_dir = helper.load_params()

Implement Generate Functions

Get Tensors

Get tensors from loaded_graph using the function get_tensor_by_name(). Get the tensors using the following names:

  • “input:0”
  • “initial_state:0”
  • “final_state:0”
  • “probs:0”

Return the tensors in the following tuple (InputTensor, InitialStateTensor, FinalStateTensor, ProbsTensor)

def get_tensors(loaded_graph):
    """
    Get input, initial state, final state, and probabilities tensor from <loaded_graph>
    :param loaded_graph: TensorFlow graph loaded from file
    :return: Tuple (InputTensor, InitialStateTensor, FinalStateTensor, ProbsTensor)
    """
    # TODO: Implement Function
    InputTensor = loaded_graph.get_tensor_by_name("input:0")
    InitialStateTensor = loaded_graph.get_tensor_by_name("initial_state:0")
    FinalStateTensor = loaded_graph.get_tensor_by_name("final_state:0")
    ProbsTensor = loaded_graph.get_tensor_by_name("probs:0")
    return (InputTensor, InitialStateTensor, FinalStateTensor, ProbsTensor)


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_tensors(get_tensors)
Tests Passed

Choose Word

Implement the pick_word() function to select the next word using probabilities.

def pick_word(probabilities, int_to_vocab):
    """
    Pick the next word in the generated text
    :param probabilities: Probabilites of the next word
    :param int_to_vocab: Dictionary of word ids as the keys and words as the values
    :return: String of the predicted word
    """
    # TODO: Implement Function
    # Return word with highest probability
    return int_to_vocab[np.argmax(probabilities)]


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_pick_word(pick_word)
Tests Passed

Generate TV Script

This will generate the TV script for you. Set gen_length to the length of TV script you want to generate.

gen_length = 200
# homer_simpson, moe_szyslak, or Barney_Gumble
prime_word = 'moe_szyslak'

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
    # Load saved model
    loader = tf.train.import_meta_graph(load_dir + '.meta')
    loader.restore(sess, load_dir)

    # Get Tensors from loaded model
    input_text, initial_state, final_state, probs = get_tensors(loaded_graph)

    # Sentences generation setup
    gen_sentences = [prime_word + ':']
    prev_state = sess.run(initial_state, {input_text: np.array([[1]])})

    # Generate sentences
    for n in range(gen_length):
        # Dynamic Input
        dyn_input = [[vocab_to_int[word] for word in gen_sentences[-seq_length:]]]
        dyn_seq_length = len(dyn_input[0])

        # Get Prediction
        probabilities, prev_state = sess.run(
            [probs, final_state],
            {input_text: dyn_input, initial_state: prev_state})
        
        pred_word = pick_word(probabilities[0][dyn_seq_length-1], int_to_vocab)

        gen_sentences.append(pred_word)
    
    # Remove tokens
    tv_script = ' '.join(gen_sentences)
    for key, token in token_dict.items():
        ending = ' ' if key in ['\n', '(', '"'] else ''
        tv_script = tv_script.replace(' ' + token.lower(), key)
    tv_script = tv_script.replace('\n ', '\n')
    tv_script = tv_script.replace('( ', '(')
        
    print(tv_script)
INFO:tensorflow:Restoring parameters from ./save
moe_szyslak:(into phone) moe's tavern. where the elite meet to drink to see my camera.
moe_szyslak: wow, this is all that?
moe_szyslak: hey, there's it all the money, homer?
homer_simpson:(big smile) hey, homer, i've got some of us.
homer_simpson:(pats stomach) i'll cover mr. burns, i'm going to see you to go to the half way to turn out his truth.


moe_szyslak:(chuckles) hey, you got it, homer?
moe_szyslak:(shocked) don't have to bet twenty dollars on a or here.
moe_szyslak:(slightly confused chuckle) i've gotta go to go to myself.(à la improv comic) okay, which one back to go to my snake handler.
moe_szyslak: marge, maggie, moe! that plank's only for duff porn from no return to save that are a little girl, moe?
lucius:(confused) there where you can go of this, it, sir?
homer_simpson:(drunk) oh, that's

The TV Script is Nonsensical

While most of the generated TV script above doesn’t make any sense, this is expected because we trained on less than a megabyte of text. Using a smaller vocabulary or more data should produce better results.

Leave a Comment