Neural Embeddings

The Magic of Neural Embeddings

with TensorFlow 2

EuroSciPy 2019, Bilbao, Spain

Oliver Zeigermann / @DJCordhose

Slides: http://bit.ly/euroscipy-embeddings

Train embedding with TensorFlow


number_of_airlines = len(airlines) + 1
embedding_dim = 1 # up to us

model.add(Embedding(input_dim=number_of_airlines, 
                    output_dim=embedding_dim))


# embedding will be n-dimensional, but Dense can only handle flat input
model.add(Flatten())

# random additional layers to at least make this train
model.add(Dense(units=50, activation='relu'))
# ...


# two airports in a route
model.add(RepeatVector(2))
model.add(SimpleRNN(units=50, return_sequences=True))

# ca. 3500 airports in routes
output_dim = len(routes_tokenizer.word_index) + 1
model.add(Dense(units=output_dim, activation='softmax'))

### Functional API allows for all kinds of wiring

<pre><code contenteditable data-trim class="fragment line-numbers python">
# example for model changes

x = Dense(units=50, ...)(x)
# second dense layer
x = Dense(units=50, ...)(x)
# ...
# less units (25 instead of 50)
x = SimpleRNN(units=25, ...)(x)
</code></pre>

<pre><code contenteditable data-trim class="fragment line-numbers python">
# Head 1: main_output, Head 2: embedding
model = Model(inputs=main_input, outputs=[main_output, embedding])
</code></pre>

<pre><code contenteditable data-trim class="fragment line-numbers python">
model.compile(loss={ 'main_output': 'categorical_crossentropy', 
                 'embedding': 'mae' },
              loss_weights={'main_output': .1, 'embedding': 1.})
</code></pre>

<pre><code contenteditable data-trim class="fragment line-numbers python">
model.fit(x=X, y={'main_output': Y, 'embedding': original_embeddings})
</code></pre>

<small>

https://keras.io/getting-started/functional-api-guide/
<br>
https://colab.research.google.com/github/DJCordhose/ml-workshop/blob/master/notebooks/tf2/embeddings-retrain.ipynb

</small>