Representation

# Representation
### <a href='https://cdsbasel.github.io/neuralnetworks/'> Introduction to Neural Networks </a> <a href='https://cdsbasel.github.io/neuralnetworks/menu/materials.html'> </a>  <a href='https://cdsbasel.github.io/dataanalytics/'> </a>  <a href='mailto:dirk.wulff@unibas.ch'> 
### May 2020

---

<div class="my-footer">
 
 
 <img src="https://raw.githubusercontent.com/therbootcamp/therbootcamp.github.io/master/_sessions/_image/by-sa.png" height=14 style="vertical-align: middle"/>
 
 <a href="https://cdsbasel.github.io/neuralnetworks/">
 
 
 cdsbasel.github.io/neuralnetworks/
 
 
 </a>
 <a href="https://cdsbasel.github.io/neuralnetworks/">
 
 Introduction to Neural Networks | May 2020
 
 </a>
 
 </div>

---

# Why does deep learning work?

<ul>
 <li class="m1">Deep feedforward networks have <high>"many" hidden layers</high>.</li>
 <li class="m2">The hidden layers learn <high>useful representations of the inputs</high>.</li>
 <li class="m3">The deeper the layer, the <high>more abstract the representation</high>.</li>
 <li class="m4">Useful does <high>not mean intuitive</high>.</li>
</ul>

]

]

---

# MNIST representations

<ul>
 <li class="m1">The <high>activation patterns in the hidden layers</high> reveal how inputs are represented.</li> 
 <li class="m2">Activation patterns can easily be computed using basic <high>matrix algebra</high> (once weights have been determined).</li>
</ul>

]

]

---

# MNIST representations

]

]

---

# MNIST representations

]

]

---

# Convolutional neural networks

<ul>
 <li class="m1">Convolutional neural networks (CNN) <high>learn input primitives</high>, e.g., lines of different orientations.</li> 
 <li class="m2">Later dense layers <high>recombine primitives</high> to form predictive higher-order characteristics, e.g., shapes.</li> 
 <li class="m3">CNNs are the <high>gold-standard</high> for image processing and <high>object recognition</high>.</li>
</ul>

]

]

---

---

# Pixel embeddings

<ul>
 <li class="m1">Activation patterns cannot be understood on their own, still they can be used to learn much about inputs by treating them as <high>encodings or embeddings of the inputs</high>.</li> 
 <li class="m2">Embeddings are particcularly useful for studying the <high>patterns of relationships</high> between inputs.</li>
</ul>

]

]

---

# Pixel embeddings

]

---

# Digit embeddings

]

]

---

# Digit embeddings

]

]

---

# Digit embeddings

]

]

---

# Word embeddings

<ul>
 <li class="m1">Embeddings are especially <high>useful and popular for words</high>.</li> 
 <li class="m2">Word embeddings approximate the (distributed) <high>meaning of a word</high>.</li>
 <li class="m3">Word embeddings are used for research as much as for <high>many practical applications</high> (e.g., document search).</li>
</ul>

]

]

---

# Word embeddings

]

]

---

# Word embeddings

]

]

---

# Word embeddings

]

]

---

# Word embeddings

]

]

---

# Auto-encoder

<ul>
 <li class="m1">Auto-encoders are, in a way, <high>pure embedding learners</high>.</li> 
 <li class="m2">In auto-encoders, <high>the input is the output</high>.</li> 
</ul>

]

]

---

# Auto-encoder

</ul>

]

]

---

<h1><a href="https://cdsbasel.github.io/neuralnetworks/sessions/Represenetation/Represenetation_practical.html">Practical</a></h1>