Neural Networks

# Neural Networks
### <a href='https://cdsbasel.github.io/neuralnetworks/'> Introduction to Neural Networks </a> <a href='https://cdsbasel.github.io/neuralnetworks/menu/schedule%20&%20materials.html'> </a>  <a href='https://cdsbasel.github.io/neuralnetworks/'> </a>  <a href='mailto:dirk.wulff@unibas.ch'> 
### May 2020

---

<div class="my-footer">
 
 
 <img src="https://raw.githubusercontent.com/therbootcamp/therbootcamp.github.io/master/_sessions/_image/by-sa.png" height=14 style="vertical-align: middle"/>
 
 <a href="https://cdsbasel.github.io/neuralnetworks/">
 
 
 cdsbasel.github.io/neuralnetworks/
 
 
 </a>
 <a href="https://cdsbasel.github.io/neuralnetworks/">
 
 Introduction to neural networks | May 2020
 
 </a>
 
 </div>

---

# What do you think?

---

# Back in the 50s and 60s

---

# Perceptron

<ul>
 <li class="m1">Developed by <high>Frank Rosenblatt</high> in 1958.</li>
 <li class="m2">The first <high>feedforward neural network</high>.</li>
 <li class="m3">Consists of...</li>
 <ul>
 <li>An <high>input layer</high> of nodes.</li>
 <li>A <high>bias</high>.</li>
 <li><high>Weights</high> projecting the inputs to a single output node.</li>
 <li>A single <high>output neuron with a <high>step activation</high> function.</li>
 <li>A <high>loss</high> function.</li>
 <li>A rule for <high>updating</high>.</li>
 </ul>
</ul>

]

]

---

# Perceptron

]

<img src="image/rosenblatt_sm.png" height=400px> 
Rosenblatt with Perceptron, adapted from <a href="https://www.nzz.ch/digital/ehre-fuer-die-deep-learning-mafia-ld.1472761">nzz.ch</a>

]

---

# Input

<ul>
 <li class="m1">Inputs take information from features, implying typically <high>n(features) = n(inputs)</high>.</li>
 <li class="m2">Typically a <high>bias</high> is added as one extra input.</li>
 <li class="m3">Higher dimensional inputs are typically <high>flattened to 1D</high>.</li>
 <li class="m4">Images are translated into <high>grey values</high> or rgb.</li>
</ul>

]

]

---

# Input

]

]

---

# Input

]

]

---

# Weights

<ul>
 <li class="m1">Weights <high>project the input</high> to the next/final layer.</li>
 <li class="m2">Typically initialized to small <high>random values</high>.</li>
</ul>

]

]

---

# Output

<ul>
 <li class="m1">A <high>linear combination</high> of the projecting inputs and their weights passed thorough a <high>step activation</high> function.</li>
</ul>

Output activation

`$$\begin{align*}  a^L &= \sigma(z^{L}) = \sigma(w^\intercal a^{L-1}) \\ & =\sigma(w_1 \cdot a^{L-1}_1 + \cdots + w_j \cdot a^{L-1}_j) \\ & =\sigma(w_1 \cdot x_1+ \cdots + w_j \cdot x_j) \end{align*}$$`

Step activation function

`$$\sigma(z) = {\begin{cases}{}
        0, & \text{for } z \leq 0 \\
        1, & \text{for } z > 0 \end{cases}}$$`

]

]

---

# Loss

<ul>
 <li class="m1">Loss summarizes the <high>mismatch between the output activation and the criterion</high>.</li> 
 <li class="m2">Loss is used as a <high>means to update the weights</high>.</li>
</ul>

`$$L_i(w_t) = y_i - \sigma(w_t^\intercal x_i)$$`

]

]

---

# Update

<ul>
 <li class="m1">A given weight is <high>updated proportional to the loss</high>, the input that is projected by the weight, and a learning rate <mono>r</mono>.</li> 
</ul>

Weight updating

`$$\large w^{t+1}_j = w^t_{j} + r \cdot L_i \cdot x_{ji}$$`

]

]

---

# Today...

<img src="image/neural_nets.png"> 
adapted from <a href="asimovinstitute.com">asimovinstitute.com</a>

---

# What changed?

]

]

---

# What changed?

Image recognition

<img src="image/image_recognition.png"> 
from <a href="https://towardsdatascience.com/the-5-deep-learning-breakthroughs-you-should-know-about-df27674ccdf2">towardsdatascience.com</a>, see <a href="https://arxiv.org/pdf/1703.06870.pdf">this</a>

]

Text translation

<img src="image/google_translation.png" height=320px> 
from <a href="https://ai.googleblog.com/2016/09/a-neural-network-for-machine.html">ai.googleblog.com</a>, see <a href="https://arxiv.org/abs/1609.08144">this</a>

]

---

# Why should psychologist care?

<ul>
 <li class="m1">Predictive tools</li> 
 <li class="m2">Representations</li> 
 <li class="m3">Cognitive models</li>
</ul>
]

<img src="image/brain.png" height=360px> 
from <a href="https://favpng.com/png_view/brain-human-brain-cognitive-science-neural-oscillation-clip-art-png/2UjbFG8S">favpng.com</a>

]

---

# Predictive tools

<ul>
 <li class="m1">What is the <high>upper bound</high> predictability of the output?</li> 
 <li class="m2">How do different <high>inputs contribute</high> to achieving predictive performance?</li>
</ul>

]

]

---

# Representations

<ul>
 <li class="m1">Treat neural network as an <high>as-if model</high> and study the inferred computations.</li> 
 <li class="m2">Use the neural network to extract <high>meaningful representations</high> from the environment.</li>
</ul>

]

]

---

# Cognitive models

<ul>
 <li class="m1">Use neural network as an <high>actual model of cognitive processing</high> in the brain.</li>
</ul>

]

]

---

# Cognitive models

<img src="image/neuron.png" height=400px> 
adapted from <a href="pinterest.de">pinterest.de</a>

]

<img src="image/cortex.png" height=500px> 
by Santiago Ramon y Cajal, from <a href="https://en.wikipedia.org/wiki/Cerebral_cortex">wikipedia.org</a>

]

---
class: middle, center

<h1><a href="https://cdsbasel.github.io/neuralnetworks/menu/schedule & materials.html">Materials</a></h1>