Jordan & Bishop "Neural Networks"

Michael I. Jordan & Christopher M. Bishop, "Neural Networks", In Tucker, A. B. (Ed.) CRC Handbook of Computer Science, Boca Raton, FL: CRC Press, 1997.

download: http://www.cs.berkeley.edu/~jordan/papers/crc.ps

http://en.wikipedia.org/wiki/Neural_network

http://www.learnartificialneuralnetworks.com/

1. Introduction

Neural network methods have had their greatest impact in problems where statistical issues dominate and where data are easily obtained.

"conjunction of graphical algorithms and probability theory":

A neural network is first and foremost a graph with patterns represented in terms of numerical values attached to the nodes of the graph and transformations between patterns achieved via simple message-passing algorithms. Many neural network architectures, however, are also statistical processors, characterized by making particular probabilistic assumptions about data.

Based on a source of training data, the aim is to produce a statistical model of the process from which the data are generated so as to allow the best predictions to be made for new data.

statistical modeling - density estimation (unsupervised learning), classification & regression

density estimation ("unsupervised learning")

: to model the unconditional distribution of data described by some vector

- to train samples and a network model to build a representation of the probability density 

- to label regions for a new input vector

 

classification & regression ("supervised learning")

: to distinguish between input variables and target variables

- to assign each input vector to one of classes and target variables to class labels 

-> estimation of conditional densities from the joint input-target space

2. Representation

2.1 Density estimation

To form an explicit model of the input density

Gaussian mixture distribution

http://en.wikipedia.org/wiki/Principal_component_analysis

http://en.wikipedia.org/wiki/Canonical_correlation

http://en.wikipedia.org/wiki/Kernel_density_estimation

http://en.wikipedia.org/wiki/Factor_analysis

ref. An Introduction to Multivariate Statistical Analysis, 3rd Edition, T. W. Anderson, Wiley, 2003

2.2 Linear regression and linear discriminants