What is shallow architecture in machine learning?

by mattosmat   Last Updated September 11, 2019 17:19 PM - source

What is a precise definition of shallow architecture in machine learning?



Answers 4


It means an artificial neural networks with few hidden layers. To be contrasted with deep neural networks. The precise boundary is debatable.

Emre
Emre
June 10, 2014 00:23 AM

Shallow and deep refer to two different (but related) ways to go about modelling a problem. Shallow architectures rely on the paper by Cybenko: Approximation by Superposition of a Sigmoidal Function, where the shows that arbitrary decision regions can be arbitrarily well approximated by MLPs.

Basically it means that you would just add more neurons to your layer, thus making your network wider, as the complexity of your problem at hand increases (in very rough terms). My point is thus, that there is no precise definition to that term. Is more about how you approach the complexity of a problem.

jpmuc
jpmuc
June 14, 2014 10:36 AM

I believe usually "shallow" means only one hidden layer. For example:

1. MLP with one hidden layer: data --> hidden --> softmax (class label)
2. SVM: data --> feature (can be considered as hidden) --> class label

Anything with more than 2 hidden layers (inclusive) can be called deep.

gwding
gwding
July 20, 2014 00:49 AM

according to one book, shallow-structured architectures, typically contain at most one or two layers of non-linear feature transformations. Examples of shallow architectures are Gaussian mixture models (GMMs), linear or non-linear dynamical systems, conditional random fields (CRFs), maximum entropy (MaxEnt) models, support vector machines (SVMs), logistic regression, kernel regression, multilayer perceptrons (MLPs) with a single hidden layer including extreme learning machines (ELMs). For instance, SVMs use a shallow linear pattern separation model with one or zero feature transformation layer when the kernel trick is used or otherwise.

AJAY
AJAY
September 11, 2019 16:23 PM

Related Questions


Can reinforcement learning be "stateless"?

Updated July 17, 2017 06:19 AM

What's the name of this recurrent neural network?

Updated March 03, 2017 23:19 PM


Clarification of Machine Learning Terminology

Updated April 15, 2018 17:19 PM