Naive Bayes

TLDR: predict the likelihood of the label, given features

\[\begin{split}& \arg \max_y P(y | \mathbf{x}) \\ & = \arg \max_y P(\mathbf{x} | y) \frac{P(y)}{P(\mathbf{x})} \\ & = \arg \max_y P(\mathbf{x} | y) P(y)\end{split}\]

Naive independence assumption: the attributes are conditionally independent given y, i.e.

\[P(\mathbf{x} | y) = \prod_j P(x_j | y)\]

So, we predict the label y that maximizes

\[P(y) \prod_j P(x_j | y)\]

This uses a generative model: pick y then generate x based on y

To implement naive bayes, we need to estimate:

  • \(P(y)\) distribution
  • for each class y, for each feature \(x_j\), need \(P(x_j | y)\) distributions

all of these features are 1-dimensional - the combination of them is the model



  • conditional independence is optimistic
  • what if an attribute-value pair is not in the training set?
    • laplace smoothing / dummy data
  • continuous features: use gaussian or other density?
  • attributes for text classification?
    • bag of words model

NB for Text

  • let \(V\) be the vocabulary (all words/symbols in training docs)
  • for each class \(y\), let \(Docs_y\) by the concatenation of all docs labelled y
  • for each word \(w\) in \(V\), let \(\#w(Docs_y)\) be the number of times \(w\) occurs in \(Docs_y\)
  • set \(P(w | y) = \frac{\#w(Docs_y) + 1}{|V| + \sum_w \#w(Docs_y)}\) (Laplacian smoothing)