Shantanu's Blog

We use "One hot encoding" all the time. Right? It will convert the categorical values into new binary columns populated with sparse data. It means there will be a lot of 0 values and only one column will have "True" value of 1. Let's see an example.

Let us create a sample dataframe with X features and y as a target. load_boston is the function available in sklearn.datasets class.

import pandas as pd
from sklearn.datasets import load_boston
bunch = load_boston()
y = bunch.target
X = pd.DataFrame(bunch.data, columns=bunch.feature_names)

Pandas method called "get_dummies" makes it very easy to quickly transform the data.

ndf=X.join(pd.get_dummies(X['RAD'], prefix='RAD'))

CRIM ZN INDUS CHA ... RAD_1.0 RAD_2.0 RAD_3.0 RAD_4.0 RAD_5.0 RAD_6.0 RAD_7.0 RAD_8.0 RAD_24.0

As you can see all the unique values in "RAD" column are provided with their own columns. If pandas find a value "24.0", it will create a new column with default value of 0 and change it to 1 only if found "True".  This is really wonderful. The only problem is that if there are thousand unique values in this column, we will have to deal with one thousand columns!

In order to reduce the number of columns we will use "category_encoders" module instead of "get_dummies" like this...

import category_encoders as ce
enc = ce.BinaryEncoder(cols=['RAD']).fit_transform(X)

CRIM ZN INDUS CHAS ... RAD_0 RAD_1 RAD_2 RAD_3 RAD_4

As you can see there are now only 5 additional columns and the 9 unique values of  original column are distributed among themselves. Unlike one hot encoding where only 1 column is populated as "True", here there are 2 or 3 columns those may have "1" in it. This will reduce the number of column and allow us to use "hot encodings" even if there are too many unique values in a given column.