close
The Wayback Machine - https://web.archive.org/web/20241114051643/https://www.geeksforgeeks.org/box-cox-transformation-using-python/
Open In App

Python | Box-Cox Transformation

Last Updated : 13 May, 2022
Summarize
Comments
Improve
Suggest changes
Like Article
Like
Save
Share
Report
News Follow

Imagine you are watching a horse race and like any other race, there are fast runners and slow runners. So, logically speaking, the horse which came first and the fast horses along with it will have the smaller difference of completion time whereas the slowest ones will have a larger difference in their completion time. 

We can relate this to a very famous term in statistics called variance which refers to how much is the data varying with respect to the mean. Here in our example, there is an inconsistent variance (Heteroscedasticity) between the fast horses and the slow horses because there will be small variations for shorter completion time and vice versa. 

Hence, the distribution for our data will not be a bell curve or normally distributed as there will be a longer tail on the right side. These types of distributions follow Power law or 80-20 rule where the relative change in one quantity varies as the power of another.PowerLawIn the above plot, we can see power-law distribution which is having peaked for short running times because of the small variance and heavy tail due to longer running times. These power-law distributions are found in the field of physics, biology, economics, etc. 

So, just think for a second that if these distributions are found in so many fields, what if we could transform it to a much comfortable distribution like normal distribution? That would make our life a lot easier. Fortunately, we have a way to transform power-law or any non-linear distribution to normal using a Box-Cox Transformation

Let us think intuitively that if we were to do this transform ourselves, how would we proceed? 

It is clear from the above-shown figure that if somehow we could inflate the variability for the left-hand side of non-normal distribution i.e peak and reduce the variability at the tails. In short, trying to move the peak towards the centre then we can get a curve close to the bell-shaped curve. 

Formally, A Box cox transformation is defined as a way to transform non-normal dependent variables in our data to a normal shape through which we can run a lot more tests than we could have. 

Mathematics behind Box-Cox Transformation: How can we convert our intuitive thinking into a mathematical transformation function? Logarithmic transformation is all we need. When a log transformation is applied to non-normal distribution, it tries to expand the differences between the smaller values because the slope for the logarithmic function is steeper for smaller values whereas the differences between the larger values can be reduced because, for large values, log distribution has a moderate slope. That is what we thought of doing, right? Box-cox Transformation only cares about computing the value of \lambda  which varies from – 5 to 5. A value of \lambda  is said to be best if it is able to approximate the non-normal curve to a normal curve. The transformation equation is as follows:y(\lambda)=\left\{\begin{array}{l} \left(y^{\lambda}-1\right) / \lambda \text { if } \lambda \neq 0 \\ \log y \quad \text { if } \lambda=0 \end{array}\right. This function requires input to be positive. Using this formula manually is a very laborious task thus many popular libraries provide this function.

Implementation: SciPy’s stats package provides a function called boxcox for performing box-cox power transformation that takes in original non-normal data as input and returns fitted data along with the lambda value that was used to fit the non-normal distribution to normal distribution. Following is the code for the same. 

Example: 

Python3

# Python3 code to show Box-cox Transformation
# of non-normal data
 
# import modules
import numpy as np
from scipy import stats
 
# plotting modules
import seaborn as sns
import matplotlib.pyplot as plt
 
# generate non-normal data (exponential)
original_data = np.random.exponential(size = 1000)
 
# transform training data & save lambda value
fitted_data, fitted_lambda = stats.boxcox(original_data)
 
# creating axes to draw plots
fig, ax = plt.subplots(1, 2)
 
# plotting the original data(non-normal) and
# fitted data (normal)
sns.distplot(original_data, hist = False, kde = True,
            kde_kws = {'shade': True, 'linewidth': 2},
            label = "Non-Normal", color ="green", ax = ax[0])
 
sns.distplot(fitted_data, hist = False, kde = True,
            kde_kws = {'shade': True, 'linewidth': 2},
            label = "Normal", color ="green", ax = ax[1])
 
# adding legends to the subplots
plt.legend(loc = "upper right")
 
# rescaling the subplots
fig.set_figheight(5)
fig.set_figwidth(10)
 
print(f"Lambda value used for Transformation: {fitted_lambda}")

                    

Output:OutputWe can see that the non-normal distribution was converted into a normal distribution or rather close to normal using the SciPy.stats.boxcox(). Does Box-cox always work? The answer is NO. Box-cox does not guarantee normality because it never checks for the normality which is necessary to be foolproof that it has correctly transformed the non-normal distribution or not. It only checks for the smallest Standard deviation. 

Therefore, it is absolutely necessary to always check the transformed data for normality using a probability plot.



Similar Reads

Python | Inverse Fast Fourier Transformation
Inverse Fast Fourier transform (IDFT) is an algorithm to undoes the process of DFT. It is also known as backward Fourier transform. It converts a space or time signal to a signal of the frequency domain. The DFT signal is generated by the distribution of value sequences to different frequency components. Working directly to convert on Fourier trans
2 min read
Time Series Data Transformation using Python
Time series data transformation is a crucial step in time series analysis and forecasting, it involves converting raw time series data into a format that is suitable for analysis and modelling. In this article, we will see how we can use time series data transformation, which will be beneficial to our analysis. Types of transformationsIn univariate
6 min read
An in-place algorithm for String Transformation
Given a string, move all even positioned elements to the end of the string. While moving elements, keep the relative order of all even positioned and odd positioned elements the same. For example, if the given string is "a1b2c3d4e5f6g7h8i9j1k2l3m4", convert it to "abcdefghijklm1234567891234" in-place and in O(n) time complexity. Below are the steps
15+ min read
2D Transformation in Computer Graphics | Set 1 (Scaling of Objects)
We can use a 2 × 2 matrix to change or transform, a 2D vector. This kind of operation, which takes in a 2-vector and produces another 2-vector by a simple matrix multiplication, is a linear transformation. By this simple formula, we can achieve a variety of useful transformations, depending on what we put in the entries of the matrix. For our purpo
5 min read
Fast Fourier Transformation for polynomial multiplication
Given two polynomial A(x) and B(x), find the product C(x) = A(x)*B(x). There is already an O([Tex]n^2 [/Tex]) naive approach to solve this problem. here. This approach uses the coefficient form of the polynomial to calculate the product.A coefficient representation of a polynomial [Tex]A(x)=\sum_{j=0}^{n-1}a_jx^j [/Tex]is a = a0, a1, ..., an-1. Exa
13 min read
Iterative Fast Fourier Transformation for polynomial multiplication
Given two polynomials, A(x) and B(x), find the product C(x) = A(x)*B(x). In the previous post, we discussed the recursive approach to solve this problem which has O(nlogn) complexity. Examples: Input : A[] = {9, -10, 7, 6} B[] = {-5, 4, 0, -2} Output : C(x) = [Tex]-12x^6 -14x^5 +44x^4 -20x^3 - 75x^2 +86x -45[/Tex] In real-life applications such as
12 min read
String transformation using XOR and OR
Given two binary strings. The task is to check if string s1 can be converted to string s2 by performing the given operations any number of times. Choose any two adjacent characters in a string s1 and replace one of them by a^b and the other by a [Tex]\oplus [/Tex]b (a OR b). Examples: Input: S1 = "11", S2 = "10" Output: YES Select two adjacent char
7 min read
NLP | Chunk Tree to Text and Chaining Chunk Transformation
We can convert a tree or subtree back to a sentence or chunk string. To understand how to do it - the code below uses the first tree of the treebank_chunk corpus. Code #1: Joining the words in a tree with space. C/C++ Code # Loading library from nltk.corpus import treebank_chunk # tree tree = treebank_chunk.chunked_sents()[0] print ("Tree
3 min read
Window to Viewport Transformation in Computer Graphics with Implementation
Window to Viewport Transformation is the process of transforming 2D world-coordinate objects to device coordinates. Objects inside the world or clipping window are mapped to the viewport which is the area on the screen where world coordinates are mapped to be displayed. General Terms: World coordinate - It is the Cartesian coordinate w.r.t which we
8 min read
Feature Transformation Techniques in Machine Learning
Most machine learning algorithms are statistics dependent, meaning that all of the algorithms are indirectly using a statistical approach to solve the complex problems in the data. In statistics, the normal distribution of the data is one that a statistician desires to be. A normal distribution of the data helps statisticians to solve the complex p
6 min read
Prompt Engineering for Transformation of Text
In the series of learning "How to write better prompts and customize them as per the specific use?" so, that we do not need to read articles that say 30 prompts to make your life easy with ChatGPT:). In this article, we will see how can we use ChatGPT to transform a piece of text and use the LLM as a ChatBot. But before that, we need to set the API
6 min read
Lexicographically largest permutation transformation
Given an arr[] of size n, the task is to find the lexicographically largest permutation from the given permutation by performing the following operations exactly once where you will choose two integers l and r and reverse the subarray [l, r] and swap [1, l-1] with [r+1, n]. Examples: Input: arr[] = { 2, 3, 1, 5, 4}Output: {5, 4, 1, 3, 2 }Explanatio
11 min read
Sequence Transformation by Cyclic Shifting
Given two arrays of positive integers A[] and B[], both of length N, the task is to determine whether it is possible to transform sequence A into sequence B by repeating the following operation any number of times: Choose an index i (1 ≤ i ≤ N) and replace A[i] with A[i+1]. If i = N, replace A[N] with A[1]. If transformation is possible then print
12 min read
Minimize the String Transformation Cost
Given a string Str and cost array C[] of size n replacing Str[i] with any character will have cost C[i]. The task is to minimize the cost where every odd i, for all i from 1 to n have to match either, S[i] = S[n-i+1] and S[i+1] = S[n-i] or,S[i] = S[n-i] and S[i+1] = S[n-i+1]Examples: Input: N = 4, Str = bdbd, C[] = {1, 2, 3, 4}Output: 0Explanation:
6 min read
Query-Based Array Transformation
Given an array arr[] of size N, initially, all the elements of arr are 0, you have to process q queries of the form "L R X" denoting that fill the array from index l to index r with the number x, the task is to print the final array after all the operations have been performed. Examples: Input: N = 4, queries = [[1, 3, 2], [2, 4, 6], [2, 3, 7]] Out
15 min read
Array Transformation with Repeated Steps and Modulo Operation
Given an array A of size N. In one move you have to follow these steps for every element Ai, Find the size of the array after T repeated steps. If Ai equals 80 at some point, add another element 0 to the end of the arrayReplace Ai by (Ai + 1) modulo 81.Examples: Input: N = 4, A[] = {65, 2, 80, 4}, T = 3Output: 5Explanation: After first move array w
6 min read
Minimizing Sum with Neighbor Value Priority in Array Transformation
Given an array arr[] of length n, where arr[i] represents some value. You have to create an array result[] of length n, such that it follows both conditions given below and the task is to find the minimum sum of the result[] while ensuring that these rules are satisfied. result[i] >= 1, for all i, such that 0 <= i <= nresult[i] with a high
8 min read
Maximum weight transformation of a given string
Given a string consisting of only A's and B's. We can transform the given string to another string by toggling any character. Thus many transformations of the given string are possible. The task is to find Weight of the maximum weight transformation. Weight of a string is calculated using below formula. Weight of string = Weight of total pairs + we
9 min read
Find number of transformation to make two Matrix Equal
Given two matrices A and B of order n*m. The task is to find the required number of transformation steps so that both matrices became equal, print -1 if it is not possible. Transformation step is as: Select any one matrix out of two matrices. Choose either row/column of the selected matrix. Increment every element of select row/column by 1. Example
9 min read
2D Transformation | Rotation of objects
We have to rotate an object by a given angle about a given pivot point and print the new co-ordinates.Examples: Input : {(100, 100), (150, 200), (200, 200), (200, 150)} is to be rotated about (0, 0) by 90 degrees Output : (-100, 100), (-200, 150), (-200, 200), (-150, 200) Input : {(100, 100), (100, 200), (200, 200)} is to be rotated about (50, -50)
7 min read
Data Transformation in Machine Learning
Often the data received in a machine learning project is messy and missing a bunch of values, creating a problem while we try to train our model on the data without altering it. In building a machine learning project that could predict the outcome of data well, the model requires data to be presented to it in a specific way such that the machine le
15+ min read
Array Parity Transformation
Given an array A[] of length N (N is even && N>=2).Where half elements are odd and the other half are even. You can apply the below operation (possibly zero time): Select two distinct indices let's say X and Y and increment A[X] and decrement A[Y].Then your task is to output YES/NO by checking whether the following conditions can be sati
5 min read
Check matrix transformation by flipping sub-matrices along the principal or anti-diagonal
Given two matrices mat1[][] and mat2[][] of size N*M, the task is check if it is possible to transform mat1 into mat2 by selecting any square sub-matrix from mat1 and flipping it along its principal diagonal or anti-diagonal. Examples: Input: N = 2, M = 3, mat1[][] = {{1, 2, 3}, {4, 5, 6}}, mat2[][] = {{1, 4, 3}, {6, 5, 2}}Output: YESExplanation: I
13 min read
Permutation transformation: Minimum operations to achieve permutation
Given an array arr[] of length n, the task is to find the minimum number of operations required to make a permutation of integers 1 to n where you are allowed to select any element arr[i] and replace it with arr[i] % x (x is any positive integer). If it is not possible to make a permutation return -1. Examples: Input: n = 5, arr[] = {2, 3, 4, 4, 5}
8 min read
Box Blur Algorithm - With Python implementation
The pixels in an image are represented as integers. After blurring each pixel 'x' of the resulting image has a value equal to the average of the pixels surrounding 'x' including 'x'. For example, consider a 3 * 3 image as [Tex] image = \begin{bmatrix} 1 & 1 & 1\\ 1 & 7 & 1\\ 1 & 1 & 1 \end{bmatrix} [/Tex] Then, the resulting
4 min read
Pattern to print X in a rectangular box
Given the value of length, print the X pattern in a box using # and " " Examples: Input : 10 Output : ########## ## ## # # # # # # # # # ## # # ## # # # # # # # # # ## ## ########## Input : 7 Output : ####### ## ## # # # # # # # # # # # ## ## ####### Below is the implementation to print X in a rectangular box pattern: C/C++ Code // CPP code to prin
4 min read
Find the position of box which occupies the given ball
Given two array A[] and B[]. Where size of A[] represent the number of rows and A[i] represent the number of boxes in the ith row. Array B[] represents an array of balls where B[i] represents a number on the ball. Given that ball i (having value B[i]) will be placed in a box whose position from beginning is B[i] (row-major). The task is to find the
8 min read
Height of Pyramid formed with given Rectangular Box
Given an array of rectangular blocks with dimensions {l, b} of length m. we can shape each rectangular block into a single square block by trimming either one of its dimensions or both. With these square blocks, a pyramid is formed. Our task is to find the maximum height of the pyramid that can be formed. Note: A pyramid is built by placing a squar
5 min read
Minimize steps to fit the carpet in box by halving its sides
There is a carpet of size A * B [length * breadth]. Given a box of size C * D, the task is to fit the carpet in the box in a minimum number of moves. In one move, you can either decrease the length or the breadth of the carpet by half (floor value of its half). Note: One can even turn the carpet by 90 degrees any number of times, won't be counted a
9 min read
Colorful box arrangement with Removal Constraint
Given an array of colored boxes called arr, the task is to arrange these boxes to form a contiguous segment, where all the boxes have the same color. The segment should have a precise length of L. However, you're allowed to remove a maximum of K boxes to make this arrangement possible. Is it possible to arrange the boxes according to a given condit
8 min read
three90RightbarBannerImg