مقدمه
میخواهیم ببینیم چه مدلی برای مسئله ما مناسب تر است. دیتاست ما مربوط به عده ای از افراد عموم جامعه است که سن و مقدار حقوق دریافتی و همچنین جنسیت آنها مشخص شده است. میخواهیم پیش بینی کنیم که آیا خرید خواهند کرد یا نه.
در این مسئله دو روش کلاسیفیکیشن را به کار میبریم و سپس با استفاده از cross validation آنها را ارزیابی میکنیم.
ایده به سادگی این است که به دفعات مختلف دیتای test را و در نتیجه دیتای train را تغییر بدهیم و هر بار مقدار دقت را اندازه گیری کنیم.
در مورد Knearest
اطلاعات بیشتر در مورد kmeans نه knearest
KNN works by finding the distances between a query and all the examples in the data, selecting the specified number examples (K) closest to the query, then votes for the most frequent label (in the case of classification) or averages the labels (in the case of regression).
از تسک کلاس خاتم: منبع 1
Cross-validation is a statistical method used to estimate the performance (or accuracy) of machine learning models.
یک متد آماری داریم که میخواهیم دقت مدل ماشین لرنینگ را تخمین بزنیم.
It is used to protect against overfitting in a predictive model, particularly in a case where the amount of data may be limited.
از آن برای جلوگیری ازOverfit در مدل های پیش بینی کننده استفاده میشود. بویژه زمانی که مقدار داده کم باشد.
In cross-validation, you make a fixed number of folds (or partitions) of the data, run the analysis on each fold, and then average the overall error estimate.
در این روش تعداد ثباتی fold یا پارتیشن برای دیتا در نظر میگیریم.
روی هر کدام آنالیز را انجام میدهیم
و سپس خطای میانگین را محاسبه میکنیم.
When dealing with a Machine Learning task, you have to properly identify the problem so that you can pick the most suitable algorithm which can give you the best score. But how do we compare the models?
Say, you have trained the model with the dataset available and now you want to know how well the model can perform.
One approach can be that you are going to test the model on the dataset you have trained it on, but this may not be a good practice.
So what is wrong with testing the model on the training dataset?
چرا تست کردن مدل با training dataset جوابگو نیست؟
If we do so, we assume that the training data represents all the possible scenarios of real-world and this will surely never be the case.
Our main objective is that the model should be able to work well on the real-world data, although the training dataset is also real-world data, it represents a small set of all the possible data points(examples) out there.
So to know the real score of the model, it should be tested on the data that it has never seen before and this set of data is usually called testing set.
But if we split our data into training data and testing data, aren’t we going to lose some important information that the test dataset may hold? Let us see the different types of cross validation and find out the answers?
Types of Cross Validation انواع ارزیابی
There are different types of cross validation methods, and they could be classified into two broad categories – Non-exhaustive and Exhaustive Methods. We’re going to look at a few examples from both the categories.
دو دسته اصلی داریم:
- non exhaustive
- exhaustive
Non-exhaustive Methods
Non-exhaustive cross validation methods, as the name suggests do not compute all ways of splitting the original data. Let us go through the methods to get a clearer understanding.
همه روش ها انجام داده نمیشود:
do not compute all ways of splitting the original data.
Holdout method –
This is a quite basic and simple approach in which we divide our entire dataset into two parts viz- training data and testing data.
روش ساده تقسیم بندی داده به train و test
As the name, we train the model on training data and then evaluate on the testing set.
با train آموزش میدهیم و با test ارزیابی میکنیم.
Usually, the size of training data is set more than twice that of testing data, so the data is split in the ratio of 70:30 or 80:20.
In this approach, the data is first shuffled randomly before splitting.
ابتدا داده را شافل میکنیم.
As the model is trained on a different combination of data points, the model can give different results every time we train it, and this can be a cause of instability.
Also, we can never assure that the train set we picked is representative of the whole dataset.
در این روش نمیتوان مطمئن بود که دیتای train نماینده خوبی از کل داده ها باشد.
Also when our dataset is not too large, there is a high possibility that the testing data may contain some important information that we lose as we do not train the model on the testing set.
همچنین یک موضوع مهم و نگران کننده که وجود دارد این است که در مواردی که دیتا زیاد بزرگ باشد، ممکن است دیتای تست شامل اطلاعات مهمی باشد که در هنگام آموزش دادن در نظر گرفته نشود.
The hold-out method is good to use when you have a very large dataset, you’re on a time crunch, or you are starting to build an initial model in your data science project.
روش hold-out method زمانی خوب است که دیتای ما بزرگ باشد. در تنگنای زمانی باشیم یا در ابتدای پیدا کردن مدل ml باشیم.
K fold cross validation
K-fold cross validation is one way to improve the holdout method.
یک روش برای بهبود معایب hold out است.
This method guarantees that the score of our model does not depend on the way we picked the train and test set.
این روش تضمین میکند که امتیاز مدل ما به نحوه انتخاب train و test ندارد.
The data set is divided into k number of subsets and the holdout method is repeated k number of times. Let us go through this in steps:
کل دیتا به k زیر مجموعه تبدیل میشود و روش hold out به تعداد k بار تکرار میشود.
randomly split your entire dataset into k number of folds (subsets)
به صورت رندوم k بار دیتا را اسپلیت میکنیم
For each fold in your dataset, build your model on k – 1 folds of the dataset. Then, test the model to check the effectiveness for kth fold
هر بار به تعداد k-1 بار دیتا را آموزش میدهیم و با نمونه باقی مانده مدل را ارزیابی میکنیم
Repeat this until each of the k-folds has served as the test set
این چرخه را آنقدر تکرار میکنیم که مطمئن باشیم هر فولد حداقل یکبار در نقش test قرار گرفته باشد.
The average of your k recorded accuracy is called the cross-validation accuracy and will serve as your performance metric for the model.
میانگین دقت های بدست آمده را cross-validation accuracy میگوییم و بعنوان معیار عملکرد مدل ما بحساب میآید.
Because it ensures that every observation from the original dataset has the chance of appearing in training and test set, this method generally results in a less biased model compare to other methods.
احتمال بایاس کمتر نسبت به سایر روش ها وجود دارد.
It is one of the best approaches if we have limited input data
این یکی از بهترین روش هاست زمانی که محدودیت داده ورودی داشته باشیم.
The disadvantage of this method is that the training algorithm has to be rerun from scratch k times, which means it takes k times as much computation to make an evaluation.
عیب این روش این است که الگوریتم مورد استفاده باید به تعداد k بار از ابتدا تا انتها تکرار شود. یعنی مرتبه زمانی و محاسباتی ما k برابر میشود.
Stratified K Fold Cross Validation
Stratified یعنی طبقه بندی شده
Using K Fold on a classification problem can be tricky.
استفاده از K Fold در مسئله classification می تواند مشکل باشد.
Since we are randomly shuffling the data and then dividing it into folds, chances are we may get highly imbalanced folds which may cause our training to be biased.
به دلیل imbalance شدن داده در هنگام شافل ، احتمال بایاس در دیتای train ما وجود دارد.
For example, let us somehow get a fold that has majority belonging to one class(say positive) and only a few as negative class.
برای مثال یک fold را در نظر بگیرید که اکثریت موارد آن به یک کلاس تعلق دارند. (فرض کنید کلاس مثبت)و تنها تعدادی در کلاس negative قرار دارند.
This will certainly ruin our training and to avoid this we make stratified folds using stratification.
این موضوع ممکن است باعث خراب شدن پروسه training ما شود. برای حل این موضوع به سراغ stratification یا طبف
Stratification is the process of rearranging the data so as to ensure that each fold is a good representative of the whole.
طبقه بندی فرآیند مرتب سازی مجدد داده ها به گونه ای است که اطمینان حاصل شود که هر فولد نماینده خوبی از کل است.
For example, in a binary classification problem where each class comprises of 50% of the data, it is best to arrange the data such that in every fold, each class comprises of about half the instances.
به عنوان مثال، در یک مسئله طبقه بندی باینری که در آن هر کلاس 50٪ از داده ها را تشکیل می دهد، بهتر است داده ها به گونه ای تنظیم شوند که در هر فولد، هر کلاس شامل حدود نیمی از نمونه ها باشد.
Exhaustive Methods
Exhaustive cross validation methods and test on all possible ways to divide the original sample into a training and a validation set.
روش جامع. یعنی رفتن تمام راه های ممکن برای تقسیم داده ها به train و test
Leave-P-Out cross validation
When using this exhaustive method, we take p number of points out from the total number of data points in the dataset(say n).
در زمان استفاده از روش exhaustive تعداد p تا از نمونه ها را از کل دیتا بیرون در نظر میگیریم. (فرض کنیم کل داده ها n تا باشد.)
While training the model we train it on these (n – p) data points and test the model on p data points.
رویn-p آموزش میدهیم و با p نمونه تست می کنیم.
We repeat this process for all the possible combinations of p from the original dataset.
این کار را برای تمام حالاتp ممکن در دیتا ست تکرار میکنیم.
Then to get the final accuracy, we average the accuracies from all these iterations.
This is an exhaustive method as we train the model on every possible combination of data points.
آموزش را روی انواع ترکیب های ممکن (از تست و train) انجام میدهیم.
Remember if we choose a higher value for p, then the number of combinations will be more and we can say the method gets a lot more exhaustive.
اگر مقدار بالاتری ازp را انتخاب کنیم تعداد حالات ترکیبی بیشتر خواهد بود. و متود جامع تر خواهد بود.
Leave-one-out cross validation
This is a simple variation of Leave-P-Out cross validation and the value of p is set as one.
در این حالتp=1 انتخاب میشود.
This makes the method much less exhaustive as now for n data points and p = 1, we have n number of combinations.
جامعیت کمتری داریم. یاد آوری(رویn-p آموزش میدهیم و با p نمونه تست می کنیم.) حال داریم p=1
What is Rolling Cross Validation?
For time-series data the above-mentioned methods are not the best ways to evaluate the models.
در دیتاهای سری زمانی روش های بیان شده بهترین روش نیستند.
Here are two reasons as to why this is not an ideal way to go:
دو دلیل دارد
- Shuffling the data messes up the time section of the data as it will disrupt the order of events
- شافل کردن سری زمانی را به هم میزند. زیرا ترتیب رویدادها به هم میخورد
- Using cross-validation, there is a chance that we train the model on future data and test on past data which will break the golden rule in time series i.e. “peaking in the future is not allowed”.
- ممکن است دیتای آینده را درtrain داشته باشیم و با دیتای گذشته test انجام دهیم که این مطلوب نیست.
Keeping these points in mind we perform cross validation in this manner
- We create the fold (or subsets) in a forward-chaining fashion.
- ایجاد فولدها با یک مد پیشرو صورت میگیرد.
- Suppose we have a time series for stock prices for a period of n years and we divide the data yearly into n number of folds. The folds would be created like:
- فرض کنید که قیمت سهام را در ده سال به صورت سری زمانی داریم و هر سال آن را به N تا فولد تقسیم میکنیم. فولدها مانند زیر تشکیل خواهند شد:
Here as we can see in the first iteration, we train on the data of the first year and then test it on 2nd year.
Similarly in the next iteration, we train the on the data of first and second year and then test on the third year of data.
Note: It is not necessary to divide the data into years, I simply took this example to make it more understandable and easy.
سوالات مصاحبه:
1. What is the purpose of cross validation?
The purpose of cross–validation is to test the ability of a machine learning model to predict new data.
It is also used to flag problems like overfitting or selection bias and gives insights on how the model will generalize to an independent dataset.
2. How do you explain cross validation?
Cross-validation is a statistical method used to estimate the performance (or accuracy) of machine learning models.
It is used to protect against overfitting in a predictive model, particularly in a case where the amount of data may be limited.
مخصوصا زمانی که محدودیت دیتایی داریم
In cross-validation, you make a fixed number of folds (or partitions) of the data, run the analysis on each fold, and then average the overall error estimate.
3. What are the types of cross validation?
The 4 Types of Cross Validation in Machine Learning are:
- Holdout Method
- K-Fold Cross-Validation
- Stratified K-Fold Cross-Validation
- Leave-P-Out Cross-Validation
4. What is cross validation and why we need it?
Cross-Validation is a very useful technique to assess the effectiveness of a machine learning model, particularly in cases where you need to mitigate overfitting.
It is also of use in determining the hyperparameters of your model, in the sense that which parameters will result in the lowest test error.
5. Does cross validation reduce Overfitting?
Cross-validation is a procedure that is used to avoid overfitting and estimate the skill of the model on new data.
There are common tactics that you can use to select the value of k for your dataset.
پایان مقاله