[artificial-intelligence] What's is the difference between train, validation and test set, in neural networks?

I'm using this library to implement a learning agent.

I have generated the training cases, but I don't know for sure what the validation and test sets are.
The teacher says:

70% should be train cases, 10% will be test cases and the rest 20% should be validation cases.

edit

I have this code for training, but I have no idea when to stop training.

  def train(self, train, validation, N=0.3, M=0.1):
    # N: learning rate
    # M: momentum factor
    accuracy = list()
    while(True):
        error = 0.0
        for p in train:
            input, target = p
            self.update(input)
            error = error + self.backPropagate(target, N, M)
        print "validation"
        total = 0
        for p in validation:
            input, target = p
            output = self.update(input)
            total += sum([abs(target - output) for target, output in zip(target, output)]) #calculates sum of absolute diference between target and output

        accuracy.append(total)
        print min(accuracy)
        print sum(accuracy[-5:])/5
        #if i % 100 == 0:
        print 'error %-14f' % error
        if ? < ?:
            break

edit

I can get an average error of 0.2 with validation data, after maybe 20 training iterations, that should be 80%?

average error = sum of absolute difference between validation target and output, given the validation data input/size of validation data.

1
        avg error 0.520395 
        validation
        0.246937882684
2
        avg error 0.272367   
        validation
        0.228832420879
3
        avg error 0.249578    
        validation
        0.216253590304
        ...
22
        avg error 0.227753
        validation
        0.200239244714
23
        avg error 0.227905    
        validation
        0.199875013416

We create a validation set to

• Measure how well a model generalizes, during training
• Tell us when to stop training a model;When the validation loss stops decreasing (and especially when the validation loss starts increasing and the training loss is still decreasing)

Why validation set used:

Would appreciate any thoughts on the situation with 3 data sets. Say a logistic regression model is fitted yielding the following accuracy (Gini): Train: 70%; Test 58% and Out-of-time validation: 66%.

Actually all the possible combinations of predictors bring the same results with quite a huge drop between train and test data sets. The sample size is around 8k divided into train and test 70/30. OOT sample contains a few thousands of cases. Regularization, ensembles didn't help in solving this.

I doubt whether this is something I should concern if OOT performance is acceptable and close to train sample performance?

Say you train a model on a training set and then measure its performance on a test set. You think that there is still room for improvement and you try tweaking the hyper-parameters ( If the model is a Neural Network - hyper-parameters are the number of layers, or nodes in the layers ). Now you get a slightly better performance. However, when the model is subjected to another data ( not in the testing and training set ) you may not get the same level of accuracy. This is because you introduced some bias while tweaking the hyper-parameters to get better accuracy on the testing set. You basically have adapted the model and hyper-parameters to produce the best model for that particular training set.

A common solution is to split the training set further to create a validation set. Now you have

• training set
• testing set
• validation set

You proceed as before but this time you use the validation set to test the performance and tweak the hyper-parameters. More specifically, you train multiple models with various hyper-parameters on the reduced training set (i.e., the full training set minus the validation set), and you select the model that performs best on the validation set.

Once you've selected the best performing model on the validation set, you train the best model on the full training set (including the valida- tion set), and this gives you the final model.

Lastly, you evaluate this final model on the test set to get an estimate of the generalization error.

Training Dataset: The sample of data used to fit the model.

Validation Dataset: The sample of data used to provide an unbiased evaluation of a model fit on the training dataset while tuning model hyperparameters. The evaluation becomes more biased as skill on the validation dataset is incorporated into the model configuration.

Test Dataset: The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset.

Cross-validation set is used for model selection, for example, select the polynomial model with the least amount of errors for a given parameter set. The test set is then used to report the generalization error on the selected model. From here: https://www.coursera.org/learn/machine-learning/lecture/QGKbr/model-selection-and-train-validation-test-sets

In simple words define Training set, Test set, Validation set

Training set: Is used for finding Nearest neighbors. Validation set: Is for finding different k which is applying to train set. Test set: Is used for finding the maximum accuracy and unseen data in future.

Training set: A set of examples used for learning, that is to fit the parameters [i.e., weights] of the classifier.

Validation set: A set of examples used to tune the parameters [i.e., architecture, not weights] of a classifier, for example to choose the number of hidden units in a neural network.

Test set: A set of examples used only to assess the performance [generalization] of a fully specified classifier.

From ftp://ftp.sas.com/pub/neural/FAQ1.txt section "What are the population, sample, training set, design set, validation"

The error surface will be different for different sets of data from your data set (batch learning). Therefore if you find a very good local minima for your test set data, that may not be a very good point, and may be a very bad point in the surface generated by some other set of data for the same problem. Therefore you need to compute such a model which not only finds a good weight configuration for the training set but also should be able to predict new data (which is not in the training set) with good error. In other words the network should be able to generalize the examples so that it learns the data and does not simply remembers or loads the training set by overfitting the training data.

The validation data set is a set of data for the function you want to learn, which you are not directly using to train the network. You are training the network with a set of data which you call the training data set. If you are using gradient based algorithm to train the network then the error surface and the gradient at some point will completely depend on the training data set thus the training data set is being directly used to adjust the weights. To make sure you don't overfit the network you need to input the validation dataset to the network and check if the error is within some range. Because the validation set is not being using directly to adjust the weights of the netowork, therefore a good error for the validation and also the test set indicates that the network predicts well for the train set examples, also it is expected to perform well when new example are presented to the network which was not used in the training process.

Early stopping is a way to stop training. There are different variations available, the main outline is, both the train and the validation set errors are monitored, the train error decreases at each iteration (backprop and brothers) and at first the validation error decreases. The training is stopped at the moment the validation error starts to rise. The weight configuration at this point indicates a model, which predicts the training data well, as well as the data which is not seen by the network . But because the validation data actually affects the weight configuration indirectly to select the weight configuration. This is where the Test set comes in. This set of data is never used in the training process. Once a model is selected based on the validation set, the test set data is applied on the network model and the error for this set is found. This error is a representative of the error which we can expect from absolutely new data for the same problem.

EDIT:

Also, in the case you do not have enough data for a validation set, you can use crossvalidation to tune the parameters as well as estimate the test error.