Introduction

This is an exercise in classifying exercise performance, given data from body-mounted sensors on six subjects, performing one type of exercise correctly or in any of four incorrect ways. The data were presented in:

Velloso, E.; Bulling, A.; Gellersen, H.; Ugulino, W.; Fuks, H. Qualitative Activity Recognition of Weight Lifting Exercises. Proceedings of 4th International Conference in Cooperation with SIGCHI (Augmented Human ’13) . Stuttgart, Germany: ACM SIGCHI, 2013.

For an overview and link to the paper, see: http://groupware.les.inf.puc-rio.br/har

Classification task

Here, using a training dataset extracted from the above, we want to predict, on a supplied test set, which of the five forms of the exercise the user is executing. This involves cleaning the data, selecting appropriate classifiers, selecting which data features to use, training and testing the classifiers, and predicting the test set.

Data cleaning

The raw features in the dataset consist of short time sequences (“windows”) of gyroscope and accelerometer readings describing the subject’s motion, along with identifying information (timestamps, window ID numbers, etc.), and aggregate information computed and supplied once per window (min, max, variance, skewness, kurtosis). The target class is supplied as letters A through E. There are approximately 20000 rows.

The aggregate information is very sparse, and not supplied for all windows. Since the number of readings per window is small, higher moments were of dubious quality. The test set does not include entire window sequences, just individual samples, so it is not useful to base the prediction on time-series information – prediction will have to be done on the basis of individual samples. Some fields include spurious text strings – these are treated as NA. So, the following features were excluded up front:

With those columns removed, the orignal 159 features (not counting the class) drops to 52. These are still all the features that contain actual sensor readings, so this is not really a reduced set of features.

At this point, one-tenth of the data, containing a representative portion of samples of each class, was split off and reserved for use as a validation set, and the remaining nine-tenths was used as the training set. (In the following, “training set” or “full training set” refers to this nine-tenths portion, not to the original pml-training.csv set.)

Initial classifier selection

Using this subset of features, a number of classifiers were tested. The choice was restricted by these needs:

These are the initial set of classifiers (package name in parentheses):

All of these were trained on the same one of the small datasets, and tested on another. All were allowed to use their default options (with the exception of gbm, for which the distribution was set to multinomial in order to support multi-class output, and which had to be told not to be verbose). Here are their accuracies.

—– Model —– — Accuracy —
randomForest 0.9093
svmPoly 0.841
svmRadial 0.7339
svm 0.7574
gbm 0.8919
lda 0.6758
PenalizedLDA 0.6758
qda 0.8552
knn 0.5912


Four had accuracies above 80% on the small set, so these were retained for use on the full set:

Feature selection

The next step is to see whether features can be omitted without overly compromising accuracy. Two methods were used.

First, highly correlated features were identified by computing their correlation matrix and allowing caret’s findCorrelation to suggest one of each pair of correlated features for removal. This was run on the full training set rather than one of the small subsets. Surprisingly, this only identified a few features as highly correlated (fourteen, of which it suggested seven to remove). It seemed hardly worth removing these few.

Next, the variable importance was extracted from each model using varImp. Two of the models – randomForest and gbm – computed their own variable importance. For the others – svmPoly and qda – varImp computed an importance based on the receiver operating characteristic (ROC). The importance scale isn’t absolute – it is re-scaled to be in the range 0-100 – so a common threshold for cutting off importance wouldn’t be meaningful across all the classifiers.

These show the distributions of variable importances for each classifier.

The importances based on the ROC (svmPoly, qda) were all fairly high. Importances for randomForest and gbm are bunched near zero with a tail at the high end, so these might seem candidates for discarding low-importance features.

However, when gbm was run on the full training set, it lost accuracy using only the reduced set of features selected using its own variable importance, and removing higly-correlated features. And randomForest was fast enough to train that it didn’t have an excuse for cutting down features. So, ultimately, the same 52 features, only excluding the useless features, were used for all.

One other avenue not explored was principal components analysis (PCA). It might be interesting to see whether accuracy survives dimension reduction by using only the highest significance principal components.

Training and testing

At last, we’re ready to run the classifiers on the full training set. (Beware – if you are considering running this, it may take an entire day to complete, depending on your hardware.)

All classifiers were trained under control of caret’s train, once with bootstrap resampling (trainControl method “boot”), and then with five-fold cross-validation but without repeating the entire run multiple times (trainControl method “cv” with number=5). The bootstrap resampling runs were used for the final models.

OUt-of-sample accuracies for the four models on the validation set are shown here. (Out-of-sample error is one minus the accuracy. This is also shown as requested, though it feels more fun to report a nice high accuracy. Also, to be pedantic about terminology, these are the requested cross-validation errors, i.e. the errors on the reserved validation set. The term cross-validation also appeared above in the context of parameters used with caret’s train and trainControl functions. That usage of “cross-validation” and this are unrelated – caret does its own internal cross-validation, and the models themselves are likely doing cross-validation as well as part of their own training procedures.)

—– Model —– — Accuracy — —– Error —–
randomForest 0.9974 0.0026
svmPoly 0.9944 0.0056
gbm 0.9714 0.0286
qda 0.8837 0.1163


Prediction

To improve classification results, we can take a vote among a collection of independent classifiers. Even weak classifiers can be combined to yield a better result than if used separately, so long as the classifiers are not biased the same way, and the majority do not tend to misclassify the the same samples. The classifiers we have at hand include two that are doing quite well, thank you, on their own, but we can see if voting will improve the prediction.

One reason for hope that this may work is that the set of classifiers includes some that are very different in structure – a tree-based classifier such as a random forest, that can subdivide the space into tiny hyperrectangles, is not much like a support vector machine that uses curvilinear decision boundaries but is structurally opposed to overfitting. (It’s unfortunate that K-nearest neighbors did not perform well, as that would be even more different. But even had it performed well, it would not a good choice for a large dataset, as it has to carry along and search though that dataset on each prediction.)

The classifiers don’t get an equal vote – the are weighted by their separate accuracy, in the hope that including a few bad classifiers won’t pull the good ones down. Note that voting was tried without weighting by accuracy, and the result was worse than the best two separate classifiers, so weighting is necessary.

Enough talk – let’s vote and see what the results are. We’ll repeat the accuracy table, but include the voting results.

—– Model —– — Accuracy — —– Error —–
voting 0.998 0.002
randomForest 0.9974 0.0026
svmPoly 0.9944 0.0056
gbm 0.9714 0.0286
qda 0.8837 0.1163