Scoring in GridSearchCV for XGBoost

Question

I'm currently trying to analyze data for the first time using XGBoost. I want to find the best parameters using GridsearchCV. I want to minimize the root mean squared error and to do this, I used "rmse" as eval_metric. However, scoring in grid search does not have such a metric. I found on this site that the "neg_mean_squared_error" does the same, but I found that this gives me different results than the RMSE. When I calculate the root of the absolute value of the "neg_mean_squared_error", I get a value of around 8.9 while a different function gives me a RMSE of about 4.4. I don't know what goes wrong or how I get these two functions to agree/give the same values?

Because of this problem, I get wrong values as "best_params_" which give me a higher RMSE than some values I initially started with to tune.

Can anyone please explain me how to get score on the RMSE in the grid search or why my code gives different values?

Thanks in advance.

def modelfit(alg, trainx, trainy, useTrainCV=True, cv_folds=10, early_stopping_rounds=50):
    if useTrainCV:
        xgb_param = alg.get_xgb_params()
        xgtrain = xgb.DMatrix(trainx, label=trainy)
        cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], nfold=cv_folds,
                          metrics='rmse', early_stopping_rounds=early_stopping_rounds)
        alg.set_params(n_estimators=cvresult.shape[0])

    # Fit the algorithm on the data
    alg.fit(trainx, trainy, eval_metric='rmse')

    # Predict training set:
    dtrain_predictions = alg.predict(trainx)
    # dtrain_predprob = alg.predict_proba(trainy)[:, 1]
    print(dtrain_predictions)
    print(np.sqrt(mean_squared_error(trainy, dtrain_predictions)))

    # Print model report:
    print("\nModel Report")
    print("RMSE : %.4g" % np.sqrt(metrics.mean_squared_error(trainy, dtrain_predictions)))

 param_test2 = {
 'max_depth':[6,7,8],
 'min_child_weight':[2,3,4]
}

grid2 = GridSearchCV(estimator = xgb.XGBRegressor( learning_rate =0.1, n_estimators=2000, max_depth=5,
 min_child_weight=2, gamma=0, subsample=0.8, colsample_bytree=0.8,
 objective= 'reg:linear', nthread=4, scale_pos_weight=1, random_state=4),
 param_grid = param_test2, scoring='neg_mean_squared_error', n_jobs=4,iid=False, cv=10, verbose=20)
grid2.fit(X_train,y_train)
# Mean cross-validated score of the best_estimator
print(grid2.best_params_, np.sqrt(np.abs(grid2.best_score_))), print(np.sqrt(np.abs(grid2.score(X_train, y_train))))
modelfit(grid2.best_estimator_, X_train, y_train)
print(np.sqrt(np.abs(grid2.score(X_train, y_train))))

Answer 1

In GridSearchCV the scoring parameter is transformed so that higher values are always better than lower values. In your example neg_mean_squared_error is just a negated version of RMSE. You should not interpret neg_mean_squared_error to be RMSE, rather in your cross-validation you should compare values of neg_mean_squared_error where a higher value is better than lower values.

In the scoring parameter portion of the model_evaluation documentation this behavior is mentioned.

Scikit-Learn Scoring Parameter Documentation

Answer 2

It's because XGBoostRegressor.score returns the coefficient of determination of the prediction, not RMSE.