I am using R and applying MART (Alg. for multiple additive regression trees) on a training set to build prediction models. When I look at the ROC curve, it is not monotone. I would be grateful if someone can help me with how I should fix this. I am guessing the issue is that initially, MART generates n trees and if these trees are not the same for all the models I am building, the results will not be comparable.
Here are the steps I take:
1) Fix the false-negative cost, c_fn. Let cost = c(0, 1, c_fn, 0).
2) use the following line to build the mart model:
mart(x, y, lx, martmode='class', niter=2000, cost.mtx=cost)
where x is the matrix of training set variables, y is the observation matrix, lx is the matrix which specifies which of the variables in x is numerical, which one categorical.
3) I predict the test set observations using the mart model found in step 2 using this line:
y_pred = martpred(x_test, probs=T)
4) I compute the false-positive and false-negative errors as follows:
t = 1/(1+c_fn) %threshold based on Bayes optimal rule where c_fp=1 and c_fn.
p_0 = length(which(y_test==1))/dim(y_test)[1]
p_01 = sum(1*(y_pred[,2]t & y_test==0))/dim(y_test)[1]
p_11 = sum(1*(y_pred[,2]t & y_test==1))/dim(y_test)[1]
p_fp = p_01/(1-p_0)
p_tp = p_11/p_0
5) repeat step 1-4 for a new false-negative cost.
