'Shapes not aligned in Python:
I'm new on stackoverflow and I'm trying to create a custom ridge regression model for a time series analysis. My goal is to predict the sign of stock return while using linear regression with ridge regularization. Details are below in the code. To perform my regression, I'm splitting my data into 5 equal blocks as training and test sets. In the for loop, I'm trying to cross validate my hyper parameters alpha and lambda to find best model in my ridge regression family. (As I'm new to Python, I couldn't make this in other simpler way. If you can correct my code in a simpler way, it would be better)
df = [[5.029188 ,0.560895,-2.432905,-0.766467,4.97],
[5.046780,-0.539126,-2.481951,-0.657977,4.990],
[5.270471,-0.398845,-2.622589,-1.455087,5.001],
[5.450671,-0.336845,-2.352584,-1.235557,4.976],
[5.247153,-0.653155,-2.246789,-1.164277,4.989],
[5.797471,-0.498845,-2.432589,-1.555087,5.03],
[5.350471,-0.598845,-2.672589,-1.655087,5.09],
[5.320471,-0.698845,-2.234589,-1.755087,5.15],
[5.650471,-0.798845,-2.765389,-1.355087,5.10],
[5.760471,-0.898845,-2.223589,-1.255087,5.05],
[5.225712,-0.198845,-2.876589,-1.155087,5.07],
[5.650471,-0.298845,-2.345427,-1.455087,5.13],
[5.122471,-0.356845,-2.586429,-1.567087,5.11],
[5.890471,-0.543845,-2.454369,-1.123087,5.07],
[5.240471,-0.235845,-2.543579,-1.789087,5.12],
[5.980471,-0.654845,-2.245435,-1.245087,5.14],
[5.120471,-0.876845,-2.466753,-1.323487,5.18],
[5.650471,-0.234845,-2.235675,-1.355087,5.20],
[5.432171,-0.765845,-2.353246,-1.765087,5.24],
[5.765471,-0.458845,-2.356535,-1.245087,5.22],
[5.029188 ,0.560895,-2.432905,-0.766467,5.21],
[5.046780,-0.539126,-2.481951,-0.657977,5.32],
[5.270471,-0.398845,-2.622589,-1.455087,5.36],
[5.450671,-0.336845,-2.352584,-1.235557,5.40],
[5.247153,-0.653155,-2.246789,-1.164277,5.32],
[5.797471,-0.498845,-2.432589,-1.555087,5.33],
[5.350471,-0.598845,-2.672589,-1.655087,5.21],
[5.320471,-0.698845,-2.234589,-1.755087,5.24],
[5.650471,-0.798845,-2.765389,-1.355087,5.30],
[5.760471,-0.898845,-2.223589,-1.255087,5.33],
[5.225712,-0.198845,-2.876589,-1.155087,5.40],
[5.650471,-0.298845,-2.345427,-1.455087,5.42],
[5.122471,-0.356845,-2.586429,-1.567087,5.45],
[5.890471,-0.543845,-2.454369,-1.123087,5.50],
[5.240471,-0.235845,-2.543579,-1.789087,5.48],
[5.980471,-0.654845,-2.245435,-1.245087,5.56],
[5.120471,-0.876845,-2.466753,-1.323487,5.60],
[5.650471,-0.234845,-2.235675,-1.355087,5.63],
[5.432171,-0.765845,-2.353246,-1.765087,5.57],
[5.765471,-0.458845,-2.356535,-1.245087,5.63]]
# creating df object with columns specified
data_new = pd.DataFrame(df, columns =['PC1', 'PC2','PC3','PC4','Price'])
PC=['PC1','PC2','PC3','PC4']
def ridge_regression(X, y, W, alpha=1, lambda_value=10, epochs=50):
cost_hist=[]
#W=initial weigths
m = np.shape(X)[0] # total number of samples
n = np.shape(X)[1] # total number of features
X = np.concatenate((np.ones((m, 1)), X), axis=1)
# iterate until the maximum number of epochs
for current_iteration in np.arange(epochs): # begin the process
# compute the dot product between our feature 'X' and weight 'W'
y_estimated = X.dot(W)
y_estimated=pd.DataFrame(y_estimated)
y=pd.DataFrame(y)
# calculate the difference between the actual and predicted value
sign_pred=np.sign(np.log(y_estimated)/np.log(y_estimated).shift(1)).fillna(0)
sign_true=np.sign(np.log(y)/np.log(y).shift(1)).fillna(0)
error = (sign_pred-sign_true)/m
error=error.fillna(0)
# regularization term
ridge_reg_term = (lambda_value / 2 * m) * np.sum(np.square(W))
# calculate the error + regularization term
cost = (1 / 2 * m) * np.sum(error ** 2) + ridge_reg_term
transpose=X.T
dot_prod=pd.DataFrame(transpose.dot(error))
gradient = (1 / m) * (dot_prod.iloc[:,0] + (lambda_value * W))
# Now we have to update our weights
W = W - alpha * gradient
cost_hist.append(cost)
weights=W
return weights
def BlockingTimeSeriesSplit(X,n_splits=5):
train=[]
test=[]
for i in range(n_splits):
n_samples = X.shape[0]
k_fold_size = int(n_samples / n_splits)
indices = np.arange(n_samples)
start =i * k_fold_size
stop = start + k_fold_size
mid = int(0.5 * (stop - start)) + start
train.append(indices[start: mid])
test.append(indices[mid: stop])
return train, test
n=5
f1_score=[]
alpha=[0.01,0.1,1,2,3,5,10,20,30,50,100,1000]
lambda_values=[1,2,3,5,10,20,30,50,100,200,300,500,1000]
a,b=BlockingTimeSeriesSplit(data_new,n)
for i in range(n):
cv_train, cv_test = data_new.iloc[a[i]], data_new.iloc[b[i]]
cv_validation,cv_test_new=train_test_split(cv_test,test_size=0.6)
m_train=np.shape(cv_train[PC])[0]
np.random.seed(11)
W = np.random.randn(np.shape(cv_train[PC])[1] + 1, )
weights_train=ridge_regression(cv_train[PC],cv_train['Price'],W)
weights_train=pd.DataFrame(weights_train)
for a in alpha:
for l in lambda_values:
weights_validation=ridge_regression(cv_validation[PC],cv_validation['Price'],weights_train,alpha=a,lambda_value=l)
weights_validation=pd.DataFrame(weights_validation)
m_valid=np.shape(cv_valid[PC])[0]
X = np.concatenate((np.ones((m_valid, 1)), cv_train[PC]), axis=1)
y_pred=pd.DataFrame(X.dot(weights_validation))
# calculate the difference between the actual and predicted value
sign_pred=np.sign(np.log(y_pred)/np.log(y_pred).shift(1)).fillna(0)
sign_true=np.sign(np.log(cv_test['Price'])/np.log(cv_test['Price']).shift(1)).fillna(0)
tn, fp, fn, tp =confusion_matrix(sign_true,sign_pred,labels=[0,1]).ravel()
precision=tp/(tp+fp)
recall=tp/(tp+fn)
f1=2*(precision*recall)/(precision+recall)
f1_score.append(f1)
print("F1 Score: {}".format(np.mean(f1_score)))
return weights_validation
I'm receiving this error which is about the shape of the two dataframes:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-439-08f7c7adf57f> in <module>
25 for l in lambda_values:
26
---> 27 weights_validation=ridge_regression(cv_validation[PC],cv_validation['Price'],weights_train,alpha=a,lambda_value=l)
28
29 weights_validation=pd.DataFrame(weights_validation)
<ipython-input-430-c693a48acc42> in ridge_regression(X, y, W, alpha, lambda_value, epochs)
40
41 transpose=X.T
---> 42 dot_prod=pd.DataFrame(transpose.dot(error))
43 gradient = (1 / m) * (dot_prod.iloc[:,0] + (lambda_value * W))
44
ValueError: shapes (11,23) and (46,2) not aligned: 23 (dim 1) != 46 (dim 0)
I'm struggling to understand how my error shape become (46,2) as when I trace back it, I see that it should be (23,1) and the dot production should be performed. I'm stuck at this point and looking forward to hearing your thoughts on that.
Solution 1:[1]
The last code:
transpose=X.T
transpose.dot(error)
from the error: (11,23) and (46,2)
So some how error is (46,2), while X is (23,10).
The challenge is to trace these arguments back to your inputs to the function
cv_validation[PC], cv_validation['Price'], weights_train
OK, X is cv_validation[PC]
Oops, I see that you did post ridge_regression. I haven't looked at that.
error = (sign_pred-sign_true)/m
Tracing dimensions thru those two inputs is more than I can readily do. Why are you using a dataframe here? Just to get the shift? I don't know if that complicates things.
You say error should be (23,1), but the error says it's (46,2). We can't verify that. You may need to add some diagnostic shape prints.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 |