'Non contiguous array pointer traversing issue in pybind11?
Please does anyone have any idea why when I pass my 2 dim Numpy float64 (Double) array normally it prints row by row correctly, col1, col2, col3, col4 as I traverse the pointer by using pos.
However when I pass a particular numpy array of shape 42 rows, 4 cols it instead prints, all of col1, then all of col2, all of col3 and all of col4, rather than a row at a time, please can anyone spot what I'm doing wrong?
I welcome any and all tips as to the potential cause or a resolution :)
I was thinking perhaps its due to memory limitations, however other arrays which are larger don't have any issues
Please note I'm using pybind11 to pass my numpy array to C++, which then traverses the ptr to print out each row at a time, which has been working nicely for "array A" & "array B" except for the new "array C" which is 42 rows by 4 columns, all arrays are float 64.
void Pricing::print_array_simple(py::array_t<double>& in_results) {
if (in_results.ndim() != 2) {
throw std::runtime_error("Results should be a 2-D Numpy array");
}
py::buffer_info buf = in_results.request();
double* ptr = (double*)buf.ptr;
int array_number_rows =in_results.shape()[0]; //rows
int array_max_size = in_results.shape()[1];//cols
cout << "rows: " << array_number_rows << " Cols: " << array_max_size << "\n";
size_t pos = 0;
std::cout << "\n---- START OF ARRAY ----\n";
for (size_t i = 0; i < array_number_rows; i++) {
//rows
for (size_t j = 0; j < array_max_size; j++) {
//cols
std::cout << std::left <<std::setw(12) <<ptr[pos];
pos++;
}
std::cout << "|\n";
}
}
Array that work correctly (72, 7) e.g. Array A or Array B:
CCY TENOR BID ASK MID LAST_BID_TIME_TODAY_REALTIME SETTLEMENT_DATE_RT
0 1.0 1.0 1265.2000 1266.7000 1265.9500 1.651646e+09 1.652051e+09
1 1.0 2.0 1265.3600 1266.3500 1265.8600 1.651661e+09 1.652656e+09
2 1.0 3.0 1265.2400 1266.2800 1265.7600 1.651662e+09 1.654729e+09
3 1.0 4.0 1264.4500 1265.5500 1265.0000 1.651662e+09 1.657494e+09
4 1.0 5.0 1263.5600 1264.6400 1264.1000 1.651662e+09 1.660000e+09
.. ... ... ... ... ... ... ...
67 6.0 8.0 6.7351 6.7411 6.7381 1.651662e+09 1.683500e+09
68 6.0 9.0 6.7870 6.7970 6.7920 1.651662e+09 1.714950e+09
69 6.0 10.0 6.8431 6.8614 6.8522 1.651662e+09 1.746486e+09
70 6.0 11.0 6.9734 6.9883 6.9809 1.651662e+09 1.778022e+09
71 6.0 12.0 7.1178 7.1277 7.1228 1.651659e+09 1.809558e+09
C++ output:
---- START OF ARRAY ----
1 1 1265.2 1266.7 1265.95 1.65165e+09 1.65205e+09 |
1 2 1265.36 1266.35 1265.86 1.65166e+09 1.65266e+09 |
1 3 1265.24 1266.28 1265.76 1.65166e+09 1.65473e+09 |
1 4 1264.45 1265.55 1265 1.65166e+09 1.65749e+09 |
1 5 1263.56 1264.64 1264.1 1.65166e+09 1.66e+09 |
1 6 1259.79 1262.01 1260.9 1.65166e+09 1.66795e+09 |
1 7 1256.05 1258.09 1257.07 1.65166e+09 1.6759e+09 |
1 8 1250.87 1254.47 1252.67 1.65166e+09 1.68359e+09 |
1 9 1242.32 1244.57 1243.44 1.65166e+09 1.71521e+09 |
1 10 1233.71 1243.59 1238.65 1.65165e+09 1.74675e+09 |
1 11 1234.94 1242.71 1238.82 1.65165e+09 1.77845e+09 |
1 12 1228.37 1236.78 1232.57 1.65165e+09 1.8099e+09 |
2 1 76.4175 76.425 76.4213 1.65166e+09 1.65179e+09 |
This array doesn't work, 42 rows by 4 cols e.g."array C":
CCY TENOR BID ASK
0 1.0 0.0 1255.9000 1256.4000
1 1.0 1.0 1256.0000 1256.5000
2 1.0 2.0 1255.4000 1256.0000
3 1.0 3.0 1254.6000 1255.2000
4 1.0 4.0 1251.4000 1252.2000
5 1.0 5.0 1247.7000 1248.5000
6 1.0 6.0 1244.0000 1244.9000
7 2.0 0.0 76.2600 76.3000
8 2.0 1.0 76.4600 76.5100
9 2.0 2.0 76.7300 76.7900
10 2.0 3.0 77.0100 77.0700
11 2.0 4.0 77.8100 77.8800
12 2.0 5.0 78.5600 78.6500
13 2.0 6.0 79.4900 79.5700
output in C++:
It iterates through the CCY's, then iterates through the Tenors, then iterates through the Bids & then the Asks.
---- START OF ARRAY ----
1 1 1 1 |
1 1 1 2 |
2 2 2 2 |
2 2 3 3 |
3 3 3 3 |
3 4 4 4 |
4 4 4 4 |
5 5 5 5 |
5 5 5 6 |
6 6 6 6 |
6 6 0 1 |
6 0 1 2 |
3 4 5 6 |
0 1 2 3 |
4 5 6 0 |
1 2 3 4 |
5 6 0 1 |
2 3 4 5 |
3 4 5 6 |
1255.9 1256 1255.4 1254.6 |
1251.4 1247.7 1244 76.26 |
76.46 76.73 77.01 77.81 |
78.56 79.49 29.46 29.41 |
29.335 29.265 29.07 28.89 |
28.705 14442 14470 14507 |
14545 14650 14755 14860 |
52.41 52.63 52.82 52.97 |
53.38 53.72 54.02 6.6681 |
6.6861 6.7026 6.7151 6.7416 |
6.7571 6.7681 1256.4 1256.5 |
1256 1255.2 1252.2 1248.5 |
1244.9 76.3 76.51 76.79 |
77.07 77.88 78.65 79.57 |
29.485 29.44 29.375 29.305 |
29.12 28.94 28.755 14442 |
14480 14523 14565 14680 |
14785 14900 52.45 52.68 |
52.88 53.03 53.45 53.8 |
54.1 6.6711 6.6911 6.7086 |
6.7221 6.7486 6.7641 6.7761 |
Python
Python code to create array C:
lst = []
for i in range(1,7):
for j in range(7):
lst.append([float(i),float(j),123.1, 124.1])
lst
arr_example = np.array(lst)
arr_example
Update: then pass arr_example into the function, it seems to work when I make the array like, this, I think it is because in python in reality I am using df.values to extra the np array from the pandas dataframe which causes issues, anyone have any ideas if this is likely, understand what causes the different memory order of the numpy array?
Solution 1:[1]
When creating a Numpy array from scratch in numpy rather than by converting from Pandas to numpy, df.values then the contiguous and correct order is maintained.
:)
e.g. the below works well:
Python code to create array C:
lst = []
for i in range(1,7):
for j in range(7):
lst.append([float(i),float(j),123.1, 124.1])
lst
arr_example = np.array(lst)
arr_example
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Py_trader |