'Segmentation fault while Importing python module to a Python script to call from C++
After an exhaustive search, I have yet to find and answer to this particular problem. My C++ code calls a python function def s3_compress(): from mytest.py as follows:
mytest.py
import gzip
import boto3
import argparse
def s3_compress(bucketname, key, intputBuf):
try:
cred = boto3.Session().get_credentials()
s3client = boto3.client('s3', aws_access_key_id=cred.access_key, aws_secret_access_key=cred.secret_key, aws_session_token=cred.token)
#s_in = b"Lots of content here"
s_in = str.encode(intputBuf)
gzip_object = gzip.compress(s_in)
s3client.put_object(Bucket=bucketname, Body=gzip_object, Key=key)
except Exception as e:
print(e)
raise e
My sample C++ code is as below:
main.cpp
#include <iostream>
#include "Python.h"
int main()
{
char *bucketname = "sample";
char *objectname = "filename.gz";
char *data = "Hellow world!!";
Py_Initialize();
PyObject *pName, *pModule, *pFunc, *pArgs, *pValue;
PyRun_SimpleString("import sys; sys.path.append('.')");
pName = PyUnicode_FromString((char*)"mytest");
pModule = PyImport_Import(pName);
pFunc = PyObject_GetAttrString(pModule, (char*)"s3_compress");
PyObject *args = PyTuple_New(3);
PyObject *bucketName = PyUnicode_FromString(bucketname);
PyObject *objectName = PyUnicode_FromString(objectname);
PyObject *outputData = PyUnicode_FromString(data);
if (PyTuple_SetItem(args, 0, bucketName))
std::cout << "Failed to pass Argument1" << std::endl;
if (PyTuple_SetItem(args, 1, objectName))
std::cout << "Failed to pass Argument2" << std::endl;
if (PyTuple_SetItem(args, 2, outputData))
std::cout << "Failed to pass Argument3" << std::endl;
pValue = PyObject_CallObject(pFunc, pArgs);
auto result = _PyUnicode_AsString(pValue);
std::cout << result << std::endl;
//Close the python instance
Py_Finalize();
}
Above code throws Segmentation fault in PyObject_CallObject(). Please note that the above python script runs perfectly when executed in the python interpreter.
If I replace the mytest.py script with some simple one as below, it works fine.
def test(BucketName, ObjectName, Data ):
return "Bucketname: " + BucketName + " Key: " + ObjectName +" Data: "+Data;
And when I add import boto3 in this script, crash is observed.
import boto3
def test(BucketName, ObjectName, Data ):
return "Bucketname: " + BucketName + " Key: " + ObjectName +" Data: "+Data;
Hence I confirmed that the root cause for this crash is import modules in python script. I also tried using the PyImport_Import() method to try to import boto3 through C++ as well. But no luck favors.
All I am doing this is because, I need to push the string buffer data into S3 bucket on-fly as a file with gz format. Since I could not find the direct solution from AWS SDK C++, but I could find from Python SDK, I am trying this.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
