'Segmentation fault while Importing python module to a Python script to call from C++

After an exhaustive search, I have yet to find and answer to this particular problem. My C++ code calls a python function def s3_compress(): from mytest.py as follows:

mytest.py

import gzip
import boto3
import argparse

def s3_compress(bucketname, key, intputBuf):
    try:
        cred = boto3.Session().get_credentials()
        s3client = boto3.client('s3', aws_access_key_id=cred.access_key, aws_secret_access_key=cred.secret_key, aws_session_token=cred.token)
        #s_in = b"Lots of content here"
        s_in = str.encode(intputBuf)
        gzip_object = gzip.compress(s_in)
        s3client.put_object(Bucket=bucketname, Body=gzip_object, Key=key)

    except Exception as e:
        print(e)
        raise e

My sample C++ code is as below:

main.cpp

#include <iostream>
#include "Python.h"

int main()
{

    char *bucketname = "sample";
    char *objectname = "filename.gz";
    char *data = "Hellow world!!";

    Py_Initialize();
    PyObject *pName, *pModule, *pFunc, *pArgs, *pValue;
    PyRun_SimpleString("import sys; sys.path.append('.')");

    pName = PyUnicode_FromString((char*)"mytest");
    pModule = PyImport_Import(pName);
    pFunc = PyObject_GetAttrString(pModule, (char*)"s3_compress");
    PyObject *args = PyTuple_New(3);
    PyObject *bucketName = PyUnicode_FromString(bucketname);
    PyObject *objectName = PyUnicode_FromString(objectname);
    PyObject *outputData = PyUnicode_FromString(data);

    if (PyTuple_SetItem(args, 0, bucketName))
        std::cout << "Failed to pass Argument1" << std::endl;
    if (PyTuple_SetItem(args, 1, objectName))
        std::cout << "Failed to pass Argument2" << std::endl;
    if (PyTuple_SetItem(args, 2, outputData))
        std::cout << "Failed to pass Argument3" << std::endl;

    pValue = PyObject_CallObject(pFunc, pArgs);
    auto result = _PyUnicode_AsString(pValue);
    std::cout << result << std::endl;

    //Close the python instance
    Py_Finalize();
}

Above code throws Segmentation fault in PyObject_CallObject(). Please note that the above python script runs perfectly when executed in the python interpreter.

If I replace the mytest.py script with some simple one as below, it works fine.

def test(BucketName, ObjectName, Data ):
    return "Bucketname: " + BucketName + " Key: " + ObjectName +" Data: "+Data;

And when I add import boto3 in this script, crash is observed.

import boto3
def test(BucketName, ObjectName, Data ):
    return "Bucketname: " + BucketName + " Key: " + ObjectName +" Data: "+Data;

Hence I confirmed that the root cause for this crash is import modules in python script. I also tried using the PyImport_Import() method to try to import boto3 through C++ as well. But no luck favors.

All I am doing this is because, I need to push the string buffer data into S3 bucket on-fly as a file with gz format. Since I could not find the direct solution from AWS SDK C++, but I could find from Python SDK, I am trying this.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source