'R CMD SHLIB External Library Linking Issue (on Ubuntu/Linux)

I have a C file that I am trying to turn into a shared object to be read into R. I want to link that SO with OpenBLAS. I am compiling R from source (and linking it to OpenBLAS) since I have read that there is an environment variable that needs to be set in order to allow external library linking to take place when using R CMD SHLIB (namely PKG_CFLAGS and their variants for C++ and Fortran), as suggested here (for example):

R CMD SHLIB to create a DLL from a source .c file which needs an external library (on Windows x64)

I install and compile OpenBLAS from source, and then run the following install script for configuring and compiling R (along with linking OpenBLAS):

#####################################################
### How to compile R and link it against OpenBLAS ###
#####################################################

# Assuming no root priviledges on the destination machine,
# so we have to compile our software locally

# set default installation directory:
INSTDIR=$HOME/opt

# NOTE: NOT INSTALLING TO /opt but $HOME/opt to avoid sudo requirement

# place to store the source packages
SOURCES=$HOME/SourceBuilds

# make dirs (if they dont exist)
mkdir $INSTDIR
mkdir $SOURCES

# make sure GNU compiler is being used
CC="gcc"
CFLAGS="-O3"
CXX="g++"
CXXFLAGS="-O3"
CPICFLAGS="-fPIC"
FPICFLAGS="-fPIC"
CXXPICFLAGS="-fPIC"

BLAS_LIBS="-L${INSTDIR}/OpenBLAS/lib -lopenblas"
DYLIB_LDFLAGS="-L${INSTDIR}/OpenBLAS/lib -lopenblas"

# THIS IS CRITICAL, AS THESE FLAGS SHOULD BE SET FOR USE WITH "R CMD SHLIB"
PKG_CFLAGS='-O3 -lopenblas'
PKG_CXXFLAGS='-O3 -lopenblas'
PKG_CPPFLAGS='-O3 -lopenblas'
PKG_FFLAGS='-O3 -lopenblas'

# NOTE: Trying 'march=znver2' is not recognized as a valid option. Likely OS is too old.

###################
### Compiling R ###
###################

cd $SOURCES

# downlad tarball 
wget https://cran.rstudio.com/src/base/R-4/R-4.2.0.tar.gz

# extract tarball
tar -xf R-4.2.0.tar.gz

# change to source directory
cd R-4.2.0

# We want all R-packages to be compiled with optimization flags
mkdir ${HOME}/.R
echo CC=${CC} > ${HOME}/.R/Makevars
echo CXX=${CXX} >> ${HOME}/.R/Makevars
echo CFLAGS= -std=gnu99 ${CFLAGS} >> ${HOME}/.R/Makevars
echo CXXFLAGS=${CXXFLAGS} >> ${HOME}/.R/Makevars

# configure with local prefix and shared-blas
./configure --prefix=${INSTDIR}/R_OpenBLAS \
    --with-x --with-cairo \
    --with-pic \
    --enable-R-profiling \
    --enable-memory-profiling \
    --enable-R-shlib \
    --disable-java \
    --with-blas \
    --with-lapack \
    --with-readline \
    --with-ICU \
    --enable-static \
    --disable-shared \
    --with-libpng \
    --with-jpeglib \
    --with-libtiff \
    --disable-prebuilt-html \
    --with-recommended-packages=no
#   --enable-BLAS-shlib \
#    2>&1 | tee configure.log

# compile
make -j8
#2>&1 | tee build.log

# install
make install

### now we manually link R's Blas against our newly
### compiled OpenBLAS

# sometimes the 'lib'-dir appears as 'lib64'
LIBSUFFIX=lib/R/lib

if [ -d ${INSTDIR}/R_OpenBLAS/lib64 ]
then
  LIBSUFFIX=lib64/R/lib
fi

# move old blas-library
mv ${INSTDIR}/R_OpenBLAS/${LIBSUFFIX}/libRblas.so ${INSTDIR}/R_OpenBLAS/${LIBSUFFIX}/libRblas.so.backup

# create a symbolic link to the OpenBLAS library
#ln -s ${INSTDIR}/OpenBLAS/lib/libopenblas.so ${INSTDIR}/R_OpenBLAS/${LIBSUFFIX}/libRblas.so
#ln -s ${INSTDIR}/OpenBLAS/lib/libopenblas.so /usr/lib/libRblas.so

ln -s ${INSTDIR}/OpenBLAS/lib/libopenblas_zenp-r0.3.20.so ${INSTDIR}/R_OpenBLAS/${LIBSUFFIX}/libRblas.so
ln -s ${INSTDIR}/OpenBLAS/lib/libopenblas_zenp-r0.3.20.so /usr/lib/libRblas.so

### R is now linked against OpenBLAS.
### In order to use this R-Version over the old one, do this:

echo export PATH=${INSTDIR}/R_OpenBLAS/bin:${PATH} >> ${HOME}/.bashrc

# do this for current session:
export PATH=${INSTDIR}/R_OpenBLAS/bin:${PATH}

# check that correct R is being called now
which R

# The libraries installed by the new install script are not found by call to '.libPaths()'
export R_LIBS_SITE=/usr/local/lib/R/site-library:$R_LIBS_SITE
export R_LIBS=/usr/local/lib/R/site-library:$R_LIBS
export R_LIBS_USER=/usr/local/lib/R/site-library:$R_LIBS_USER

# NOTE: TO VALIDATE INSTALL, OPEN RSTUDIO AND TYPE IN: sessionInfo()
# RESULT SHOULD BE: Matrix products: default
# BLAS/LAPACK: /home/<username>/opt/OpenBLAS/lib/libopenblas_zenp-r0.3.20.so

I then run R CMD SHLIB on the C file in question and the compile line looks like so:

gcc -I"/home/<username>/opt/R_OpenBLAS/lib/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -std=gnu99 -O3 -c hergmod.c -o hergmod.o
gcc -shared -L/home/<username>/opt/R_OpenBLAS/lib/R/lib -L/usr/local/lib -o hergmod.so hergmod.o -L/home/<username>/opt/R_OpenBLAS/lib/R/lib -lR

Clearly some redundancy there, but I don't believe anything that will cause a problem. It seems from this compile line that everything looks good. However, I did not see any performance increase, and after running ldd on the shared object created by R CMD SHLIB I see the following:

$ ldd hergmod.so 
    linux-vdso.so.1 (0x00007ffeff3f0000)
    libR.so => not found
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f429dc3e000)

The fact that libR.so is not found is clearly a problem, but this list should be much more comprehensive than this as well. For example, on an R install that was not compiled from source (and not trying to link OpenBLAS), when I run ldd on this same file after R CMD SHLIB the result is this:

$ ldd hergmod.so 
    linux-vdso.so.1 (0x00007ffcb55ce000)
    libR.so => /usr/lib/libR.so (0x00007f8ee3ec8000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8ee3ad7000)
    libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3 (0x00007f8ee386a000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8ee34cc000)
    libreadline.so.7 => /lib/x86_64-linux-gnu/libreadline.so.7 (0x00007f8ee3283000)
    libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f8ee3012000)
    liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f8ee2dec000)
    libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f8ee2bdc000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f8ee29bf000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8ee27b7000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8ee25b3000)
    libicuuc.so.60 => /usr/lib/x86_64-linux-gnu/libicuuc.so.60 (0x00007f8ee21fb000)
    libicui18n.so.60 => /usr/lib/x86_64-linux-gnu/libicui18n.so.60 (0x00007f8ee1d5a000)
    libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f8ee1b2b000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8ee190c000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f8ee474e000)
    libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f8ee16e2000)
    libicudata.so.60 => /usr/lib/x86_64-linux-gnu/libicudata.so.60 (0x00007f8edfb39000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f8edf7b0000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8edf598000)

I highly suspect that the problem is with configure and either a missing configuration variable, or perhaps one or several of them need to be removed. I am not sure where the problem lies and was hoping someone more versed in R might have an idea. Any tips/pointers would be greatly appreciated. Thanks!



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source