Linker error related to OpenSSL when Python is modified to use OpenSSL in the tokenizer

yanwencheng · December 28, 2020, 1:14pm

I created the C language class method of openssl rsa,
Modified Makefile.pre.in to make it compile to xxx.o.
But when compiling and installing in redhat, the error as shown in the figure was finally reported

       RE_ BUILTIN  -c ./Modules/hashtable.c -o Modules/hashtable.o
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-generate  -I. -I./Include    -DPy_ BUILD_ CORE_ BUILTIN  -c ./Modules/symtablemodule.c -o Modules/symtablemodule.o
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-generate  -I. -I./Include    -DPy_ BUILD_ CORE_ BUILTIN  -c ./Modules/xxsubtype.c -o Modules/xxsubtype.o
    gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-generate  -I. -I./Include    -DPy_ BUILD_ CORE \
    -DGITVERSION="\"`LC_ ALL=C git --git-dir ./.git rev-parse --short HEAD`\"" \
    -DGITTAG="\"`LC_ ALL=C git --git-dir ./.git describe --all --always --dirty`\"" \
    -DGITBRANCH="\"`LC_ ALL=C git --git-dir ./.git name-rev --name-only HEAD`\"" \
    -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c
    gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-generate  -I. -I./Include    -DPy_ BUILD_ CORE -o Python/frozen.o Python/frozen.c
    rm -f libpython3.7m.a
    ar rcs libpython3.7m.a Modules/getbuildinfo.o Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o Parser/ykkjrsa.o Objects/abstract.o Objects/accu.o Objects/boolobject.o  Objects/bytes_ methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o  Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/capsule.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o  Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/weakrefobject.o Python/_ warnings.o Python/Python-ast.o Python/asdl.o Python/ast.o Python/ast_ opt.o Python/ast_ unparse.o Python/bltinmodule.o Python/ceval.o Python/compile.o Python/codecs.o Python/dynamic_ annotations.o Python/errors.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/graminit.o Python/import.o Python/importdl.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/peephole.o Python/pyarena.o  Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/context.o Python/hamt.o Python/pythonrun.o Python/pytime.o Python/bootstrap_ hash.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_ unicode.o Python/fileutils.o Python/dynload_ shlib.o    Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_ sre.o  Modules/_ codecsmodule.o  Modules/_ weakref.o  Modules/_ functoolsmodule.o  Modules/_ operator.o  Modules/_ collectionsmodule.o  Modules/_ abc.o  Modules/itertoolsmodule.o  Modules/atexitmodule.o  Modules/signalmodule.o  Modules/_ stat.o  Modules/timemodule.o  Modules/_ threadmodule.o  Modules/_ localemodule.o  Modules/_ iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/zipimport.o  Modules/faulthandler.o  Modules/_ tracemalloc.o Modules/hashtable.o  Modules/symtablemodule.o  Modules/xxsubtype.o Python/frozen.o
    gcc -pthread    -fprofile-generate -Xlinker -export-dynamic -o python Programs/python.o libpython3.7m.a -lcrypt -lpthread -ldl  -lutil   -lm
    Libpython3.7m. A (ykkjrsa. O): in the function "RSA"_ In "encode":
    /Yunku / python-3.7.7-rsa / parser / ykkjrsa. C: 188: for 'bio'_ new_ mem_ BUF 'undefined reference
    Libpython3.7m. A (ykkjrsa. O): in the function "RSA"_ In "encode":
    Ykkjrsa. C: (. Text + 0x72): for "PEM"_ read_ bio_ RSA_ Undefined reference of pubkey '
    Ykkjrsa. C: (. Text + 0x8e): for "bio"_ free_ Undefined reference
    Ykkjrsa. C: (. Text + 0x9e): for "RS"

merwok · December 10, 2020, 3:00pm

Hello! Are you adding a new function to a module in the standard library?

Have you followed the development instructions in the devguide?

Next time, please copy-paste text rather than an image, and set LANG=C so that everyone can read output and error messages.

yanwencheng · December 11, 2020, 12:13am

When I use make & make install to compile and install python, the error printed on the console.
These references use openssl

RE_BUILTIN  -c ./Modules/_tracemalloc.c -o Modules/_tracemalloc.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-generate  -I. -I./Include    -DPy_BUILD_CORE_BUILTIN  -c ./Modules/hashtable.c -o Modules/hashtable.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-generate  -I. -I./Include    -DPy_BUILD_CORE_BUILTIN  -c ./Modules/symtablemodule.c -o Modules/symtablemodule.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-generate  -I. -I./Include    -DPy_BUILD_CORE_BUILTIN  -c ./Modules/xxsubtype.c -o Modules/xxsubtype.o
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-generate  -I. -I./Include    -DPy_BUILD_CORE \
      -DGITVERSION="\"`LC_ALL=C git --git-dir ./.git rev-parse --short HEAD`\"" \
      -DGITTAG="\"`LC_ALL=C git --git-dir ./.git describe --all --always --dirty`\"" \
      -DGITBRANCH="\"`LC_ALL=C git --git-dir ./.git name-rev --name-only HEAD`\"" \
      -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-generate  -I. -I./Include    -DPy_BUILD_CORE -o Python/frozen.o Python/frozen.c
rm -f libpython3.7m.a
ar rcs libpython3.7m.a Modules/getbuildinfo.o Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o Parser/ykkjrsa.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/capsule.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/compile.o Python/codecs.o Python/dynamic_annotations.o Python/errors.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/graminit.o Python/import.o Python/importdl.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/peephole.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/context.o Python/hamt.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/dynload_shlib.o    Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/_operator.o  Modules/_collectionsmodule.o  Modules/_abc.o  Modules/itertoolsmodule.o  Modules/atexitmodule.o  Modules/signalmodule.o  Modules/_stat.o  Modules/timemodule.o  Modules/_threadmodule.o  Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/zipimport.o  Modules/faulthandler.o  Modules/_tracemalloc.o Modules/hashtable.o  Modules/symtablemodule.o  Modules/xxsubtype.o Python/frozen.o
gcc -pthread    -fprofile-generate -Xlinker -export-dynamic -o python Programs/python.o libpython3.7m.a -lcrypt -lpthread -ldl  -lutil   -lm  

libpython3.7m.a(ykkjrsa.o)：In the function ‘rsa_encode’:
ykkjrsa.c:(.text+0x72)：Undefined reference to ‘PEM_read_bio_RSA_PUBKEY’
ykkjrsa.c:(.text+0x8e)：Undefined reference to‘BIO_free_all’ 
ykkjrsa.c:(.text+0x9e)：Undefined reference to‘RSA_size’ 
ykkjrsa.c:(.text+0xcd)：Undefined reference to‘RSA_public_encrypt’

steve.dower · December 11, 2020, 10:44am

I don’t think libpython references OpenSSL anywhere - that’s all kept in a separate module (_ssl) - so you’ll either have to add your function to that module or else add a reference to OpenSSL into the main lib.

yanwencheng · December 14, 2020, 2:44am

I modified tokenizer.c in decode_str method function used RSA_ public_ Encrypt, how can I add a reference to OpenSSL in the main library.

steve.dower · December 14, 2020, 7:53pm

No idea, I’m afraid. I’m not familiar with the Linux build system, only the Windows one.

I’d recommend moving your modification to the _ssl module, as it already has the reference.

kpfleming · December 14, 2020, 8:23pm

Also, it would be really nice if someone edited the subject of this topic, since it’s not really about running ‘make’, it’s about modifying the CPython code and getting it to build.

vstinner · December 28, 2020, 1:12pm

Which OS is that? RHEL? If yes, which RHEL version? How did you get Python? Which commands did you type to build Python? Which Python version are you trying to build (full version with the 3 numbers please: Python 3.7.x: what is x)?

What are your changes? Why did you modify this file?

It seems like you are using OpenSSL functions in Python, but you didn’t link libpython to OpenSSL. You need to add $OPENSSL_LIBS to $LIBS in Makefile.pre.in in this case: tokenize.c is part of “libpython”.

vstinner · December 28, 2020, 1:13pm

I don’t think that you injected your code at the right place. The tokenizer is too deep into Python internals. I would suggest you to add an import hook or replace existing import hook, to decrypt your binary files on import.

vstinner · December 28, 2020, 1:14pm

I modified the topic. The linker error is caused by changes in the code base.