Xcode architecture independent LDFLAGS arm64 x86 Apple Silicon M1

Question

I have a project which comes in project.pbxproj with this

OTHER_LDFLAGS = (
    "-lgit2",
    "-force_load",
    External/libgit2.a,
    /usr/local/lib/libssh2.a,  <-- The problem
    "-lcrypto",
    "-lssl",
    "-lcurl",
);

but on a Apple Silicon M1 homebrew comes with

/opt/homebrew/lib/libssh2.a instead

/usr/local/lib/libssh2.a like in x86

I already tried to add both, but with less success

OTHER_LDFLAGS = (
    "-lgit2",
    "-force_load",
    External/libgit2.a,
    /usr/local/lib/libssh2.a,
    /opt/homebrew/lib/libssh2.a,
    "-lcrypto",
    "-lssl",
    "-lcurl",
);

How to make it work on x86 and on arm64 without alternate code changes here ?

e.g. a variable which is set during a pre-build script ?