libcrypto from the openssl project seemed like the right choice due to its ubiquity, when I needed to build a program that used base64 encoding, sha1 digests, and both symmetric and asymmetric encryption. Only after my project was finished did I notice NSS which is the cryptographic library used by firefox, which in hindsight I probably would have used. There is in fact a fedora initiative to migrate all applications from openssl to NSS due to standardisation issues. Another interesting cryptographic library I noticed later was nettle.
The main issue I (and others) have with openssl is its interface. I.E. both the API and the ABI. The API seems to have evolved, rather than have been designed. There are multiple ways to access the cryptographic algorithms, like the EVP, BIO and PEM interfaces. This wouldn't be so bad if it was documented appropriately. Unfortunately the documents are incomplete and even the examples which seem to be copied from mailing lists are incomplete and have gotchas. Note I only noticed the the demos directory distributed with openssl after I completed my project, but it does seem like a useful source of information. Anyway hopefully my libcrypto wrapper library above will shield you from a lot of the interface issues. As for the ABI, it seems very unstable, with new versions of the library being released in quick succession, which makes problems for binary compatibility.
Generally I think the BSD originated openssh and openssl projects do not care enough about interface. I've had problems also with openssh. Consider the meanings of the following openssh command options:
command | meaning |
ssh -P22 | use privileged port, use protocol version 2 |
ssh -p22 | use port 22 |
scp -P22 | use port 22 |
ssh -X | Enable X11 forwarding (but break apps in subtle ways) |
ssh -Y | Really enable X11 forwarding |