You cannot use self-signed certificate. To sign a file (msi, exe, dll, pdf, etc ...) you must have a valid signature certified and generated by hard key (like usb token or other hardware token).
To be more precise, the process consists of two stages:
1. Create a self-signed certificate signed by your custom CA (like, MyCompanyCA.crt)
2. Create a self-generated certificate signed by your custom CA (like, MyCompanyCodeSign.crt)
I have usb tokens, but this is only
recomendation. Files can be used also.
Cert must be generated RSA2048/4084 and have OID for "Code signing". No other MUST, as I know.
A self signed is valid only for test, no one can trust (and will not trust) about a certificate not relaesed by a CA.
I've seen many applications that use this approach in production.
The key is that the installer and the application itself are signed with the same certificate. Then, when you run the installer, you'll be asked to trust the publisher using information from your certificate. Once you agree to trust it, the OS will remember this certificate as trusted. When you launch the application, the OS won't display a warning because you trusted the certificate during installation. Therefore, it's important to create an installer with the same signature so that the user only sees the trust prompt once during installation.
Commercial CA don't tell to as that 3 dialogs exists:
1. Unknown publisher. When code not signed at all.
2. Unverified publisher. When code signed by self-generated (self-signed CA) cert.
3. Verified publisher. When buy cert.
Last two options in prompt dialog have info about company name, site, contacts, adress. I attached two old screens.
If you want to try there plenty of user guides that explain how to do: this is one signcode tool that works in Linux (Ubuntu) https://manpages.ubuntu.com/manpages/noble/en/man1/signcode.1.html and you can use it to sign PE files with your self-signed certificate.
Wow! I don't have this in my distro. I'll definitely try it when I get a chance.