Classically, it was a relatively simple lookup operation: the opcode determined some bitfields in the ouyput byte(s). \the operands determined the remainder.
These days, particularly on "mature" architectures like x86 and even moreso x86_64, it is vastly more complex since in many cases the operands determine at least part of the opcode selection.
The bottom line is that you need to be intimately familiar with the processor documentation, which in the case of x86_64 runs to around three volumes (plus multiple commentaries from Intel, AMD and others).
I would suggest that the best way of going about this would be to research a suitable subset of the overall x86 opcode set that will be adequate for what you actually need to do. For example, if you only need basic computation and control transfers it would be a fairly safe bet to leave out anything related to floating point and SIMD operations.
A couple of years ago I implemented a disassembler for the V20/V30 chips which are 16-bit and substantially simpler than what we have today. It was fairly hard work.
MarkMLl