Just as airplanes are legally grouped as certified or Experimental, engines follow the same scheme. Certified engines are specific, FAA-certified models sold on certified airplanes such as Cessnas and Pipers. These come only from certified engine makers such as Continental, Lycoming and Rotax and are rigidly specified right down to the exact part number rod bolts and so on. On the other hand, Experimental engines can be just about anything and sometimes are. In practice many Experimental engines closely follow certified practice with certified parts, but wild creativity is allowed.
Using a new certified engine on an Experimental airplane is certainly no problem other than it’s the most eye-wateringly expensive option available. Also, a certified engine must be kept absolutely stock to retain its certified status. If you modify it with a different intake, ignition or even rocker covers, it is no longer a certified engine and drops in value as it’s now an Experimental engine. The same goes for complying with service bulletins and airworthiness directives.
