Historically the word “motor” was used when referring to solid fueled rockets, and “engines” for liquid fueled. Unfortunately, this rule breaks down for hybrid rockets where the fuel can be solid, or the oxidiser can be solid. That being said, most refer to hybrid rockets as motors.

These days the two words are used interchangeably. For example, the Wikipedia page for ullage motors uses a combination of “motor” and “engine”.

Which should you use? In lieu of a more specific name the following guideline would serve you well:

If there’s a solid component, use “motor”, otherwise use “engine”, as “rocket engine” is the all-encompassing term.

As for the words of “motor” vs “engine” themselves, MIT has an interesting page on the etymology of both words.