Modular Open is a type of game where the role abilities (or modules) that are in the game are known in advance. However, the distribution of these role abilities is not known.
As a silly example, the moderator may promise that the only abilities (aside from factional abilities, obviously) in the game are Cop, Doctor, Vigilante, and Roleblocker. No other abilities are in the game.
Modular Open games generally use this restriction on design space to create unusual role combinations, such as a player who can both act as a Cop and a Vigilante, or a player who can act as a Doctor and a Roleblocker, or a Mafia Vigilante. Note that most of the time, the rule that a player can only use one type of Night Action per Night is in effect to prevent some of these combinations from being overpowered on their own.
Contrast Semi-Open, where the exact roles in the game are known in advance and thus these ability combinations are ruled out.