A modkill is the moderator's means of removing player slots from a game for administrative reasons.
The decision to modkill should not be made lightly (on mafiascum.net, doing so requires permission from a List Moderator, as the offending player will likely need to be given out-of-game in addition to in-game sanctions). It's most effective to modkill when the integrity of a player slot has been broken, as it may be possible to forcibly replace a compromised player with a new player. An example of a modkillable offense would be a player quoting their Role PM (or other private information) to "confirm" themselves. On the other hand, a player who accidentally learns someone else's alignment through out-of-game means or an error in moderation may just need to be forcibly replaced with a different player. However, the decision to modkill becomes more of a headache as the situation grows more complex - for instance, if a player inadvertently learns another player's role and then proceeds to tell the rest of the game what they know, then a modkill could be warranted. Whatever the outcome, it's most important for a moderator to think their action through.
In addition, on websites where the Mafia community is very small and replacements for players who are inactive cannot be found, the modkill is a last resort to deal with that player slot.
As a general rule, modkills should be designed to be as disadvantageous to the player and their faction as possible. A modkill on Town will end the Day (thus preventing them from lynching), and a modkill on any kind of scum will keep the Day going (so that the Town can ideally lynch another scum). In addition, players who are modkilled as a punishment (i.e. for causing deliberate harm) should have their role retconned to Survivor; since modkilled players are by definition dead and Survivors only win if they are alive at the end of the game, this denies the modkilled player any reason to claim a win from the game. All of these are important to prevent strategies where it may be advantageous to a faction to deliberately cheat in spite of the punishments; for instance, simply killing cheaters with no other effects allows for a breaking strategy where players who are to be lynched instead quote their Role PM, getting modkilled and allowing the Town to "lynch" someone else (or rather, force them to quote their Role PM as well).
This has nothing to do with actually killing the mod as if they were a player in the game.