We've had PvP focused Talismans planned for a while now. It just so happens that a few of them would help add some counterplay to characters like the Prisoner.
Also, the buffs generated by the Talisman Passives are still just buffs and can be stolen, stripped, etc. by other character abilities, so we're less worried that they will create a new problem as there are already plenty of existing counters to them.
-Kaz