We can't as fans get rid of Melnyk. We can however make it obvious we don't support him (the team) and then we risk losing the team. I don't know the mechanism to move a team but if Ottawa is seen to be a town in which hockey is not viable we are more likely to lose the team than to lose Melnyk.
I posted the following thoughts in a different thread. My view is that the Senators will never contend as long as 15% to 20% of the budget is tied up in two highly overpaid players (Phaneuf and Ryan). If they can't be traded then the Senators will only have a hope of competing in about three years. In the meantime players will be traded for cost reasons or they will simply leave on their own.
Once Karlsson and/or Stone leave simply because they want to play elsewhere the boycott you propose will happen naturally. I suspect it is happening on a small scale already. I no longer go to games, nor do my friends who all had partial seasons tickets.
The next few months will be critical. Imagine the worst (and possible) case of losing during the next two years via trade or free agency many of Karlsson, Stone, Hoffman, Duchene, Pageau, Smith, Dzingle while keeping Phaneuf and Ryan. You won't have to organize a boycott.