I’m guessing you’re talking about URLs in menus and the like? This wouldn’t break links on your Actuses index page, for example. So, the number of links you need to add 'plugin' => false on should be fairly small.
If you did, then all of your views in your plugin would need to explicitly specify the plugin for links into themselves, which you don’t need to do right now. For example, your Tasks/index.ctp doesn’t currently need to mention the plugin in order to link to tasks/view.