I think ultimately, it's personal preference. Some people don't want to tempt fate by having the port removed. Others feel removing the port can be symbolic of turning the page and moving on. It's probably good to factor in the probability of future need just to avoid the hassle of having to have another one installed, but if you have it taken out and then find you need it, you can get another one (they usually put it on the other side).
Generally, after the port's been in a while, you shouldn't feel it and it shouldn't limit activity. It's very low risk for clotting or other problems when you keep it long terms so that doesn't have to be a worry. If you keep it and never use it, bonus.
You have to decide what works best for you. There's no right or wrong to it. Whatever you decide, it's safe, manageable and reversible.
Personally, if I had one, I'd keep it forever. I'm young, healthy, have great veins and always make sure to be well hydrated before any sort of needle stick yet I've still had major drama with blood draws, IVs and dye injection. It's no joking matter, but I have had moments where I wish I had port.