Usually ports aren't removed if a patient is still undergoing treatment (or might resume treatment in the future) unless the port is giving them problems (soreness, infection, hard to access, etc.). I think some oncs like to recommend port removal in stable patients because they think it will make the patient feel more "normal". It also save the time and effort of coming in to flush the port every 4-6 weeks to maintain its efficacy.
Your DH is doing absolutely fantastic! Stability for 4 years is awesome! Clearly his immune system is keeping things at bay.
So if there's no sign of chemo in the near future, it might be worth it psychologically to have it removed, realizing (eyes wide open) there is a possibility he might need to have another one put in some day. My vote is to free him of the port!
Have you looked into immunotherapy? He seems like a good candidate if there's a trial that involves lymph node spread. Or has your onc considered a continuous low-dose Xeloda (used in a metronomic way)?
Also, make sure he keeps ups his bloodwork even though he's moving to yearly scans. Try to include LDH and AFP blood tests into the regular mix since they work well for tracking nodes.
Hope this helps!
Juliej