I was in a similar situation and got different opinions from surgeon and oncologist. Ultimately, you and your partner will have to make the choice that you are most comfortable with, as there is no way of knowing the 'correct' decision in advance.
It's important that you think through the consequences and that you really understand the choice you are making. In a way, you have to thing about which 'wrong' choice is worse. ie. Is 'over-treatment' (ending up with permanent colostomy that was not really required) worse that 'under-treatment' (experiencing recurrence that would not have occurred if surgery had been done) ? The truth is you will never know for sure if you were overtreated or undertreated anyway, it's more about probabilities and how you would live with the thoughts in your head.
It's a question of risk and the surgeon/oncologist can't really assess that any better than you. You have access to the same research as they do. For example: http://gicasym.org/watchful-waiting-ins ... al-cancers
You should also try and get your head around what living with a colostomy would really be like and not make a decision based on fear. There are many videos on YouTube which help to visualise what it's like and there is a lot of information from people on here who have adapted to life with a bag with minimal disruption.
However, if you are seriously considering 'watch and wait', then I would encourage you to push for as much diagnostic information as you can get before you commit to it. I had MRI, PET and flexible sigmoidoscopy and the fact that all three came back clear was what gave me the confidence to avoid surgery. I'm not sure of costs etc and you need to get in quick as the surgeons typically don't like waiting for more than 8-12 weeks after radiotherapy, but there's no harm asking. It's a stressful time and I feel for you, but the more information you have, the better.