I personally wouldn't. The older studies all said 6 weeks but the newer studies are showing that a time frame between 8-10 weeks out is more optimal.
Allow the chemoradiation to continue to work. Every surgeon I spoke to (5 of them) indicated that radiation continues to work long after the last treatment. If you go through the treatments in order to accomplish tumor shrinkage and nodal field sterilization, then why not allow the treatments to continue to function?
I find it odd that a surgeon would even consider anything sooner than 6 weeks. Are you positive that this is a board certified colorectal surgeon and not just a general surgeon who does this surgery? Verify at this site and think carefully about what type of surgeon you want doing your surgery.
http://www.abcrs.org/verify-a-physician-2/Think carefully about what you want to do. In addition to allowing the chemorad treatments to continue working, you also want your body to have adequate amount of time to heal because the next phase, surgery, isn't a walk in the park. This is your decision....not just your surgeon's. If your surgeon isn't willing to work with you....I'd find another surgeon.