I'm not married so I always say yes. I've asked some out too. Dated a few (only 1 was older, rest were much younger than me). U say yes bcoz it's the gentlemanly thing to do. After said dinner, u can at least say u gave it a try but think it would be better to just stay "friends" and keep it as a simple business relationship, nothing more. Declining right off the bat will almost always end it so you're kinda stuck between a rock and a hard place.
I'm more of the go with the flow, happy go lucky type of guy so this approach may not work for some.