The answer is cut and dried, YES, pay it off. The interest you pay on your mortgage is higher than the interest you can earn. You can toss that money in savings each month. I paid mine off several years ago and love having the money going into savings. Set up a monthly transfer of the mortgage payment to go into the savings.
Let no one fool you, being debt free is the ultimate of freedom ;)
