It's a good point, there are several different needs we're trying to satisfy, and I'm not sure that we have right now is the best compromise.
You are right, sometimes you replies to someone else are not answers - they could be comments, clarifications or even actual questions.
But sometimes they are - I could post a follow-up reply on your answer to a question, adding a detail which you omitted from your answer.
The opposite is also true - you could post a reply to someone's question which in fact is not an answer, but a comment or a request for clarification.
So, when is it an answer and when not?