I happen to know the answer to your question.

问题描述:

I happen to know the answer to your question.
the answer to your question.
to可以换成of 或者 for
还是说,这是一个习惯用法,所以不能换?

没错.
的确是习惯用语.
其他的还有比如
the road to
the key to
你是有心人啊