Neither a borrower nor a lender be;
For loan oft loses both itself and friend
--Lord Polonius in Hamlet
Would you have given him (or her) the money as a gift if he had asked you for it? If yes, let it go as if it had happened that way. You don't really begrudge him the money, have your peace of mind back, and if he does return it, it is a pleasant surprise.
If not, please do bring it up. Explain to him why you want him to give it back (you'll know the reasons by now, because they are what's making you not gift it to him). If he is really your friend, he ought to understand how you feel.
You can also consider it as a cheap wall against higher future borrowings from the same friend. "Borrow me $100. No way, you still owe me $10".