Can a mortgage lender charge me a credit report fee when I didn't take the loan (no fault of my own)?

I applied and was accepted for a mortgage, but the property fell through after the inspection revealed significant defects that the seller was unwilling to fix. At this point, I'll probably be renting and not purchasing a house. I got a letter from my would-be lender stating that I owe them a small fee for pulling my credit report (which they did, but only to approve the loan. They pull it again before the closing of the loan). It was never discussed that if I didn't buy the house I would still owe the lender money. I would understand if it was an appraisal fee that cost them money but for this (which actually costs them nothing) I don't feel this is appropriate.