Returning an item

I've a question which keeps disturbing me. Before traveling to another city where my mom lives, I bought an abaya from a local shop in Jeddah for my mother. Unfortunately that abaya was not my mom's size. After 2 weeks my husband and I went to the shop and returned it but the shopkeeper was reluctant to take it back and when he got to know that it's been weeks he refused to take it back because he didn't want to give us the cash back. When my husband persuaded him he said okay if you bring back the box in which the abaya came I'd take it back. the box was of almost no monetary value and he had tons of it in his shop and usually people don't take boxes with them they just take the abaya in a bag. We had the abaya in the bag and the abaya was never worn but he wouldn't agree. He just didn't want to take it back. My husband was very polite in the beginning but then he got angry and with some argument the shopkeeper cut 10% from the original cost and gave us the cash back very rudely. My husband said to his face that bengalis are bad people, I tried to stop my husband but he was way too angry. The shopkeeper said that I'll see you on the judgment day and take my money. It's been 2-3 months but I still feel guilty. I dunno if we were at right or wrong? Should We return the shopkeeper his money or should we let it go? Usually there no conditions on return or exchange. And abayas are sold too quickly so even if you got an abaya on order you can return and it and get another. I dunno law of shariah here. I just want to know that if the money was not haram for us because my husband forced the shopkeeper.