plus my filter has some tiny scratches which I think came from wiping the lens when there were still some dust particles
That's one good reason for using a filter .... better it gets scratched than your front lens element. Scratches from cleaning usually occcur if the lens is wiped while dry or something other than a proper lens cleaning tissue is used.
I suggest you set your camera up on a tripod and do some A-B test shots of various scenes with and without the filter on. Then compare them and see if you can notice a difference. I keep filters on all my lenses. Sometimes when shooting I'll leave them on and sometimes remove them, depending on how critical the shot is.