Is this a Safety Concern?

Hello, I recently bought some Enkei wheels for my is350 and had to deal with some tire rub until I got back from college and could roll the fenders.
Finally I am able to roll the fenders however theres some small chunks of the tread that has been taken out by my fender. I plan on buying coil overs and have fun on backroads but I don’t want to end up wrapped around a tree because of some missing rubber. I know having chunks out of your side wall is very dangerous, however these chunks are small and mainly in the tread. Should I be worried about this or is it not a big deal? The tires are Michelin Pilot Sport 4S.