Well, “concern” maybe a strong word. I see this type of feeling as an indicator that shows you that you might need to look into the matter.
See if you can improve your current process. Or maybe share responsibilities with a colleague. Or if there is no other option, move on.