Hey there Daan!
As you have mentioned in your question the surface should not be so damaged so we think that applying scratch remover on the buffing pad(buffing machine required) and working the way on the surface should be the fix to your problem.
We have the solution for your concern, recently there was a fresh post added on our web page which covers all your needs on fixing scratches on your car.
Thank you for your question, hope this helps!
DIY Auto Service