Those little bugs are a nightmare for everyone, not just you. They show up no matter how good you are with plants. Try blasting them with some soapy water, it usually helps.
Is the panic more about the bugs or feeling like you failed? Gardening has a way of making small problems feel huge sometimes. Maybe it's not just about skill.