I agree with Bob 100%. You'll never regret fixing it the right way and you'll sleep easier.
I found many of these areas when I did my car. My first step was always to remove any poorly adhered material and look at what's underneath to assess the scope of the repair/removal required. In the case of my car the outer layers of the repair literally peeled off (because the repairs were so poorly completed) exposing some very shoddy work underneath. It was obvious the area needed to be redone.
My suggestion - strip off any loose body filler with a putty knife. Also, strip off the paint over this area. Those two things will help you asses the extent of the issue. Depending on what you find it may be easier to patch the area rather than replace the body panel. While the thought of repairing fiberglass can seem like a lot of work, I urge you not to underestimate the effort associated with replacing a body panel. That's no easy task either. Note that I replaced several of my body panels because there was significant damage all over, it just didn't make sense to try to repair them.
Check out this blog post I did. It addresses paint removal and also shows the benefit of removing the paint to understand body damage.
http://www.corvette-restoration.com/res ... emoval.htm
Good luck! Let us know if you have questions - many of us have been there, done that!