Note: the more scarifiers/depth you use the more mess you have to clean up.
Don't put all of the scarifiers down. Just put one down and cut the wheel back and forth until the pothole base is broken up. It will likely take a few passes. The space between the scarifiers is likely much wider than the pothole.
There are a lot of videos/articles that discuss that you have to break through the hard part of the pothole.
I also agree that you probably have plenty of gravel if you did before. It doesn't really disappear in most cases if had a proper base to start with.