There are a lot of things that might be going on.
Maybe you are fishing in too shallow of water with a diver that is made to dive deep
Running the boat in a strieght line might help too
Thicker line will help divers not dig so deep too, 20# mono with Hot N Tot's is what I like.
Running the divers on a dropper might also help out.