Why No Big Fish When Plenty of Bait Fishes Around?

Yesterday I went fishing in a pier inside a harbor that is facing Long
Island Sound in New York State. I could see many bait fishes (called
spears) swimming around, and I also caught bait fishes as live baits.
But all I could caught were small snapper and small sea-robins --
nothing big. People who used bigger baits caught slightly bigger
fishes, like someone used a 5 inches snapper to catch a 10-inches blue
fish. But no one in the pier caught anything big.

I thought if there were plenty of small bait fishes and "not-so-small"
fishes that fed on the bait fish, I should see big fishes that fed on
the "not-so-small" fishes - a complete food chain, right? What kept
the big fish away?

The boat traffic in the harbor was quite light; I could count the
number of boats passed by in that morning in one hand; I doubt that
was keeping the big fish away.

The temperature was like 76 degree and sunny just like the two
previous days -- meaning that the weather was stable.

The depth around the pier was like 5-ft to 10-ft. We were casting to
the deep water near the channel and reeling in to the relatively
shallow water near the pier. I were fishing from low tide to mid-high
tide. I am not sure if this made any different though. I mentioned
this just in case this might make a difference.

Should I catch a 8-inches snapper and use it as bait? Big bait, big
fish, right?

Any idea? Thanks.

Jay Chan