Unfortunately Banks has to go. Every unit has improved besides the safeties and corners.
I'm not sure I put the blame solely on Banks, but to your overarching point, I don't guess I've really thought about how simple it really is. You're right, every unit on this team has improved from the previous regime, even just from Heupel's first season...
QB-

Hooker/Milton both improved
RB-

Small/Wright both improved, and we lost Evans
WR-

Hyatt/Keyton...massive improvement
TE-

Fant/Warren both improved
OL-

Probably the biggest transformation
DL-

...some may have an argument, here
LB-

Beasley/Mitchell both improved
CB-

S-
It's easy to see where the problems lie on paper. Neither of those backend units have improved a lick. And talent can't be an excuse because we didn't have a ton of talent at any of those other positions, either, yet they found a way to make passable units out of all of them but the secondary.
So the blame comes down to two ideas. Is Martinez just not developing these guys, motivating them to want to do more film study, inspiring them to be better? Or is it the scheme that he is being tasked with trying to put these guys in by his boss (Banks)? Or is it both?
In either case, it's pretty easy to narrow the focus of our struggle down to two people. And frankly, someone has to answer for what happened, Saturday, at some point.