It depends on the fish of course, but most people I've seen don't filet down over the rib bones and just cut the loin off and toss the ribs. I do that as well but I pile the ribs up and when I'm done I shave that little filet of meat off the ribs. Like little tater chips.

BKB