There are screws buried in the carpet - you have to find them. There is one right in front of your fingers, but since the piece it screws into is broken off, you don't need to deal with that one for now. There are two screws at the far right end of the carpeted section - about an inch or two in from the corners (near the storage step). Then I believe there is at least one nut holding the vinyl portion of the panel on (the carpet and vinyl comes off as all one big awkward piece/panel). You have to put your hand up through the access holes in the plastic piece above the cupholders and feel for the bolt/nut. It's been a couple years since I removed mine, so I can't remember exactly.
If you get those screws and nuts off and it's still not coming off, hunt for another screw buried in the carpet and/or another nut.