Heater Unit: Replacing Flap Foam

There are some detailed posts on heater unit refurbishment - I am impressed by the quality of the work!

My dashboard has been removed for refinishing. My question: Is it possible to replace the deteriorated foam on the heater flap without removing the heater itself? I am ambivalent about whether the heater works effectively, since I plan to drive the car only in warm weather (I can’t remember the last time I used the heater on my other classic cars). I thought I could at least take care of the foam while the dashboard is off. In addition to the screws on the heater side, I noticed that the fiberglass base is riveted to the side of the heater. If the rivets were drilled out, could the side be removed?

Thanks, Kevin