boat plug trouble

I normally remove my boat plug after a trip. It appears to be a brass, pipe threaded plug. The plug threads into a square metal fitting that is attached to the hull with four small screws. Last time out that metal fitting started to move and one of the screws fell out! I'm not sure what to do about it. Can someone describe to me what is going on and how I should proceed to fix it? Is it a mistake to remove the plug after each trip? I like to keep the bilge dry. On my skiff, I had a rubber plug and there wasn't a lot of torque on the thru hull fitting when I removed it. Seems like a different deal here.