Is my boss allowed to take my phone charger away?

My boss is constantly threatening to take our phone chargers away if he sees us using them. I understand that if it’s against the rules then so be it, no charging, but is he allowed to physically take the charger if we do charge?