There's not much you can do if its his server and he's the only admin.
If its part of a group of admins/clan then you should try appealing to them.
My advice - find a different server to play on together with your friends.
Unless its the only server that gives you a decent ping...:)...