A systematic study to characterize fine-mesh PMTs in high magnetic fields