The Wade family did indeed have many branches of distant relatives. These distant relatives were spread all over the country and even all around the world. Although they also had the last name Wade, compared to the Wade family in Eastcliff, their economic situation was indeed far worse off.
Therefore, if he was a member of the Wade family in Eastcliff, Tom certainly would not be able to afford to mess with them. However, if Charlie was just a distant relative of the Wade family, Tom really did