First and foremost, Charlie hadn't saved Tanya to do Yoshitaka a favor, so he shouldn't think too highly of himself.
Secondly, Charlie didn't care how big a deal he was in Japan.
To Charlie, he was nothing.
Yoshitaka certainly felt embarrassed as well, as he could see that Charlie thought nothing of him when he didn't even know who Charlie really was.
Until now, if anyone ever did Yoshitaka a favor, he just had to tell them he would remember it, and they would be so overjoyed as if they had been granted a favor.
That definitely wasn't the case with Charlie, as he was showing his disdain for Yoshitaka in every way.
Moreover, he had indeed screwed up big time in front of Charlie and Kathleen when Peter and his gang duped him.
Smiling in embarrassment, he nodded, "Yes, you're right, Mr. Wade. I will do my best to correct that and refresh my memory…"
Charlie simply ignored him and turned to Tanya, "Putting aside the fact that my wife and I flew hundreds of miles to watch your con