What is it with me and bus drivers?

So today was going sooooo well. I finished my week at my new job and am feeling pretty good about it. I was taking the trek home when it happened. First I walked to the kintetsu station. Then I grabbed a train there and get off at Midoribashi to catch a bus from there. Unfortunately, the bus only goes to Imazatou so I had to catch another bus there. So this is when it happened. I forgot to get the little transfer ticket from bus to bus. They are free, so it wasn't a matter of forgetting to pay, or not paying enough or anything like that - I had already paid for the whole trip by the time I'd exited the subway - you can buy your subway to bus transfer there which costs you a whole 100 yen. Once you get on a bus, you can transfer from one to another almost indefinitely without paying any extra as long as you push the little button that gives you a transfer. So when I got off my first bus, I dropped the little slip into the money box - forgetting to get the little transfer ticket in the process. Big mistake. I realised it as I got on the second bus, but by then it was too late. So I decided to do the right thing and explain to the bus driver what had happened. Usually when this sort of thing happens in Osaka (at the bus or train station), they tell you to please be more careful next time or something like that and you apologize and the matter is finished. Except that this guy turned all Hitler on my ass and told me that I had to pay another 200 yen (the price without subway transfer or anything) when I got off. My whole trip is really expensive - 540 one way - so I wasn't going to stick any more money into their little box for a small paperwork error. I got the feeling that perhaps I was just the icing on the top of a very bad day or something for him, as I felt more than a little animosity headed in my direction. Or I could've run across the bus driver who hates foreigners. Really it could have been anything. I told him that perhaps I could speak to his manager? They must have little walkie talkies or something on those buses. Nope. He won't even consider it as 'the manager' will say the same. Well, we will never know if you don't contact him will we? He didn't contact him of course. So, I told him I wasn't going to pay because I had already paid 540 yen for my trip. I felt kinda bad for him cause he was obviously pissed about it, but I wasn't going to pay just to make him feel better. The bad bit is, that it's my regular route and I don't know how the guy doesn't recognize me in the first place. I must have taken his bus before and I'm seriously the only white person for miles who lives in our area AND takes the bus. And I will continue having to take that route to get anywhere really! I just hope he doesn't hold it against me. *sigh*