Back before the gambit was destroyed Oliver was a playboy, one may say he was a bit douchey. But he did have a single scar, which as I mentioned I'm attributing this scar to a bar fight Oliver could very well have been in.
Coincidentally a bar fight, or stopping a bar fight is how Stephen Amell obtained this scar.
There is limited information about Oliver before the island. We know he's a playboy, he's been arrested a number of times and assorted other tidbits. I mean there isnt even a shirtless scene pre-island, good thing Hung existed.
Note: I am aware this exchange happened in the S2 finale, but being as it is a Hong Kong flashback and starts this adventure this is where I've started