it might not be that he dosnt like spending time with you when he is with his m8s or dont want you to be seen with him it could be that his m8s tease him about you saying things that could upset him cos he really likes you.
or could even be he dosnt want u to see him with them as he wants to show you that he isnt like his m8s that are always causing trouble etc (this might not be the case here, but is a common thing)

hope this helps in some way