The host connects to everyone. Each person has their own private exchange of packets with the host; player 1 -- host, player 2 -- host...player 11 -- host, back to player 1 -- host
Because of this everyone's ping factor into your connection. If everyone is averaging 10m/s of ping then by the time you get a refresh of where everyone is at 110ms will have gone by
Now assume that you have a flawless connection of 0ms ping (which is impossible) but two people in your lobby are hitting 100ms of ping and two others 50ms of ping whilst everyone else is at an awesome 10ms
The time it takes you to communicate with the host is instant but those 4 other people are combined 300ms and the combined total of everone else is 60ms. The time it takes for the host to do his work and get back to you is 1/3 of a second.
Now you see why there's lag even when you have a low ping