Re: your second point, it is strictly synchronous, though since there are 8 GPUs per process (thus have 1 process per machine) the gradient reduction is done in 3 phases. First they are reduced within the process, then across processes/machines, and then broadcast within the processes.
I misphrased that point, agreed. It's not classic driver-driven synchronous training, as you would do in tensorflow. It's using all-reduce (not available in tensorflow yet, i think).
