They better represent if a participant has video available or not. There are cases when even a participant in the last N set would not have video because it disconnected momentarily, for example.