If I open another video in VLC when one is loaded, it will be replaced, while IINA will open it in a 2nd window. I did some googling and it looks like neither of them support tabs.
By - TheTwelveYearOld
Mac app self-proclaimed grandmaster here.
The top 3 mac video players are:
1. Movist Pro
3. Infuse Pro
None support tabs. I haven't seen a decent one that does.