This is related to the
MOOV atom that contains all the metadata and information about the frame locations other important information the browser needs to know about the video. For some reason, all other major browsers load it with no issues but Safari prefers to wait until the whole resource has downloaded in order to decide what to do with it.
mp4 files, downloading FFMPEG and running the following command solved my issue:
./ffmpeg -y -i source.mp4 -movflags faststart dest.mp4
What it does is that it takes the
MOOV atom from the end and adds it right at the start.