Binary files often do not have any kind of header giving directory information. Some binary files are organized; it would depend upon which program was used to write the file.
There is no one format for "binary files". "binary file" is the term used for any non-text file. There is no one format for files with ".bin" extension: that extension is used by many many different programs to produce files that might not have anything to do with the other program.
Often you need to know exactly how the program wrote the files and then copy that exact format when you read. This is typical with binary files.
There is no requirement that it be possible to work it out with a little trial and error: binary files may be compressed or may have their information packed in arbitrary ways or may be encrypted.
Binary file should be understood to be any data file that is not human readable. No standards at all, overall.
