Update for binary signature of MP3?

SafetyCar · 2008-11-12 15:46

Today I found some files without extension, and I tried to find the extension with renamer.

As I couldn't do it, I searched a little on it, and inside the file (with notepad) I found something like: ...LAME3.97 (beta)...

On google I got this URL:
http://lame.sourceforge.net/

This encoding is used in various web players (MySpace for example).

I don't know how the binary signatures work, but maybe it can be added as mp3.

Andrew · 2008-11-16 15:59

Actually, I don't think this is a job that Renamer should do at all... primarily because it is not a simple task that will stop at just MP3s. There are literally millions of file formats out there (many with overlapping extensions) with new ones being created daily. Even in case of MP3s, LAME is just one of many possible MP3 encoders out there, so that's not a valid test for all MP3s.

Frankly, I think what you're looking for is something like a file type identifier, and what do you know, there are such tools out there!

http://mark0.net/soft-trid-e.html
http://mark0.net/onlinetrid.aspx (Online)

http://www.optimasc.com/products/fileid/index.html

BTW, can someone tell me which binary signatures Renamer supports internally? If this feature is indeed to be supported and be made actually useful, then it should ideally support loading something like an XML file at runtime, which will contain signature data. The file can then be extended by users and the database shared with all.

Sample format for xml signature file:

<renamer_file_db>
<file>
<ext>JFIF JPE JPEG JPG</ext>
<sig>FF D8 FF E0 xx xx 4A 46 49 46 00</sig>
</file>
<file>
<ext>PNG</ext>
<sig>89 50 4E 47 0D 0A 1A 0A</sig>
</file>
<file>
<ext>PDF</ext>
<sig>25 50 44 46</sig>
</file>
</renamer_file_db>

Note: xx is an ignored byte. Also, one might need <start_sig> and <end_sig> tags for some file types, instead of just <sig>

P.S. All this is just an idea of what could be done. I'm certainly not advocating to Denis that it be worked on, but if he ever feels like it, this might be one of the approaches he can think of.

SafetyCar · 2008-11-16 18:36

The file identifier help (thanks), but the second suggestion would very interesting.

den4b · 2008-11-18 00:52

Nice suggestion there Andrew

But I thought that very few people would be messing around with such settings. And for those who would - it might be better and easier if they just tell the signature of some new file format so I can add it to ReNamer. For these and other performance reason I decided to have that signature database as an internal collection.

I thought it would be of interest to you and perhaps other people to know what this database contains.

The format is: first bytes in hex, extension, description. "?" is for any hex digit.

00000002FFFFFFFFFFFFFFFF0000000000000000000000, MAC, MacPaint Bitmap Graphic
24464C3240282329205350535320444154412046494C45, SAV, SPSS Data
4C0000000114020000000000C000000000000046, LNK, Windows Shortcut
000100005374616E64617264204A6574204442, MDB, Microsoft Access Database
0006156100000002000004D200001000, DB, Netscape Navigator (v4) database file
52494646????????43444441666D7420, CDA, Compact Disc Digital Audio (CD-DA)
52494646????????415649204C495354, AVI, Windows Audio Video Interleave
52494646????????524D494464617461, RMI, Windows Musical Instrument Digital Interface
52494646????????57415645666D7420, WAV, Waveform Audio
89504E470D0A1A0A0000000D49484452, PNG, Portable Network Graphics
50532D58204558450000000000000000, EXE, Playstation Executable
3026B2758E66CF11A6D900AA0062CE6C, WMA|WMV|ASF, Windows Media File
??BE000000AB0000000000000000, WRI, Microsoft Write Document
0000000020000000FFFF0000FFFF, RES, Resource File
0000020006040600080000000000, WK1, 1-2-3 Spreadsheet
38425053000100000000000000, PSD, Photoshop Image
414F4C204665656462616720, BAG, AOL Instant Messenger Buddy List
52494646????????41434F4E, ANI, Windows Animated Cursor
110000005343410F000000, PF, Windows Prefetch
4D54686400000006000100, MID, Musical Instrument Digital Interface (MIDI)
504B0304140008000800, JAR, Java Archive
2E524D460000001200, RM, RealMedia Streaming Media
67696D702078636620, GZ, GIMP Image
D0CF11E0A1B11AE1, DOC|PPT|XLS, Microsoft Office Document
5245474544495434, REG, Windows Registry Data 
300000004C664C65, EVT, Windows NT/2000 Event Viewer Log
4D53434600000000, CAB, Microsoft Cabinet File
????????6D6F6F76, MOV, QuickTime Movie
FF4B455942202020, SYS, Keyboard Driver
255044462D312E, PDF, Adobe Portable Document Format
526172211A0700, RAR, WinRAR Compressed Archive
000001BA210001, MPG, MPEG 1 System Stream
377ABCAF271C, 7Z, 7-Zip Compressed Archive
AC9EBD8F0000, QDF, Quicken Data
D7CDC69A0000, WMF, Windows Metafile
010009000003, WMF, Windows 3.x Metafile
4A4152435300, JAR, JARCS Compressed Archive
7B5C727466, RTF, Rich Text Format File
000001BA44, MPG, ProgDVBR MPEG2 Video
464F524D00, AIFF, Audio Interchange File
DF0000?F, DCU, Delphi Compiled Unit 
00000100, ICO, Windows Icon
01000000, EMF, Extended (Enhanced) Windows Metafile Format
CFAD12FE, DBX, Outlook Express E-mail Folder 
47494638, GIF, Graphic Interchange Format
49492A00, TIF, Tagged Image Format
4D4D002A, TIF, Tagged Image Format
00000200, CUR, Windows Cursor
C5D0D3C6, EPS, Encapsulated PostScript
3F5F0300, HLP, Windows Help File
49536328, CAB, Install Shield v5.x or 6.x Compressed File
504B0304, ZIP, ZIP Compressed Archive
E3828596, PWL, Windows Password List
EDABEEDB, RPM, RedHat Package Manager
50533244, SYS, PlayStation 2 Icon
FF575043, WPD, WordPerfect Document
464C5601, FLV, Flash Video
465753, SWF, Macromedia Flash Format
435753, SWF, Shockwave Flash (v5+)
FFD8FF, JPG, JPEG/JIFF Image
1F8B08, GZ, Gzip Compressed Archive
1F9D90, Z, UNIX Compressed Archive
494433, MP3, MP3 Audio
FFFB, MP3, MP3 Audio
FFFA, MP3, MP3 Audio
4D5A, EXE|COM|DLL|DRV|SYS|OCX|CPL|SCR|VXD, Windows Executable
424D, BMP, Windows OS/2 Bitmap Graphics
9501, SKR, PGP Private Keyring
9901, PKR, PGP Public Keyring)

Note: Signatures table is from ReNamer Beta / 12 Nov 2008.

SafetyCar · 2008-11-18 16:20

FFF3A260FF000001A400000000000003480000000055555555555555555555555555555555555555555555555555555555555555555555555555554C414D45332E393720286265746129, MP3, LAME-MP3
0000001C667479706D703432, MP4, MPEG-4 Video (MP4)
0000001C667479706D4D3456, M4V, MPEG-4 Video (M4V)
0000001C6674797033677034, 3GP, MPEG-4 Video (3GP)

This are from some files I had in my computer

I was searching on the wikipedia somethings about de MP4 and I found tha are related to M4V and 3GP extensions.
As I had sme video with this extensions I analized them and internally they had "...M4V" and "...3GP" so I respected it, but may be could put them as M4V|MP4 or 3GP|MP4, as you prefer...
http://en.wikipedia.org/wiki/MP4

Andrew · 2008-11-18 16:39

Actually, come to think of it this would indeed be an awesome feature if implemented, for I have often come across badly named files on the net (especially archives). One more instance where something like this would really help is when a hard disk fails and recovery is carried out. I had to do this once, and believe me, the data retrieval (to whatever extent possible) was simply nothing compared to the ages it took me to figure out what type of file Fxxxxxxx.REC was (x = digit)! I had a directory full of thousands of these .REC files, and how I wish now I had Renamer back then and it had an extensive file ID database!

Again, I'm not sure how much work will be involved in chaging the app. from the format used now to an external XML-based one. The existing method is very interesting and the way file extensions, signature bytes etc. are specified seems quite similar logically to the format I proposed, but whether it can be converted easily is another matter that only Denis will know for sure.

The only reason I feel it should be an external DB is obviously the ease of signature addition by users (less work for Denis!), the extensibility of it, plus the fact that adding thousands of signatures internally will only serve to make the app unnecessarily bloated. Those who don't want the feature can simply ignore the XML file and the app will continue to work in all other cases.

Anyway, no recommendations from my side as of now to add more internal sigs, Denis. The XML suggestion is certainly not a priority, but maybe you can check it out in the future sometime when you've nothing else to do and are feeling slightly bored.

- Andrew

P.S. I just thought of something else. How about a PascalScript function like the following:

String[] readBytes(String filePathAndName, long numBytes, int pos)

where numBytes could be -ve or +ve, and pos could be BEG=0, MID=1 or END=2 (with proper error checking for file boundaries of course)?

We could then have a generic PascalScript that would read binary file sigs from a text file (that would be in a specific format) and compare them to the String array returned by the above function. It would then change the file ext accordingly.

Is something like this possible and would it be easier to implement compared to the XML file suggestion (though I still think that would be best)?

P.P.S. Denis, just came across this very interesting page on one of the sites I mentioned above while replying to SafetyCar - http://mark0.net/code-tridlib-e.html

From the page about TrIDLib - "This standard Win32 DLL library, based on TrID's core engine, make adding file identification / file recognition capabilities to any kind of application a very easy & quick process." It's free and best of all, it also contains sample Delphi source!

What do you think about this 3rd approach which IMO might require less work then the first two? Plus the main job of identification will be done by using established code from a project that is dedicated to this problem, and so obviously does a far better job of identifying a large database (and growing) of file types compared to Renamer. Why reinvent the wheel when the work's already done and available for free?

Last edited by Andrew (2008-11-18 17:04)

den4b · 2008-12-16 18:28

Andrew, thanks for the great discovery - the TrID library!

I've wrote a script for integrating that library into ReNamer: ReNamer and TrID library for detecting file extension

Andrew · 2008-12-17 16:53

Fantastic, Denis! I was thinking that the TrID .DLL would have to be linked to the ReNamer .EXE or something, but the script approach is far easier to implement I guess, customisable, further extendable and also optional for those who don't care about such stuff. That's a great solution to the issue.

eR@SeR · 2009-02-02 13:14

Here are mine 'magic bytes' for, IMO, 'essential' extensions.
More or less names and first bytes in hex of extensions are taken from File extension defs.
Extensions (files) that I have were checked with Hex Editor XVI32 to convince myself if they detected properly, but some of their first bytes in hex are not included in Mark0's database so I named them by myself.
I tried to check/scan mine existing files as many as I can, but it could happen that I've made a mistake so if you notice anything wrong let me know.

P.S. I have problem to determine genuine bytes of M4A and MP4 cause they are too similar

64383A616E6E6F756E6365????3A687474703A2F,    TORRENT, BitTorrent Link
5468656D652E786D6C0000000000000000,    THM, Theme For Mobile Phone
000100005374616E6461726420414345,    ACCDB, Microsoft Access 2007 Database
3026B2758E66CF11A6D900AA0062CE6C,    Advanced Streaming Format
41565020416E7469766972616C204461,    AVC, Kaspersky Anti-virus data base
5B436C6F6E6543445D0D0A5665727369,    CCD, Elaborate Bytes/SlySoft CloneCD CDImage (subchannel-data)
51420105000000000000000000000000,    INI, Microsoft QBX local INI
4D454449412044455343524950544F52,    MDS, Media Descriptor
1A45DFA3934282886D6174726F736B61,    MKV/MKA, Matroska stream
00000000000000000000000000000000,    NRG, Nero BurningROM CDImage
FFFE570069006E0064006F0077007300,    REG, Windows Registry Data (Ver. 5.0 - UTF16)
57696E646F7773205265676973747279,    REG, Windows Registry Data (Ver. 5.0)
FFFFFFFFFFFFFFFFFFFFFFFF41010100,    SUB, Elaborate Bytes/SlySoft CloneCD CDImage (subchannel-data)
FFD8FFE107FC45786966000049492A00,    THM, Sony Ericsson Theme
42004500470049004E003A0056004D00,    VMG, Saved SMS (Unicode) (vMessage)
00FFFFFFFFFFFFFFFFFFFF00000200,    ISO,  ISO CD Image
4F67675300020000000000000000,    OGG, OGG Vorbis Audio
5B52756C65305D0D0A4E616D653D,    RNP, ReNamer Preset (you missed your own one :D)
5B626F6F74206C6F616465725D,    INI, Windows Boot Loarder information
??????10123A001019040010,    SIS, Nokia Phone Program
000000??66747970??67703,    3G2|3GP, 3GPP multimedia audio/video
00FFFFFFFFFFFFFFFFFFFF,    IMG, Elaborate Bytes/SlySoft CloneCD CDImage (subchannel-data)
5B6175746F72756E5D0D0A,    INF, Autorun File
5B706C61796C6973745D,    PLS, WinAmp/SHOUTcast PlayList
234558544D33550D0A,    M3U, Winamp Playlist File
D0CF11E0A1B11AE100,    MSI, Microsoft Windows Installer
2E524D460000001200,    RA, Real Audio
636C6F????20616C,    M, MATLAB program
7A1A201000000000,    SIS, Nokia Phone Program (Cracked)
414143000100,    AAC, Astrid/Quartex AAC encoded audio
2321414D520A,    AMR, Adaptive Multi-Rate
424547494E3A,    VCF, vCard - Business Card
2E7261FD00,    RA, Real Audio Sound
FFFE31000D,    SRT, SubRip subtitles (Unicode)
0001000000,    TTF, TrueType Font
4D414320,    APE, Monkey's Audio
49545346,    CHM, Windows HELP File
52494646,    DAT, VideoCD MPEG
3F5F0300,    GID, GID Help index
49735A21,    ISZ, ISO Zipped format
4B4C7377,    KEY, Kaspersky Lab Key file
EFBBBF23,    M3U8,Winamp Playlist File (Unicode)
4D4D4D44,    MMF, Yamaha SMAF Synthetic music Mobile Application Format
4D502B07,    MPC, Musepack encoded audio (SV7.0)
93B20000,    LNG, SourceEdit Language Definition
5B74693A,    LRC, Lyric file (with ID tags)
4F676753,    OGG, OGG stream
52494646,    TRD, TrID defs package
000001BA,    VOB, VOB video files
484D3357,    W3M, WarCraft III map
504B0304,    WIDGET, Konfabulator widget
77646774,    WIDGET, Yahoo! Widget
445644,    IFO, DVD info file
4D502B,    MPC, Musepack encoded audio (generic)
7B??3,    SUB, Subtitle
0D0A,    M, MATLAB program
310D,    SRT, SubRip subtitles
78,    DMG, Disk Image (Macintosh)
5B,    INI, Generic INI configuration
25,    M, MATLAB program

Note: Append PPS and PUB to DOC|PPT|XLS, Microsoft Office Documents.

den4b · 2009-04-16 14:26

I have added a bunch of signatures (many are from your eR@SeR's list):

482B424544562050726F6475637473204C6963656E7365204B65792046696C650000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001A, KEY, Avira Product Key
FFFE570069006E0064006F0077007300200052006500670069007300740072007900200045006400690074006F0072002000560065007200730069006F006E00200035002E0030003000, REG, Registry Data File 5.00
41565020416E7469766972616C2044617461626173652E202863294B6173706572736B79204C616220313939372D32, AVC, Kaspersky Anti-Virus Database
D0CF11E0A1B11AE1000000000000000000000000000000003E000300FEFF090006, MSI, Windows Installer File
1A45DFA3934282886D6174726F736B61428781014285810118538067, MKV, Matroska Video Stream
5B436C6F6E6543445D0D0A56657273696F6E3D, CCD, CloneCD Control File
000100005374616E6461726420414345204442, ACCDB, Access 2007 Database File
4D454449412044455343524950544F52, MDS, Media Descriptor CD Image File
4F67675300020000000000000000, OGG, Ogg Vorbis Audio
??????10123A001019040010, SIS, Symbian OS Installer File
EFBBBF234558544D33550D0A, M3U8, MP3 Playlist (UTF-8)
5B6175746F72756E5D0D0A, INF, Autorun File
64383A616E6E6F756E6365, TORRENT, BitTorrent Metainfo File
424547494E3A564D5347, VMG, Nokia Text Message
5B706C61796C6973745D, PLS, Winamp Playlist
234558544D33550D0A, M3U, MP3 Playlist
52454745444954, REG, Registry Data File
424547494E3A, VCF, vCard File
2E7261FD00, RA, RealMedia Streaming Media
49735A21, ISZ, UltraISO ISO Zipped Format
4B4C7377, KEY, Kaspersky Anti-Virus Key
4D502B07, MPC, Musepack Audio
93B20000, LNG, SourceEdit Language Definition
000001BA, VOB, DVD Video Movie File

It would be great if someone could check these out with real files...

den4b Forum

#1 2008-11-12 15:46

Update for binary signature of MP3?

#2 2008-11-16 15:59

Re: Update for binary signature of MP3?

#3 2008-11-16 18:36

Re: Update for binary signature of MP3?

#4 2008-11-18 00:52

Re: Update for binary signature of MP3?

#5 2008-11-18 16:20

Re: Update for binary signature of MP3?

#6 2008-11-18 16:39

Re: Update for binary signature of MP3?

#7 2008-12-16 18:28

Re: Update for binary signature of MP3?

#8 2008-12-17 16:53

Re: Update for binary signature of MP3?

#9 2009-02-02 13:14

Re: Update for binary signature of MP3?

#10 2009-04-16 14:26

Re: Update for binary signature of MP3?

Board footer