Is there any matrix of the new lua parsers and the old parser(s) they replace? Some of them are obvious, others less so. Anybody have experience switching over?
Is there any matrix of the new lua parsers and the old parser(s) they replace? Some of them are obvious, others less so. Anybody have experience switching over?
Our product engineer and Lua expert said he'd get you some more substantive detail later in the day but for now he had some quick notes:
Not all of the listed lua parsers are in Live yet, but will be.
| LUA PARSER | REPLACES FLEX OR NATIVE PARSER(S) |
| AIM_lua | aim_oscar |
| dr_watson_lua | dr_watson |
| BGP_lua | bgp |
| bittorrent_lua | bittorrent, bittorrent-id, fingerprint_bittorrent, BITTORRENT |
| botnet_lua | botnet |
| creditcard_detection_lua | creditcard_detection |
| db2_lua | db2 |
| DNP3_lua | dnp3 |
| DNS_verbose_lua | DNS, dns_verbose |
| duqu_lua | duqu |
| ein_detection_lua | ein_detection |
| ethernet_oui | MAC_Vendor |
| fingerprint_access_db_lua | fingerprint_access_db |
| fingerprint_apple_dmg_lua | fingerprint_apple_dmg |
| fingerprint_appleExec_lua | fingerprint_apple_exec |
| fingerprint_apple_ios_lua | fingerprint_apple_ios |
| fingerprint_apple_iwork_lua | fingerprint_apple_iwork |
| fingerprint_cab | fingerprint_cab_files |
| fingerprint_cad_lua | fingerprint_cad |
| fingerprint_chm_lua | fingerprint_chm, malware_chm |
| fingerprint_flash | fingerprint_swf, base64_swf |
| fingerprint_gif_lua | fingerprint_gif |
| fingerprint_java | fingerprint_jar, fingerprint_java_class |
| fingerprint_javascript_lua | fingerprint_javascript, javascript, javascript_suspicious, javascript_packers, javascript_shellcode |
| fingerprint_jpg_lua | fingerprint_jpg |
| Fingerprint_Private_Key | fingerprint_private_encryption_keys |
| fingerprint_lnk_lua | fingerprint_lnk, exploit_lnk_file |
| fingerprint_msi_lua | fingerprint_msi |
| fingerprint_mssql_lua | fingerprint_mssql |
| fingerprint_office_lua | fingerprint_office95-2003, fingerprint_office_2007, encoded_file_fingerprinting |
| fingerprint_pdf_lua | fingerprint_pdf, malware_pdf, malware_pdf_v201 |
| fingerprint_php_lua | fingerprint_php |
| fingerprint_key | fingerprint_pkcs12 |
| fingerprint_png_lua | fingerprint_png |
| fingerprint_rar_lua | fingerprint_rar |
| fingerprint_rtf_lua | fingerprint_rtf, encoded_file_fingerprinting |
| fingerprint_unix_script_lua | fingerprint_unix_script |
| fingerprint_zip | pkware |
| FIX_lua | FIX |
| Form_Data_lua | Form_Data_Elements |
| ghost | ghost_protocol |
| gnutella_lua | GNUTELLA |
| htran_lua | htran |
| HTTP_lua | HTTP, HTTP-flex, http_connect, http_error_codes, NTLMSSP, crafted_http_header, http_header, xfwdfor, ICAP_HTTP |
| HTTP_SQL_Injection | http_sql_injection |
| IMAP_lua | IMAP, IMAP-flex |
| IRC_verbose_lua | irc, irc-expanded |
| iSCSI | iscsi |
| MAIL_lua | MAIL, MAIL-flex, email-ip |
| modbus | modbus-w_port |
| NFS_lua | NFS, nfs-flex, sunrpc |
| NTLMSSP_lua | NTLMSSP |
| ntp_lua | NTP |
| OCSP_lua | OCSP |
| Packers | packers |
| phishing_lua | phishing, email_url_host |
| pwdump | encoded_hashes |
| QQ_lua | |
| RDP_lua | RDP |
| ripng_lua | ripng |
| rtmp_lua | RTMP |
| shadyrat_lua | shadyrat |
| SMB_lua | SMB, SMB-flex, SMB-ID |
| socks_lua | socks |
| SoulSeek_lua | SoulSeek |
| spectrum_lua | spectrum, spectrum11 |
| SSH_lua | SSH |
| TDS_lua | TDS |
| TLD_lua | TLD |
| TLS_lua | TLSv1, TLS-flex, TLS_id |
| TN3270E_lua | tn3270e |
| VNC | vnc-rfb |
| windows_command_shell_lua | SHELL, windows_command_shells |
| windows_executable | advanced_windows_executable, CMS_windows_executable |
| X11_lua | x11_flex |
| xor_executable_lua | xor_executable |
This is great information - as there is no Matrix that I have found and simply configuring a new LUA parser does not automatically deprecate the equivalent flex parser. Please keep this list updated as LUA parsers continue. Also - some questions regarding your list as I have gone through this exercise:
bittorrent_lua - doesn't exist
botnet_lua - doesn't exist
DNS_verbose_lua - doesn't exist
fingerprint_pdf_lua - does this also replace malware_pdf_v201?
fingerprint_php_lua - doesn't exist
fingerprint_key - doesn't exist
htran_lua - doesn't exist
IRC_verbose_lua - does this also cover irc-expanded?
As of this writing I was not able to search the above lua parsers (those indicated as doesn't exist).
Not all of the listed lua parsers are in Live yet, but will be soon. Several were posted to Live on Friday.
For fingerprint_pdf_lua: yes (malware_pdf and malware_pdf_v201 are the same thing)
For IRC_verbose_lua: yes - I missed including irc-expanded in the list
That should cover nearly all of the existing flex parsers in Live, except for some website-specific parsers that will likely be converted in a more generic form, and some that won't be converted at all (e.g., "tcp-flags"). Many of the native parsers will eventually have lua equivalents, but those will be slower in coming.
Can the source of the lua parsers be viewed directly?
It would be very helpful to reference these when creating my own parsers.
Currently they are all encrypted.
There will be some made available in some manner as unencrypted for demonstration purposes. But I don't yet know which, when, or how.
Thanks for your response.
Any word on the availability of some of these parsers unencrypted, for demonstration purposes? It would be useful for custom parser creation.
Is there any update to table? we're trying to apply the LUA parser but don't know which one to apply.
1. Can LUA parser and Flex parser be applied at same time?
2. Some Flex parsers don't have replacement LUA parser, like OS and browser parser, when will it be available?
Thank you.
Sorry for the late reply.
1. Yes, they can be enabled at the same time. But if a flex and a lua parser that parses the same thing are both enabled, then the decoder will be doing more work and registering duplicate meta.
2. Those two parsers really just tried to match bits of user-agent headers and make inferences based on that. Instead HTTP_lua simply registers the entire user-agent header.
if like this, can the system auto detect if both flex and lua pasers enabled for the same thing, it will disable one of them?
That would be nice, but it doesn't do that currently.
Now that HTTP_lua is available and replaces NTLMSSP does it also replace NTLMSSP_lua?
Correct, if you have HTTP_lua enabled you shouldn't need NTLMSSP_lua as well.
- Bill
will it become a KB link？
Great information! Is there an easy way to remove the old flex parsers, or is it a manual/scripted process?
> Great information! Is there an easy way to remove the old flex parsers, or is
> it a manual/scripted process?
Its a manual process for now unfortunately.
In regards to the packers lua parser, you indicate it replaces the existing packers parser. Does this include all of the malware_packers_X parsers and javascript_packers?
> In regards to the packers lua parser, you indicate it replaces the existing
> packers parser. Does this include all of the malware_packers_X parsers
> and javascript_packers?
The 'packers' flex parser file actually contains all of the individual malware_packers_X parsers. The 'packers' lua parser replaces all of them.
The 'javascript' flex parser file contains 'javascript_suspicious', 'javascript_packers', and 'javascript_shellcode' parsers. The 'fingerprint_javascript_lua' parser replaces all of them.
> In regards to the packers lua parser, you indicate it replaces the existing
> packers parser. Does this include all of the malware_packers_X parsers
> and javascript_packers?
The 'packers' flex parser file actually contains all of the individual malware_packers_X parsers. The 'packers' lua parser replaces all of them.
The 'javascript' flex parser file contains 'javascript_suspicious', 'javascript_packers', and 'javascript_shellcode' parsers. The 'fingerprint_javascript_lua' parser replaces all of them.