Analyzing Log files of very large size

[David Christensen]

On 7/11/21 5:13 AM, KK CHN wrote:
> List,
> 
> I am in a requirement to analyze large log files of sonic wall firewall
> around 50 GB. for a suspect attack.
> 
> What tools and solutions need to be deployed for handling this much large
> files and pls enlighten me with your expertise and reference materials if
> any.
> 
> All are tcp / ip communications, DNS UDP transports ..


On 7/11/21 5:31 AM, Korolev Sergey wrote:
 > Is it a plain text file?


On 7/11/21 7:13 AM, KK CHN wrote:
 > Yes, it is.


On 7/11/21 7:38 AM, Vlad Markov wrote:
 > I used to use split to break up large log files into manageable 
pieces. From there it depends on how you work. At first we used grep 
then we moved on to using perl regex to analyze logs.


If this is personal project, I could see doing it in Perl.  But, this 
sounds like the kind of problem that would benefit from concurrent 
and/or distributed programming; and Perl was not designed for such.  So, 
you will have to work harder if you want those features.


But if this project is for an employer or client, I would recommend 
starting with the commercial-off-the-shelf (COTS) log analysis tool made 
by the hardware vendor.  Train up on it.  Buy a support contract:

https://www.sonicwall.com/wp-content/uploads/2019/01/sonicwall-analyzer.pdf


David