AnsweredAssumed Answered

Unable to read custom application log using Winrm

Question asked by David Waugh on Oct 22, 2018
Latest reply on Oct 31, 2018 by David Waugh

Like • Show 0 Likes0
Comment • 14

Hello

We have a system that writes to the application log.

I can collect other application log messages, but the particular messages written by this application are not rendered correctly.

What I can see is:

%NICWIN-4-Application_1_BLAH: Application,rn=191245 cid= eid=,Sun Oct 21 23:15:17 2018,1,BLAH,,Classic,mycomputer.mydomain.com,0,,

Here you can see that the event is truncated.

Now other application logs are displayed correctly so the collecting user is a member of the event log readers group.

If I look in the XML of the event in the Microsoft Event Viewer then it has the following fields

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

- <System>

<Channel>Application</Channel>

<Computer>mycomputer.mydoamin.com</Computer>

</System>

- <EventData>

</EventData>

</Event>

At the moment I just want to be able to see the fill message. Then I will write a parser to pick out the meta data.

Any ideas why the full message is not being collected?

This is a Windows 2016 Server, but other application log messages are being received fine.

I checked the normal reasons for truncated messages, but have hit a blank on this one. Can anyone help?

On the VLC I have put the Event Source into Verbose mode and can see the message come in correctly when I look in the VLC Logs.

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="BLAH"/>
<EventID Qualifiers="0">1</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2018-05-26T23:15:13.148221600Z"/>
<EventRecordID>123724</EventRecordID>
<Channel>Application</Channel>
<Computer>mycomputer.mydomain.com</Computer>
<Security/>
</System>
<EventData>
<Data>CEF:0|BLAH|Field1|Version|OtherStuff|Process succeeded|Low|Lot of Other CEF Fields</Data>
</EventData>
<RenderingInfo Culture="en-US">
<Message/>
<Level>Information</Level>
<Task/>
<Opcode>Info</Opcode>
<Channel/>
<Provider/>
<Keywords>
<Keyword>Classic</Keyword>
</Keywords>
</RenderingInfo>
</Event>

David Waugh Oct 30, 2018 8:47 AM

Correct Answer

I managed to bodge a work around for this.

1) Set up the event source for winrm collection as normal, and confirm you are getting events in (even if they are truncated)

2) Put the event source into debug verbose mode.

3)This causes all events to be written to the /var/log/netwitness/logcollector/ log files

4) Change the size of the Log Files to 10MB. This causes them to rollover more frequently

5) Grab events out of log files and re inject them into the system

See the reply in context

No one else had this question

Outcomes

David Waugh Oct 22, 2018 5:47 AM

Looking in the data of the messages I can see that they contain \r\n characters. Could this be the issue?

Actions

David Waugh Oct 22, 2018 8:14 AM

I found an event which was not being processed that did not contain \r or \n. The only special characters might be I (pipe) symbol.

Actions

David Waugh Oct 23, 2018 5:10 AM

From what i have seen so far, it seems that the log message is being successfully collected, but just not sent completely onwards.

We have event source -> VLC -> Local Collector ->Log Decoder

Actions

Eric Partington

Oct 23, 2018 9:29 AM

can you try to tcpdump on the vlc and use the -vvv option to capture verbose? curious to see if there is a tab or something in there that breaks the parsing that you are hitting. THere are a few options in the VLC that remove or replace tabs and a few other things depending on your version.

Could you also try to point this to a local LC to see if that removes the issue (maybe VLC is manipulating a character at capture)? (not sure if this is prod or testing)

Actions

David Waugh Oct 24, 2018 3:41 AM

Morning Eric

Thank you for your response.

I did a tcpdump on the logdecoder and was able to verify that the message is arriving truncated before it hits the log decoder.

I will try to get it coming directly into a LC to see if that helps. This is a production system so it is quite difficult with firewall rules etc ...

Actions

David Waugh Oct 24, 2018 5:03 AM

To get around having to raise firewall rules i used:

Log Collector in DMZ getting error with SysLog

To set my test Log Collector to a Local Log Collector.

Im reinstalling the remote log collector as a local log collector to test...

Actions

David Waugh @ David Waugh on Oct 24, 2018 5:39 AM

Hi the same result occured when it was a local collector.

Actions

Arnab Chakraverty

Oct 24, 2018 4:58 AM

Hi David,

Since the collection is through Windows Collection, the logs shown are correct. In windows Transformation the second part of the event is formed from the /RenderingInfo/Messages attribute. In your case the said attribute has No Data, thus the events looked truncated.

with regards

Arnab

Actions

David Waugh @ Arnab Chakraverty on Oct 24, 2018 5:39 AM

Hi Arnab is there a solution to this?

Actions

David Waugh @ David Waugh on Oct 29, 2018 4:27 AM

Hi Arnab Chakraverty is this likely to be possible?

Actions

Arnab Chakraverty

@ David Waugh on Oct 29, 2018 5:26 AM

Hi David,

I do not have an idea on how these said messages are getting generated and how they are being published in the channel.

Actions

David Waugh Oct 30, 2018 8:47 AM

I managed to bodge a work around for this.

1) Set up the event source for winrm collection as normal, and confirm you are getting events in (even if they are truncated)

2) Put the event source into debug verbose mode.

3)This causes all events to be written to the /var/log/netwitness/logcollector/ log files

4) Change the size of the Log Files to 10MB. This causes them to rollover more frequently

5) Grab events out of log files and re inject them into the system

Actions

Arnab Chakraverty

@ David Waugh on Oct 30, 2018 4:36 PM

Hi David,

Just being curious here, could you not directly send the CEF logs to VLC or LD 514 port from the event source itself? Why do we need a Windows Layer, as it seems to be a custom channel.

with regards

Arnab Chakraverty

Actions

David Waugh @ Arnab Chakraverty on Oct 31, 2018 5:10 AM

Hi Arnab Chakraverty that would be the ideal solution. Unfortunately the application only supports writing to the windows application log, and I don't have control over it. Sending via syslog would be so much easier!

Actions

14 Replies

David Waugh Oct 22, 2018 5:47 AM
Mark Correct
Correct Answer
Looking in the data of the messages I can see that they contain \r\n characters. Could this be the issue?
Like • Show 0 Likes0
Actions
David Waugh Oct 22, 2018 8:14 AM
Mark Correct
Correct Answer
I found an event which was not being processed that did not contain \r or \n. The only special characters might be I (pipe) symbol.
Like • Show 0 Likes0
Actions
David Waugh Oct 23, 2018 5:10 AM
Mark Correct
Correct Answer
From what i have seen so far, it seems that the log message is being successfully collected, but just not sent completely onwards.

We have event source -> VLC -> Local Collector ->Log Decoder
Like • Show 0 Likes0
Actions
Eric Partington Oct 23, 2018 9:29 AM
Mark Correct
Correct Answer
can you try to tcpdump on the vlc and use the -vvv option to capture verbose? curious to see if there is a tab or something in there that breaks the parsing that you are hitting. THere are a few options in the VLC that remove or replace tabs and a few other things depending on your version.

Could you also try to point this to a local LC to see if that removes the issue (maybe VLC is manipulating a character at capture)? (not sure if this is prod or testing)
Like • Show 0 Likes0
Actions
David Waugh Oct 24, 2018 3:41 AM
Mark Correct
Correct Answer
Morning Eric
Thank you for your response.
I did a tcpdump on the logdecoder and was able to verify that the message is arriving truncated before it hits the log decoder.

I will try to get it coming directly into a LC to see if that helps. This is a production system so it is quite difficult with firewall rules etc ...
Like • Show 0 Likes0
Actions
David Waugh Oct 24, 2018 5:03 AM
Mark Correct
Correct Answer
To get around having to raise firewall rules i used:

Log Collector in DMZ getting error with SysLog
To set my test Log Collector to a Local Log Collector.
Im reinstalling the remote log collector as a local log collector to test...
Like • Show 0 Likes0
Actions
- David Waugh @ David Waugh on Oct 24, 2018 5:39 AM
  Mark Correct
  Correct Answer
  Hi the same result occured when it was a local collector.
  Like • Show 0 Likes0
  Actions
Arnab Chakraverty Oct 24, 2018 4:58 AM
Mark Correct
Correct Answer
Hi David,

Since the collection is through Windows Collection, the logs shown are correct. In windows Transformation the second part of the event is formed from the /RenderingInfo/Messages attribute. In your case the said attribute has No Data, thus the events looked truncated.

with regards
Arnab
Like • Show 0 Likes0
Actions
- David Waugh @ Arnab Chakraverty on Oct 24, 2018 5:39 AM
  Mark Correct
  Correct Answer
  Hi Arnab is there a solution to this?
  Like • Show 0 Likes0
  Actions
  - David Waugh @ David Waugh on Oct 29, 2018 4:27 AM
    Mark Correct
    Correct Answer
    Hi Arnab Chakraverty is this likely to be possible?
    Like • Show 0 Likes0
    Actions
    - Arnab Chakraverty @ David Waugh on Oct 29, 2018 5:26 AM
      Mark Correct
      Correct Answer
      Hi David,
      
      I do not have an idea on how these said messages are getting generated and how they are being published in the channel.
      Like • Show 0 Likes0
      Actions
David Waugh Oct 30, 2018 8:47 AM
Unmark Correct
Correct Answer
I managed to bodge a work around for this.

1) Set up the event source for winrm collection as normal, and confirm you are getting events in (even if they are truncated)
2) Put the event source into debug verbose mode.
3)This causes all events to be written to the /var/log/netwitness/logcollector/ log files
4) Change the size of the Log Files to 10MB. This causes them to rollover more frequently
5) Grab events out of log files and re inject them into the system
Like • Show 0 Likes0
Actions
- Arnab Chakraverty @ David Waugh on Oct 30, 2018 4:36 PM
  Mark Correct
  Correct Answer
  Hi David,
  Just being curious here, could you not directly send the CEF logs to VLC or LD 514 port from the event source itself? Why do we need a Windows Layer, as it seems to be a custom channel.
  
  with regards
  Arnab Chakraverty
  Like • Show 0 Likes0
  Actions
  - David Waugh @ Arnab Chakraverty on Oct 31, 2018 5:10 AM
    Mark Correct
    Correct Answer
    Hi Arnab Chakraverty that would be the ideal solution. Unfortunately the application only supports writing to the windows application log, and I don't have control over it. Sending via syslog would be so much easier!
    Like • Show 0 Likes0
    Actions