AnsweredAssumed Answered

secure 2.3 GB file under /var/log

Question asked by Deepanshu Sood on Aug 9, 2016
Latest reply on Aug 9, 2016 by Deepanshu Sood

Like • Show 0 Likes0
Comment • 23

Hi,

I have one Virtual log collector running on v 10.4.0.2 and what i observed in that VLC, that there is one file which is being created under /var/log/ and the file name is secure and it have a size of more 2.3 GB, which i have deleted many times, but after some time it again gets appeared at the same location which is slowing down the vlc.

And below are some line of logs which are in the file if if i look into it.

Aug 9 06:58:14 NCORP-VLC-01 sshd[14558]: debug1: user sftp matched group list uploads at line 159

Aug 9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: match found

Aug 9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: reprocess config:160 setting ChrootDirectory /var/netwitness/logcollector/upload_chroot

Aug 9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: reprocess config:161 setting X11Forwarding no

Aug 9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: reprocess config:162 setting AllowTcpForwarding no

Aug 9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: reprocess config:163 setting PasswordAuthentication no

Aug 9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: auth_shadow_acctexpired: today 17022 sp_expire -1 days left -17023

Aug 9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: account expiration disabled

Aug 9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1

Aug 9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: mm_request_send entering: type 8

Aug 9 06:58:14 NCORP-VLC-01 sshd[14558]: debug2: monitor_read: 7 used once, disabling now

Aug 9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: mm_request_receive entering

Aug 9 06:58:14 NCORP-VLC-01 sshd[14560]: debug2: input_userauth_request: setting up authctxt for sftp

Aug 9 06:58:14 NCORP-VLC-01 sshd[14560]: debug3: mm_inform_authserv entering

Aug 9 06:58:14 NCORP-VLC-01 sshd[14560]: debug3: mm_request_send entering: type 3

Aug 9 06:58:14 NCORP-VLC-01 sshd[14560]: debug3: mm_inform_authrole entering

Aug 9 06:58:14 NCORP-VLC-01 sshd[14560]: debug3: mm_request_send entering: type 4

Aug 9 06:58:14 NCORP-VLC-01 sshd[14560]: debug3: mm_auth2_read_banner entering

Aug 9 06:58:14 NCORP-VLC-01 sshd[14560]: debug3: mm_request_send entering: type 9

Aug 9 06:58:14 NCORP-VLC-01 sshd[14560]: debug3: mm_request_receive_expect entering: type

So I am just wondering that why the logs are being generating and how to get rid from them.

Kindly advise. Thanks.

Regards,

Deepanshu Sood.

No one else has this question

Outcomes

23 Replies

Khaled Gamal Aug 9, 2016 3:17 AM
Mark Correct
Correct Answer
Hi Sood,

This is due to that debug logs are written to the /var/log/secure file. Debug logs are often not a good idea to keep on. Please send me the content of the "/etc/rsyslog.conf" so I can check.

Best regards
Khaled
2 people found this helpful
Like • Show 1 Like1
Actions
- Deepanshu Sood @ Khaled Gamal on Aug 9, 2016 3:22 AM
  Mark Correct
  Correct Answer
  Hi,
  
  Thanks Khaled for your response.
  
  I have actually checked it from my end for troubleshooting, but now also I have checked the file and found nothing in it.
  
  There is no IP address is added in the file.
  Below is the output.
  
  #$ActionQueueType LinkedList # run asynchronously
  #$ActionResumeRetryCount -1 # infinite retries if host is down
  # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
  #*.* @@remote-host:514
  # ### end of the forwarding rule ###
  [root@NCORP-VLC-01 yum.repos.d]#
  [root@NCORP-VLC-01 yum.repos.d]#
  
  Regards,
  Deepanshu Sood.
  Like • Show 0 Likes0
  Actions
  - Khaled Gamal @ Deepanshu Sood on Aug 9, 2016 3:33 AM
    Mark Correct
    Correct Answer
    Hi Sood,
    
    In the rsyslog.conf file you should find an entry like the below:-
    
    # The authpriv file has restricted access.
    authpriv.* /var/log/secure
    
    Change it to
    
    # The authpriv file has restricted access.
    authpriv.info /var/log/secure
    
    then restart the rsyslog service and check what happens.
    
    Also another thing to check is the sshd configuration in /etc/ssh/sshd_config
    
    LogLevel INFO
    
    Make sure that it is either commented out or left as INFO not DEBUG.
    
    The main issue you have in the secure logs is that there are debug logs there which are a lot and fills up the file quickly. You need to check how the debug logs were enabled and then disable them. Most likely it would be one of the 2 methods above.
    
    Best regards
    Khaled
    2 people found this helpful
    Like • Show 1 Like1
    Actions
    - Deepanshu Sood @ Khaled Gamal on Aug 9, 2016 3:53 AM
      Mark Correct
      Correct Answer
      Khaled I have checked both the ways.
      
      1st way outcome
      
      I have changed to the below and then restarted the service but when i check the time of the file it was the latest after doing the changes and the restart of the service also.
      
      # The authpriv file has restricted access.
      authpriv.info                                              /var/log/secure
      
      2nd way outcome
      
      I have check /etc/ssh/sshd_config and under it LogLevel INFO
      It's already info not debug.
      
      Note: But one thing i have noticed that when i stop the rsyslog service the file modification get stopped and the file get stopped updating.
      But i am not sure that what impact would be if i keep the service disable, bcuz under /etc/ssh/sshd_config what I can see that there are some configuration exist for the file reader collection.
      
      # SFTP server settings added for NwLogCollector
      StrictModes no
      
      Subsystem sftp internal-sftp
      
      Match User sftp
          AllowTCPForwarding no
          PasswordAuthentication no
          X11Forwarding no
          ForceCommand internal-sftp
          ChrootDirectory /var/netwitness/logcollector
      
      Match Group uploads
          ChrootDirectory /var/netwitness/logcollector/upload_chroot
          X11Forwarding no
          AllowTcpForwarding no
          PasswordAuthentication no
      [root@NCORP-VLC-01 log]#
      
      Please suggest what to do?
      
      Regards,
      Deepanshu Sood.
      Like • Show 0 Likes0
      Actions
      - Khaled Gamal @ Deepanshu Sood on Aug 9, 2016 4:10 AM
        Mark Correct
        Correct Answer
        Hi Sood,
        
        If I understand you correctly, When you stop the rsyslog service, the secure file doesn't get modified. If this is the case then this means that the change you did on the rsyslog configuration file will help and will prevent debug logs from being written to the file.
        
        Make sure that you restarted the rssylog service by the below command:-
        
        service rsyslog restart
        
        Then monitor to check if the size of the secure file gets big frequently again.
        
        Hope this helps
        
        Best regards
        Khaled
        2 people found this helpful
        Like • Show 1 Like1
        Actions
        Deepanshu Sood @ Khaled Gamal on Aug 9, 2016 4:15 AM
        Mark Correct
        Correct Answer
        I mean before doing any change on any file,
        for a try i have stopped the rsyslog service after that the file modification gets stopped.
        
        But After the above, i started the service again and then I followed your suggested steps and make changes in the file and then restarted the service but the file modification is still going on.
        Like • Show 0 Likes0
        Actions
        Khaled Gamal @ Deepanshu Sood on Aug 9, 2016 4:18 AM
        Mark Correct
        Correct Answer
        Are there any debug logs still?
        
        If there is, send me the whole rsyslog.conf file.
        
        Best regards
        Khaled
        1 person found this helpful
        Like • Show 1 Like1
        Actions
        Deepanshu Sood @ Khaled Gamal on Aug 9, 2016 4:57 AM
        Mark Correct
        Correct Answer
        sent the attachment at your email address.
        Like • Show 0 Likes0
        Actions
        Khaled Gamal @ Deepanshu Sood on Aug 9, 2016 5:04 AM
        Mark Correct
        Correct Answer
        The file seems OK.
        
        Is the debug logs still present in the secure logs?
        1 person found this helpful
        Like • Show 1 Like1
        Actions
        Deepanshu Sood @ Khaled Gamal on Aug 9, 2016 5:11 AM
        Mark Correct
        Correct Answer
        Yes the secure file is still there and it's keep on modifying.
        The below are some sample logs..
        
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4616]: debug2: kex_parse_kexinit:
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: reserved 0
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit:
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit:
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: first_kex_follows 0
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: reserved 0
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: mac_setup: found hmac-md5
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug1: kex: client->server aes128-ctr hmac-md5 none
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4617]: debug3: mm_request_send entering: type 78
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4612]: debug3: monitor_read: checking request 78
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4616]: debug2: kex_parse_kexinit:
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4616]: debug2: kex_parse_kexinit: first_kex_follows 0
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4616]: debug2: kex_parse_kexinit: reserved 0
        Aug 8 06:19:01 NCORP-VLC-01 sshd[4616]: debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
        Like • Show 0 Likes0
        Actions
        Khaled Gamal @ Deepanshu Sood on Aug 9, 2016 5:24 AM
        Mark Correct
        Correct Answer
        Hi Sood,
        
        From the logs you sent it shows that SSH is still having debug logs. From the below snippet in my lab:-
        
        [root@sa ssh]# cat /var/log/secure | grep -i debug
        [root@sa ssh]#
        
        This shows that there is no debug logs by default. The changes that were done on your VLC has to be found out to revert it back as it was.
        
        Check both sshd_config and ssh_config under /etc/ssh and check if there is any option that was added to increase the log level.
        
        You can compare the files with another appliance that doesn't have DEBUG logs in the secure files to determine what changes were done.
        
        Best regards
        Khaled
        1 person found this helpful
        Like • Show 1 Like1
        Actions
        Deepanshu Sood @ Khaled Gamal on Aug 9, 2016 5:35 AM
        Mark Correct
        Correct Answer
        Hi Khaled,
        
        I found one thing in the sshd_config file related to Logging Level which is 3.
        I have sent an email to you of the files.
        
        Regards,
        Deepanshu Sood
        Like • Show 0 Likes0
        Actions
        Khaled Gamal @ Deepanshu Sood on Aug 9, 2016 5:37 AM
        Mark Correct
        Correct Answer
        Hi Sood,
        
        I checked the files sent and in the sshd_config there is the below line:-
        
        LogLevel DEBUG3
        
        This has to be changed to:-
        
        LogLevel INFO
        
        After that you will have to restart the sshd service for the new config to take effect.
        
        PS:- When restarting do a restart not a stop then start as if you do a stop, you will lose access to the SSH session.
        
        Best regards
        Khaled
        2 people found this helpful
        Like • Show 1 Like1
        Actions
        Deepanshu Sood @ Khaled Gamal on Aug 9, 2016 5:44 AM
        Mark Correct
        Correct Answer
        Khaled, after the change in the file and restarting the service. The file is still modifying.
        Like • Show 0 Likes0
        Actions
        Khaled Gamal @ Deepanshu Sood on Aug 9, 2016 6:00 AM
        Mark Correct
        Correct Answer
        Send me the output of the below commands:-
        
        1- date
        2- tail -10 /var/log/secure
        3- cat /etc/ssh/sshd_config | grep -i debug
        4- cat /etc/ssh/ssh_config | grep -i debug
        5- cat /etc/sysconfig/sshd | grep -i debug
        6- cat /etc/ssh/sshd_config | grep -i log
        7- cat /etc/ssh/ssh_config | grep -i log
        8- cat /etc/sysconfig/sshd | grep -i log
        2 people found this helpful
        Like • Show 1 Like1
        Actions
        Deepanshu Sood @ Khaled Gamal on Aug 9, 2016 6:05 AM
        Mark Correct
        Correct Answer
        output sent.
        
        Regards,
        Deepanshu Sood.
        Like • Show 0 Likes0
        Actions
        Khaled Gamal @ Deepanshu Sood on Aug 9, 2016 6:10 AM
        Mark Correct
        Correct Answer
        Hi Sood, I checked the output and all is good now.
        
        Note that before all the logs you sent in your last post here in Link is in the same second "Aug 8 06:19:01" because all were debug logs.
        
        Now in the mail you sent me, in a 3 minute interval there were only 10 logs. This is normal and that is the way the log file should be.
        
        You could delete the file if it is already big now from the debug logs and from now it won't log debug logs and it won't get big in size as it did before.
        
        Hope this helps!
        
        Best regards
        Khaled
        2 people found this helpful
        Like • Show 1 Like1
        Actions
        Deepanshu Sood @ Khaled Gamal on Aug 9, 2016 6:48 AM
        Mark Correct
        Correct Answer
        You're rocking Khaled. Big thanks.
        But one thing I'm amazed that how the changes are made in the files?
        Any guess?
        
        Regards,
        Deepanshu Sood.
        Like • Show 0 Likes0
        Actions
        Khaled Gamal @ Deepanshu Sood on Aug 9, 2016 6:51 AM
        Mark Correct
        Correct Answer
        Hi Sood,
        
        Anytime
        
        Any SSH session that occurs gets logged in including any protocol that uses SSH for example sFTP. You can check the IPs that are logged in the file to understand exactly what sessions are being logged.
        
        Best regards
        Khaled
        1 person found this helpful
        Like • Show 1 Like1
        Actions
        Deepanshu Sood @ Khaled Gamal on Aug 9, 2016 6:57 AM
        Mark Correct
        Correct Answer
        Okay understood.
        One thing which I am assuming is that the event sources which are added on the VLC by file reader collection method their logs collection will not be get impacted by this change or will they get impacted?
        
        Pls suggest. This is what I am only afraid of.
        
        Regards,
        Deepanshu Sood.
        Like • Show 0 Likes0
        Actions
        Khaled Gamal @ Deepanshu Sood on Aug 9, 2016 6:59 AM
        Mark Correct
        Correct Answer
        No Impact will affect the log collection at all. You have nothing to be afraid of
        
        Best regards
        Khaled
        3 people found this helpful
        Like • Show 1 Like1
        Actions
        Deepanshu Sood @ Khaled Gamal on Aug 9, 2016 7:04 AM
        Mark Correct
        Correct Answer
        Cool
        Like • Show 0 Likes0
        Actions
        Deepanshu Sood @ Khaled Gamal on Aug 9, 2016 7:24 AM
        Mark Correct
        Correct Answer
        I'll keep an eye on the file and would inform you if it goes too much in size.
        
        many many thanks again.
        
        Regards,
        Deepanshu Sood.
        Like • Show 1 Like1
        Actions

secure 2.3 GB file under /var/log

Outcomes

Related Content

Recommended Content