Disk Space Concern - Multiple imsTrace.log.YYYY-MM-DD file in /opt/rsa/am/server/logs

[DavidThomson1]

I noticed the disk space on my Primary instance was starting to creep up, so I had a look around the file structure to see what was going on.  I noticed on in the /opt/rsa/am/server/logs directory there are 757 file with the name imsTrace.log.YYYY-MM-DD, starting at imsTrace.log.2019-04-01 and going to imsTrace.log.2021-04-26.  These files are on average around 40 MB, so altogether they are taking up serval GB.  I assume I'm okay to remove these files, especially files that are over 2 years old (please let me know if I shouldn't remove them) but how do I stop these files being retained for so long, as I guess I only need a few weeks worth of these files for troubleshooting purposes.  Is there an automatic archival task I can setup to do this?  I've all ready got the Administration > Archive Audit Logs > Schedule Log Archival to export those logs to a Windows server nightly, so these imsTrace.log.YYYY-MM-DD logs don't seem related to that.

Another area that I saw using disk space was /opt/rsa/am/updates.  In this folder there are files (such as backup_sschelp_timestamp.tar.gz and backup_wls_timestamp.tar.gz) that date back to 2018.  The latest files seem to to date back to our last upgrade but the files with older dates total at around 5.7 GB.  Am I okay to delete these older update files or are they required?

[IanStaines]

This appears to be a question for RSA Authentication Manger not RSA Access Manager.

I will try to move it to one of the other groups so it gets visibility. 

[PiersB]

Yes. The imsTrace files are diagnostic files containing system tracing information. Old imsTrace. files can safely be removed.

I would consult with customer support to have your system restored to the original configuration. The logging problem you're encountering is not normal. The log file name and roll-over policies appear have been manually altered in your AM configuration. An unaltered configuration generates 30 trace files each capped at 10 MB. Perhaps the logging on your system was altered to be compatible with some external consumer (i.e. Splunk)?

[DavidThomson1]

Thanks Piers.

We are sending our audit and admin logs to Splunk but we are also doing this in our test system and we're not seeing this behaviour there.

Do you know if there is a config file that controls the log file name and roll-over policies, so that I compare our production and test environments, just to see what has been altered?

[PiersB]

The "/opt/rsa/am/utils/resource/log4j.xml" was altered. This is the normal "imsTrace.log" stanza:

    <appender name="IMS_APPENDER_TRACE" class="org.apache.log4j.RollingFileAppender">
        <param name="File" value="logs/imsTrace.log"/>
        <param name="MaxFileSize" value="10MB"/>
        <param name="MaxBackupIndex" value="30"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="@@@%d, [%t], (%13F:%L), %c, %p, %m%n"/>
        </layout>
    </appender>

Files are numbered 1-30 and roll-over past that point. This caps the total trace at 300MB total, ever. Changing this file is not supported and, depending on the changes, can certainly cause your appliance to exhaust all disk space.

By collecting all the records in a file named for the date, it forces the logger only roll-over the file every 24-hours. Under heavy load, AM server trace will create and roll-over all 30 log files within the course of a couple of hours. 

[DavidThomson1]

Thanks for the information.  However I have compared my production and test log4j.xml files and they are exactly the same.  The test environment is doing exactly as you describe, with the files being named imsTrace.log.1 to 30 and each limited to 10MB.

Is there anything that could be causing my production environment to ignore the log4j.xml settings?  

[JayGuillette]

In support we have found that 

/opt/rsa/am/utils/resources/log4j.xml does not work even after reboot, at least for imsTrace.log.

There is an RSAUtility Command line

    ./rsautil update_config ims.logging.trace.file.rotation_size

which works for the size of the imsTrace.log file, but not for the number of backup copies.

What really works is updating log settings in the ims_config_value table in the postgres internal database. However making changes to your internal database is not something to be undertaken lightly, you do not want to make any mistakes.

I would delete or move all 'old' imsTrace.log files from the /opt/rsa/am/server/logs directory, keeping maybe the last 30 or less copies. Then turn off verbose logging in the Security console.

If your imsTrace.log files are still not rolling over and purging, then open a Support case ask about updating the settings in postgres.

[JayGuillette]

One thing to check, is the logging type setting.  

SSH to AM server Linux with rsaadmin account

cd /opt/rsa/am/utils

./rsautil export-config | fgrep -i logging.trace

You will see something like this

Configuration Export 8.3.0.0.0 (1397825)

Copyright (C) 1994-2018 Dell Inc. or its subsidiaries. All Rights Reserved.

ims_config_value:009ad0efbb10e5ba69db9a987bde2087,2c796aa1110110ac08dce51d58dd3a1a,ims.logging.trace.file.layout,%d, [%t], (%13F:%L), %c, %p, %m%n

ims_config_value:5ce7dfc2147441eb88d10cf1e345d59f,2c796aa1110110ac08dce51d58dd3a1a,ims.logging.trace.file.rotation_size,10

ims_config_value:60eedc663222d88c5aaf501d13b934ab,2c796aa1110110ac08dce51d58dd3a1a,ims.logging.trace.file.rotation_schedule,2

ims_config_value:7365d4de516f599f3a6011f7b5235968,2c796aa1110110ac08dce51d58dd3a1a,ims.logging.trace.file.name,logs/imsTrace.log

ims_config_value:bfeb0165100110ac013673f2c1b9ea0f,2bd9846e100110ac08dce52390776e69,ims.logging.trace.file.name,logs/imsTrace.log

ims_config_value:bfeb0165100110ac01367d928f9a7ce0,2bd9846e100110ac08dce52390776e69,ims.logging.trace.file.rotation_type,2

ims_config_value:bfeb0165100110ac01368457d2123c60,2bd9846e100110ac08dce52390776e69,ims.logging.trace.file.rotation_size,10

ims_config_value:bfeb0165100110ac01368dcc4e19052e,2bd9846e100110ac08dce52390776e69,ims.logging.trace.file.max_backup_index,10

ims_config_value:bfeb0165100110ac0136951e677d1b41,2bd9846e100110ac08dce52390776e69,ims.logging.trace.file.rotation_schedule,2

ims_config_value:bfeb0165100110ac01369bed70beaac3,2bd9846e100110ac08dce52390776e69,ims.logging.trace.file.layout,%d, [%t], (%13F:%L), %c, %p, %m%n

ims_config_value:ed15c57ed862638196ad06a1e0547480,2c796aa1110110ac08dce51d58dd3a1a,ims.logging.trace.file.max_backup_index,10

ims_config_value:fa915614ea64603dc62554db3b0bdd66,2c796aa1110110ac08dce51d58dd3a1a,ims.logging.trace.file.rotation_type,2

If you see 'rotation_type,2' instead of type, 1 - that could explain seeing the DATE at the end of the imsTrace.log file name.