RSA Identity Governance and Lifecycle failed to connect due to space issue
3 years ago
Originally Published: 2017-10-26
Article Number
000041009
Issue

ACM fails to start. 
Logs complains there is no space left on device


java.lang.RuntimeException: java.io.IOException: No space left on device
        at com.aveksa.client.component.cache.FileSystemCache.persist(FileSystemCache.java:320)
        at com.aveksa.client.component.cache.FileSystemCache.getDataWriter(FileSystemCache.java:249)
        at com.aveksa.client.datacollector.collectors.identitydatacollectors.IdentityDataCollector.collect(IdentityDataCollector.java:299)
        at com.aveksa.client.datacollector.collectors.identitydatacollectors.IdentityDataCollector.collectTestData(IdentityDataCollector.java:272)
        at com.aveksa.client.datacollector.framework.DataCollectorManager.collect(DataCollectorManager.java:532)
        at com.aveksa.client.component.collector.DefaultCollectorManager.actUpon(DefaultCollectorManager.java:203)
        at com.aveksa.client.component.collector.DefaultCollectorManager.handle(DefaultCollectorManager.java:102)
        at com.aveksa.client.component.event.DefaultEventManager.handle(DefaultEventManager.java:60)
        at com.aveksa.client.datacollector.framework.SimpleEventSource.notifyListeners(SimpleEventSource.java:67)
        at com.aveksa.client.component.communication.DefaultCommunicationManager.notifyEvent(DefaultCommunicationManager.java:377)
        at com.aveksa.client.component.communication.ChangeListHandler.applyChanges(ChangeListHandler.java:364)
        at com.aveksa.client.component.communication.ChangeListHandler.access$300(ChangeListHandler.java:58)
        at com.aveksa.client.component.communication.ChangeListHandler$ChangeApplyingRunnable.run(ChangeListHandler.java:275)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: No space left on device
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:345)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
        at java.io.BufferedWriter.flush(BufferedWriter.java:254)
        at java.util.Properties.store0(Properties.java:838)
        at java.util.Properties.store(Properties.java:812)
        at com.aveksa.client.component.cache.FileSystemCache.persist(FileSystemCache.java:316)
        ... 13 more
Cause
This can happen if  one of the partition is 100% full.
The file system partition on the RSA Identity Governance and Lifecycle Server is 100% ,causing ACM  fails to start.
Resolution

 



 
Workaround

Free Up Space on  Drive by removing some unwanted stuff.

1]  Run the command "df -k" to determine which partition is full.
In below example we have /home partition reached at capacity 100% full.  

[ Note :- Example below you can see that 5 partitions exists  including /dev/shm , a temporary file storage filesystem.
This example is focused on  /home partition.   These steps can be applied to any partition that reached at capacity.]
 
[root@server log]# df -k
Filesystem            1K-blocks     Used  Available Use% Mounted on
/dev/mapper/vg00-lvroot
                       50264772 32637628   15067144  69% /
tmpfs                  24713544       12   24713532   1% /dev/shm
/dev/sda1                487652   113960     348092  25% /boot
/dev/mapper/vg00-lvhome
                       10190136  9649464      16384 100% /home
/dev/mapper/vg01-lvaveska
                     1849433716 13190972 1742290336   1% /RSA-VIA

2] Identify files/directories consuming majority of the space within /home  using command below :  

Below example showing that oracle directory (inside the /home) is the one using high space on disk  
                     
[root@server home]# du -sh *
40K     admin
40K     aveska
48K     cbangaru
36K     cmulcron
20K     cyberark
36K     dekerman
64K     demipat1
48K     demipat2
56K     echavesc
36K     hdcoper
36K     ibm_uat
6.7M    inhsrv
16K     lost+found
972K    myaqoob
40K     nakbar
9.2G    oracle
28K     pdintaku
40K     pmohatta
40K     racharje
12K     smuvva
1.9M    srmuvva
152K    srvcnow
36K     svc_aveksaftp
100K    svc_cc-chef
16K     taddm

     
3]  Above command shows oracle is the directory takes up the majority of the space within /home.
Check all files/sub directories closely inside /home/oracle directory and verify if there are any unwanted sub-directories /files with high usage are safe to remove.

4] Do not remove files/Directories without understanding their purpose. Consult L2 before proceeding.

5] Another option is to extend partition by adding drive. 
It is not normally advised to remove files/directories inside /home/oracle unless there is some unwanted stuff.
Because /home/oracle is the important directory where our software resides. You should be quite safe deleting stuff under /home/oracle that might break application. In that case extending partition by adding drive will be the option.