000012779 - How to manually synchronize or interpret the sync report for DPM appliances?

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000012779
Applies ToRSA Data Protection Manager
IssueHow to manually synchronize or interpret the sync report for DPM appliances?
Appliances are not in sync, and some rows are missing or have different values.
ResolutionTo verify is appliances in a cluster are in sync run the script /opt/rsa/setup/sh/cs_verify_sync.sh. This script is available when installing the CS status pages.
This script will do a database compare of the local database vs the database of all other nodes in the cluster.

NOTE: Rules of transitivity applies. Example:



On a 2 nodes cluster, if I run the script on Node1 and it shows Node2 in sync, Node2 is automatically in sync with Node1.

On a 3 nodes cluster, if I run the script on Node1 and it shows Node2 in sync and Node3 in sync, Node2 and Node3 can be assumed to be in sync.

On a 4 nodes (or more), if I run the script on Node1 and it shows Node2, Node3 and Node4 in sync, Node2 can be out of sync with Node3 and Node4. etc.



ASSUMPTION:

This example will assume a 2 nodes cluster, and the script was run on Node1.

When the script completes, get the zip file generated and look at the files where the filename is compare.TWO.log. At some point you will see a section that starts like this:



***************************************************

TOKEN

***************************************************



Everything below down to the next section is the result of the comparison. The script compares the data on the local database MINUS the data on the remote database. LOCAL MINUS REMOTE. See the Oracle MINUS operator for more details : http://docs.oracle.com/cd/B19306_01/server.102/b14200/queries004.htm

The output will then put side by side, each columns of the table. The LOCAL column value then the REMOTE column value, for all columns. Example with the TOKEN table:




Local Remote

Token Token

Class Class

MUID MUID

------------------------------------------------------------------------ ------------------------------------------------------------------------

Local

Input

Value

------------------------------------------------------------------------------------------------------------------------------------------------------

Remote

Input

Value

------------------------------------------------------------------------------------------------------------------------------------------------------

Local     Remote

Hash     Hash

Value     Value

-------------------------------------------- --------------------------------------------

Local

Token

------------------------------------------------------------------------------------------------------------------------------------------------------

Remote

Token

------------------------------------------------------------------------------------------------------------------------------------------------------

Local     Remote

      Local   Remote  Last       Last

     Create   Create Access     Access

Time     Time  Time       Time

-------------------- -------------------- -------------------- --------------------

Local

Hash

MUID

Value

------------------------------------------------------------------------------------------------------------------------------------------------------

Remote

Hash

MUID

Value

------------------------------------------------------------------------------------------------------------------------------------------------------




Now that you see the columns side by side, you will need to understand the data that will come below:





312e7757882b4de576af6cba4c7289eafd4be5ab83fce9c42f6771003d50773c 312e7757882b4de576af6cba4c7289eafd4be5ab83fce9c42f6771003d50773c

ZjVhNGIxODZlYTU0MzdmM2Y1M2ZmMzVmOWU0YzMxYzg2OGY1ODlhMzJjOGFmMTNjMzgxNDE4ZDY1ZWE1ZTQ3Nw==-AES/CBC/PKCS5Padding-123-8r4jraYLQPIZsARZLzlxtQ==-USJ29ulU4Kj

iPsxpD5N9LUITYNAi6grMFJ5CQygfS78=

ZjVhNGIxODZlYTU0MzdmM2Y1M2ZmMzVmOWU0YzMxYzg2OGY1ODlhMzJjOGFmMTNjMzgxNDE4ZDY1ZWE1ZTQ3Nw==-AES/CBC/PKCS5Padding-123-8r4jraYLQPIZsARZLzlxtQ==-USJ29ulU4Kj

iPsxpD5N9LUITYNAi6grMFJ5CQygfS78=

BIPqVqS4iWeqXz9GRcnf13rg7OM=     BIPqVqS4iWeqXz9GRcnf13rg7OM=

201401S5SJH20004

201401S5SJH20004

       1390831869717    1390831869717 1390831869717      1390829934059

312e7757882b4de576af6cba4c7289eafd4be5ab83fce9c42f6771003d50773cBIPqVqS4iWeqXz9GRcnf13rg7OM=2

312e7757882b4de576af6cba4c7289eafd4be5ab83fce9c42f6771003d50773cBIPqVqS4iWeqXz9GRcnf13rg7OM=2





From the output above, we can now identify those:



Token Class MUID, same local and remote : 312e7757882b4de576af6cba4c7289eafd4be5ab83fce9c42f6771003d50773c

Input Value, same local and remote : ZjVhNGIxODZlYTU0MzdmM2Y1M2ZmMzVmOWU0YzMxYzg2OGY1ODlhMzJjOGFmMTNjMzgxNDE4ZDY1ZWE1ZTQ3Nw==-AES/CBC/PKCS5Padding-123-8r4jraYLQPIZsARZLzlxtQ==-USJ29ulU4KjiPsxpD5N9LUITYNAi6grMFJ5CQygfS78=

Hash value, same local and remote : BIPqVqS4iWeqXz9GRcnf13rg7OM=

Create time, same local and remote : 1390831869717

Last access time





Local value : 1390831869717       

Remote value : 1390829934059



 
So we got it. The reason why this specific record was listed is that the "Last Access Date" of this token has not replicated, which means we need to manually update this single row. We will run the script cs_sync_row.sh:

 



[root@dpm-2 sh]# ./cs_sync_row.sh

Enter the table name to sync (ex. local.client): local.token

Enter the remote cluster code (ex. TWO): TWO

Enter the database columns to sync (use a comma to update multiple columns): last_access_time

Enter the primary key of this table: token

Enter the primary key value that will be synched: '201401S5SJH20004'

The following statement will be executed.

update local.token set (last_access_time) = (select last_access_time from local.token@TWO where token = '201401S5SJH20004') where token = '201401S5SJH20004';

Run it now? (y/n) y

1 rows updated.

Done.



 
Now, the example above is not a good example as the primary key of the TOKEN table is a composite key of TOKEN_CLASS_MUID and TOKEN. But in this case I had only one token class and I knew this token only existed once ins the database, so I know I'm good anyway.

 

Now lets talk about MISSING RECORDS. Assuming we have this:

 



***************************************************

CLIENT

***************************************************

Local  Remote

Client  Client

ID  ID

------------------------------------------------------------------------- -------------------------------------------------------------------------

Local Remote

Client Client

IP IP

-------------------------------- --------------------------------

Local

Client

Hostname

------------------------------------------------------------------------------------------------------------------------------------------------------

Remote

Client

Hostname

------------------------------------------------------------------------------------------------------------------------------------------------------

Local

Client

Appname

------------------------------------------------------------------------------------------------------------------------------------------------------

Remote

Client

Appname

------------------------------------------------------------------------------------------------------------------------------------------------------

    Local   Remote

Local     Remote  Actively Actively    Local   Remote Local       Remote

Client     Client   Managed  Managed Register Register Client       Client

Version     Version      Flag     Flag Mode Mode Status       Status

-------------------- -------------------- -------- -------- -------- -------- -------------------------------- --------------------------------

Local     Remote

      Local   Remote Client     Client

   Identity Identity      Register   Register

 ID       ID  Time       Time

-------------------- -------------------- -------------------- --------------------

c61c6011-ee93-4b65-9daf-6069c8643493-94947233-ad8e-4272-b75b-c29baf93ab1e

10.4.25.42

myhost.internal.rsa.com

SomeDPMApp

2.7 0   0      Approved

1951 1392941705605



We can see from the output above that there is absolutely NO VALUES for the REMOTE values. LOCAL MINUS REMOTE, do not forget this. This means the record is MISSING on Node2. So we need to go on Node2 and PULL the data in locally. For this we will use cs_sync_table.sh
 



[root@dpm-1 sh]# ./cs_sync_table.sh -select

Enter the table name to sync (ex. local.client):

local.client

Enter the remote cluster code (ex. TWO):

TWO

old   1: select &1 from &_table@&remote..WORLD minus select &1 from &_table

new   1: select * from local.client@TWO.WORLD minus select * from local.client

no rows selected



 

Forget about the "no rows selected", the example above was from a synchronized environment, but in reality this command using "-select" will only show rows of REMOTE MINUS LOCAL, so Node1 Minus Node2, so we should technically get the same row count as above. Note : If that table has more than one missing rows, this command will list ALL missing rows. Now let's insert them:
 



[root@dpm-1 sh]# ./cs_sync_table.sh -insert

Enter the table name to sync (ex. local.client):

local.client

Enter the remote cluster code (ex. TWO):

TWO

old   1: insert into &_table (select * from &_table@&remote..WORLD minus select * from &_table)

new   1: insert into local.client (select * from local.client@TWO.WORLD minus select * from local.client)

0 rows created.

Commit complete.



 

The output from the script above would then show you all rows inserted. That table should then be in sync with Node1.
 

Now some technical considerations.


Replication




If you do an INSERT on Node2, that change WILL get replicated to Node1. This WILL generate an apply error on Node1 because the primary key already exists. You can then delete this apply error OR you could stop the capture while you run the script above, this would prevent the change from being captured and replicated.

 

However, if you have a row on Node1 that is missing on Node2 and Node3, if you do run the script from Node2, the change WILL get replicated to Node3, which is what we would like to happen.






 






BLOB. Oracle's Bloc Of Bytes.










Some tables uses BLOB, or LOB data types. You would then get this error:













[root@dpm-1 sh]# ./cs_sync_table.sh -select

Enter the table name to sync (ex. local.client):

local.security_object

Enter the remote cluster code (ex. TWO):

TWO

old   1: select &1 from &_table@&remote..WORLD minus select &1 from &_table

new   1: select * from local.security_object@TWO.WORLD minus select * from local.security_object

select * from local.security_object@TWO.WORLD minus select * from local.security_object

       *

ERROR at line 1:

ORA-00932: inconsistent datatypes: expected - got CLOB






Oracla has a limitation that you cannot do a SELECT from a remote database via a DB link. However you can do an insert. So instead of comparing all fields, a safe thing is to compare the primary key. In this example above we are looking for missing KEYs. We would then have to use the security_object's table primary key which is MUID:

 







[root@dpm-1 sh]# ./cs_sync_table.sh -select muid

Using muid to compare

Enter the table name to sync (ex. local.client):

local.security_object

Enter the remote cluster code (ex. TWO):

TWO

old   1: select &1 from &_table@&remote..WORLD minus select &1 from &_table

new   1: select muid from local.security_object@TWO.WORLD minus select muid from local.security_object

no rows selected




If rows are returned, you can use the same syntax to insert:

 




[root@dpm-1 sh]# ./cs_sync_table.sh -insert muid
Using muid to compare
Enter the table name to sync (ex. local.client):
local.security_object
Enter the remote cluster code (ex. TWO):
TWO
old   1: insert into &_table (select * from &_table@&remote..WORLD where &1 in (select &1 from &_table@&remote..WORLD minus select &1 from &_table))
new   1: insert into local.security_object (select * from local.security_object@TWO.WORLD where muid in (select muid from local.security_object@TWO.WORLD minus select muid from local.security_object))
0 rows created.

Commit complete.

 






 







 




Legacy Article IDa64548

Attachments

    Outcomes