Category: Oracle GoldenGate 12.3

Oracle GoldenGate – Parallel Replicat

Initially, when Oracle GoldenGate was launched, there was only one EXTRACT and REPLICAT processes. Later in OGG 11.2.x, Oracle introduced Integrated Extract. So, the normal or already existing Extract was renamed to Classic Extract.Similarly, the Replicat process was also renamed to Classic Replicat. Oracle GoldenGate became a powerful tool for replication and migration. As there were many version upgrades happened in OGG, there was also many processes got introduced which you can see in the below diagram.,

From the above table, you can see a new type of replicat process has been introduced from OGG 12.3, which is call “PARALLEL REPLICAT”. Integrated Replicat itself is very fast in applying the data to the target as it has parallelism concepts in it. But this Parallel Replicat process is even more faster than the Integrated Replicat process.

  • Parallel Replicat has a highly scalable apply engine which achieves a apply rate up to 1 million+ operations per second.
  • It is 5X faster than Integrated Replicat
  • Applies a single large transaction in parallel
  • Parallelizes apply by carefully handling the dependent transactions
  • Dependency calculation and parallelism outside the database
  • Ability to Parallelize a Single Large Transaction
  • Processes large transactions faster as they are parallelized
  • Dependencies are still considered while parallelizing large transaction
  • Ability controlled by a SPLIT_TRANS_RECS parameter which specifies the transaction split size (in records). Default is 100,000.

There are two types of Parallel Replicat processes available.

  • Integrated Parallel Replicat

  • Non-Integrated Parallel Replicat

Let us see about the architecture and working concepts of this newly introduced Parallel Replicat process,

Like Integrated Replicat, Parallel Replicat process have background processes and each one of them plays it’s role in order to apply the transactions or data to the target server. Below is the architecture diagram of the Parallel Replicat process.,

So, what these background processes does? Check below,

MAPPERS

The Mappers read the trail file and map records, forward the mapped records to the Master.

APPLIERS

Applies the records to the target database.

MASTER

The Master process consists of two separate threads, Collater and Scheduler.

COLLATER

The Collater is responsible for managing and communicating with the Mappers, along with receiving the mapped transactions and reordering them into a single in-order stream.

SCHEDULER

The Scheduler is responsible for managing and communicating with the Appliers, along with reading transactions from the Collater, batching them, and scheduling them to Appliers.

As per above diagram, we do have some special parameters for the Parallel Replicat according to which it behaves. Let’s see what these parameters do,

MAP_PARALLELISM

Configures number of mappers. This controls the number of threads used to read the trail file. The default value is 2.

APPLY_PARALLELISM

Configures number of appliers. This controls the number of connections in the target database used to apply the changes. The default value is 4.

MIN_APPLY_PARALLELISM | MAX_APPLY_PARALLELISM

The Apply parallelism is auto-tuned. You can set a minimum and maximum value to define the ranges in which the Replicat automatically adjusts its parallelism. There are no defaults. Do not use with APPLY_PARALLELISM at same time.

SPLIT_TRANS_REC

Specifies that large transactions should be broken into pieces of specified size and applied in parallel. Dependencies between pieces are still honored. Disabled by default.

COMMIT_SERIALIZATION

Enables commit FULL serialization mode, which forces transactions to be committed in trail order.

The difference between a Integrated Parallel Replicat and Non-Integrated Parallel Replicat is

    • APPLIER process runs outside of the database in Non-Integrated Parallel Replicat where in Integrated mode of Parallel Replicat, the APPLIER process will run inside the database.
    • Integrated Replicat process can be configured only when your Oracle RDBMS version is 12.2 and above.

Now, let’s configure a Non-Integrated Parallel Replicat process.

I have installed Oracle GoldenGate 12.3.0.1.2 version,

Add the parameters for the Parallel Replicat process.

Add the Parallel Replicat process using the below command.

You can see the Parallel Replicat process has been added now.

How do we know if the replicat added is a parallel or integrated or classic? You can see a keyword PARALLEL when seeing the info of the replicat process.

Let’s start the Parallel Replicat process now.

Now the replicat is running.

Below is the report file of the Parallel Replicat process PREP which has been added now.

GGSCI (ogg123-2.localdomain as ggadmin@ORCL) 12> view report PREP


***********************************************************************
                 Oracle GoldenGate Delivery for Oracle
    Version 12.3.0.1.2 OGGCORE_12.3.0.1.0_PLATFORMS_171208.0005_FBO
   Linux, x64, 64bit (optimized), Oracle 12c on Dec  9 2017 01:06:26

Copyright (C) 1995, 2017, Oracle and/or its affiliates. All rights reserved.


                    Starting at 2018-11-18 19:29:33
***********************************************************************

Operating System Version:
Linux
Version #1 SMP Wed Feb 2 18:40:23 EST 2011, Release 2.6.32-100.28.5.el6.x86_64
Node: ogg123-2.localdomain
Machine: x86_64
                         soft limit   hard limit
Address Space Size   :    unlimited    unlimited
Heap Size            :    unlimited    unlimited
File Size            :    unlimited    unlimited
CPU Time             :    unlimited    unlimited

Process id: 9554

Description:

2018-11-18 19:29:33  WARNING OGG-02904  Replication of PARTIAL XML containing NCHAR/NVARCHAR/NCLOB data may cause divergence.

***********************************************************************
**            Running with the following parameters                  **
***********************************************************************

2018-11-18 19:29:33  INFO    OGG-03059  Operating system character set identified as UTF-8.

2018-11-18 19:29:33  INFO    OGG-02695  ANSI SQL parameter syntax is used for parameter parsing.

2018-11-18 19:29:34  INFO    OGG-01360  REPLICAT is running in Parallel mode.
replicat PREP
userid ggadmin, password ***
MAP_PARALLELISM 2
MIN_APPLY_PARALLELISM 2
MAX_APPLY_PARALLELISM 8
SPLIT_TRANS_RECS 100
MAP source.t1, TARGET target.t1;

2018-11-18 19:29:34  INFO    OGG-06067  Spawned Applier with pid 9,564.

2018-11-18 19:29:34  INFO    OGG-06067  Spawned Applier with pid 9,565.

2018-11-18 19:29:34  INFO    OGG-06067  Spawned Applier with pid 9,566.

2018-11-18 19:29:34  INFO    OGG-06067  Spawned Applier with pid 9,567.

2018-11-18 19:29:34  INFO    OGG-06067  Spawned Applier with pid 9,569.

2018-11-18 19:29:34  INFO    OGG-06067  Spawned Mapper with pid 9,568.

2018-11-18 19:29:34  INFO    OGG-06067  Spawned Mapper with pid 9,570.

2018-11-18 19:29:36  INFO    OGG-06451  Triggers will be suppressed by default.

2018-11-18 19:29:36  INFO    OGG-01815  Virtual Memory Facilities for: COM
    anon alloc: mmap(MAP_ANON)  anon free: munmap
    file alloc: mmap(MAP_SHARED)  file free: munmap
    target directories:
    /u01/app/ogg/dirtmp.

2018-11-18 19:29:37  INFO    OGG-06604  Database ORCL CPU info: CPU Count 1, CPU Core Count 0, CPU Socket Count 0.

***********************************************************************
**                     Run Time Messages                             **
***********************************************************************


2018-11-18 19:30:09  INFO    OGG-06067  Spawned Applier with pid 9,624.

As we know, the replicat process has it’s own checkpoint table which is mainly used for recovery purpose. This Parallel Replicat also has it’s own checkpoint table. But this does not take the default or the checkpoint table which is mentioned in the ./GLOBALS parameter

The checkpoint table in here is ggadmin.ggchkpt

This checkpoint table should have an entry in it for the parallel replicat process PREP. But, we will not see it in here.

This is because, the Parallel Replicat process creates it’s own checkpoint table. It uses the same schema name which is mentioned in the ./GLOBALS parameter, but the table name will be created with an extension as below,

Querying this checkpoint table will give the information of the parallel replicat process.

We have multiple applier processes, but only 3 were used and hence we can see the information for the 3 processes. So, this is something different when comes to Parallel Replicat process. Always do not check for the default checkpoint table which will be created.

So, we have created Non-Intgrated Replicat process. Let’s try to create a Integrated Parallel Replicat process now. Add the parameters for the Integrated Parallel Replicat process.

Add the replicat IPREP now. The keyword used in here is “PARALLEL INTEGRATED”.

Let’s start the Integrated Parallel Replicat process IPREP.

Oops! The process is stopped. Why? Let us see the report file to check for the error messages.

As I mentioned earlier in this post, Integrated Parallel Replicat process is only supported when the Database version 12.2 and above. The database which I have used in here is 12.1.x and hence this got failed.

Finally, below is the comparison report between various Replicat processes available in Oracle GoldenGate.

Hope the post gave you much information about the new replicat process “Parallel Replicat” which was introduced in OGG 12.3. Your comments are always welcome.

Cheers 🙂

Oracle GoldenGate 12.3 Microservices – Installation and Configuration

OGG Microservices Architecture has been introduced from OGG version 12.3. A kind of architecture mainly designed on the perspective of Cloud operations. So, the older one is now called as Classic Architecture. In this article, I would be explaining about the Installation and Configuration of Oracle GoldenGate Microservices Architecture.

To know more about the OGG Microservices Architecture, you can check the below link,

From OGG 12.3, you will can there would be two different binaries which is as follows,

(i) OGG Microservices Architecture
(ii) OGG Classic Architecture

1. So, you need to download the binary which is with keyword Microservices. I have downloaded the binary in the location /u01/softs

2. Unzip the binary

3. Before installing OGG 12.3. MA, you need to create three directories which are below,

Why you need the above three directories? What is the need of it?

/u01/app/ogg/ogg123_ma is OGG_HOME
/u01/app/ogg/ogg123_sm is  Service Manager Home
/u01/app/ogg/ogg123_deploy is Deployment Home

4. Now, navigate to the directory where binary is stored.

5. Navigate to the below directory and run the runInstaller file to start the installation of OGG.

6. Now execute the runInstaller to start the installation wizard,

7. Choose option according to your database version and click next. Here, I have installed Oracle Database 12c and hence I have selected the Oracle 12c option.

8. Choose software location. This is OGG HOME /u01/app/ogg/ogg123_ma

9. Save a response file if required and then click Install

10. When it is successfully Installed, you will see the following

11. Oracle GoldenGate for Microservices is installed now. If you want to see the logs for the installation, you can get it from below location,

Understanding the installed binaries.

Files under the bin directory,

——————————————————–DEPLOYING OGG MICROSERVICES——————————————————–

Setting Environment Variables

[oracle@OGG123-1 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1
[oracle@OGG123-1 ~]$ export ORACLE_SID=ORCL
[oracle@OGG123-1 ~]$ export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
[oracle@OGG123-1 ~]$ TNS_ADMIN=$ORACLE_HOME/network/admin
[oracle@OGG123-1 ~]$ export OGG_HOME=/u01/app/ogg/ogg123_ma

oggca.sh script under $OGG_HOME/bin will help to deploy OGG services for MA.
[oracle@OGG123-1 bin]$ ./oggca.sh

1. Configure Service Manager. The location of Service manager is /u01/app/ogg/ogg123_sm

2. If you want to register as a service check the box Register Service Manager as a system service/daemon

3. Click on Add new deployment.

4. Give a name to deployment and software home will be OGG HOME.

5. Provide a deployment home. We chose /u01/app/ogg/ogg123_deploy

6. Set the environment variables if you have not set it before invoking oggca.sh

7. Set the username and password for Service manager and Services Server logins as oggadmin/oggadmin

8. Uncheck the SSL/TLS checkbox. If you have the SSL/TLS certificate created, then check the box. For this demo, uncheck the box

9. Choose the administration server port. If you click on Distribution port it will be automatically set to next port. Eg; If you set admin server 7001, Distribution server will be 7002 and son on.

10. Set default schema to add in GLOBALS file ggchema parameter

11. Save the response file if required. Also review all the port numbers and environmental variables before proceeding and ensure that they are correct.

12. Click finish to complete the deployment. You will be prompted to run the servicemanager.sh script as root

13. Run the script and click OK.


[root@OGG123-1 ~]# sh /u01/app/ogg/ogg123_sm/bin/registerServiceManager.sh
———————————————————————————-
Oracle GoldenGate Install As Service Script
———————————————————————————-
OGG_HOME=/u01/app/ogg/ogg123_ma
OGG_CONF_HOME=/u01/app/ogg/ogg123_sm/etc/conf
OGG_USER=oracle
Running OracleGoldenGateInstall.sh…

14. The configuration of OGG deployment is completed.

15. Now the Service Manager is started

——————————————————–CONFIGURING OGG FOR MICROSERVICES——————————————————–

1. Open web browser and enter below URL to open Service manager console. /u01/app/ogg/ogg123_sm

http://localhost:7820

Since I am opening the Service Manager console from the remote machine, I have used the IP address. If you are opening it in the same machine, then you can use the “localhost”

2. Enter the username and password as oggadmin

3. This will show the service manager which has all the service under it.

1. Administration Server

2. Distribution Server

3. Receiver Server

4. Performance Metrics Server

——————————————————-PREPARING THE DATABASE FOR ORACLE GOLDENGATE——————————————————–

1. Enable database level supplemental logging
2. Force logging should be set to TRUE
3. ENABLE_GOLDENGATE_REPLICAT should be set to true

SQL> select supplemental_log_data_min , force_logging, log_mode, cdb from v$database;

SQL> alter database add supplemental log data;
SQL> alter database force logging;
SQL> alter system set enable_goldengate_replication=TRUE;


The parameter ENABLE_GOLDENGATE_REPLICATION is not present in the Oracle 12c database and hence it has thrown the error. Please ignore it.

4. Create ggadmin user

CREATE USER ggadmin identified by oracle;
GRANT CREATE SESSION, ALTER SESSION, RESOURCE, CONNECT to ggadmin;
GRANT SELECT ANY DICTIONARY, FLASHBACK ANY TABLE to ggadmin;
GRANT SELECT ANY TABLE to ggadmin;
GRANT DBA to ggadmin;
GRANT SELECT on dba_clusters to ggadmin;
GRANT EXECUTE on DBMS_FLASHBACK to ggadmin;
exec dbms_goldengate_auth.grant_admin_privilege ( grantee => 'ggadmin', privilege_type => 'CAPTURE', grant_select_privileges => TRUE ); 

SQL> select * from dba_sys_privs where grantee=’GGADMIN’;

5. Creating source user

Create user source identified by oracle;
Grant connect, resource, create session, alter session to source;
Grant unlimited tablespace to source;

6. Creating target user

Create user target identified by oracle;
Grant connect, resource, create session, alter session to target;
Grant unlimited tablespace to target;

——————————————————–CONFIGURE EXTRACT AND REPLICAT PROCESSES——————————————————–

1. Already we have opened the Service Manager console. Click on Administration Server to configure extract/replicat process.

Note: Each and every Services has it’s own login page for it.

2. Click on the + symbol to add an Extract process.

3. Select the type of Extract you need to add.

4. Enter the extract name and enable log retention if you want to register the extract with RMAN. This makes the RMAN to retain the log files required by extract process

5. In Microservices Architecture, normal userid dblogin is not supported. The default login is using alias stored in credentialstore. First add credentialstore, enter the username details and click submit.

6. Enter the domain and alias that was created in previous step. Give a name to trail file and Click Next.

7. This will show the parameter file. Add the tables which you want to capture. We have added all tables under schema source. Now click the button “Create and Run”.

Now classic extract is added and started.

8. Before proceeding with performing DML operations. We need to enable supplemental logging at table level. So stop the process and add schematrandata for schema source.

To enable trandata, Click on Application Navigation->Configuration->user icon on right side

Click on (+) after Trandata

9. Enter source schema and click submit

In the notifications icons, you will see the trandata added to schema source

10. You can configure the autorestart, autostart, purge trail files in maintenance and edit parameter files

——————————————————–ADD DISTRIBUTION SERVER——————————————————–

In the Service Manager console, click the Distribution Server

Click the Plus symbol (+) to create the distribution server

Add the pump details and click the button “Create and Run”

Now the pump is added and it will be received by Receiver server running on port 7003.

In the Service Manager console, click the Receiver Server

——————————————————–ADDING REPLICAT PROCESS——————————————————–

1.Already we have opened the Administration Server.

2. Click on the + symbol to add a Replicat process.

3. Select the type of Replicat process need to be added.

4. Add the details for the Replicat process. Here the replicat process name is RMICRO.

5. Modify the parameters respectively and click the button “Create and Run”.

You can see the replicat process is added and it is running.

——————————————————–REPLICATION TESTING——————————————————–

Create a table on source and target and load some data to it.

Insert rows to the table in the source.

Open the Service Manager console and click the Performance Metrics Server to check the database statistics.

Click on the Extract process.

Click on the Database Statistics tab in the top to check the statistics. You could see there were 2 INSERTS.

Now click the Replicat process.

You can again see the 2 INSERT operations under the Database Statistics.

Now the check the Target table if the row got replicated.

Yes it is replicated.

In this article, I have explained on how to create Classic Extract and Classic Replicat. Similarly, we can also create Integrated Extract/Replicat, Parallel Replicat (newly added in OGG 12.3)

Hope this article helped you.

Cheers 🙂

Oracle GoldenGate Microservices Architecture

Oracle GoldenGate provides low-impact capture, routing, transformation, and delivery of database transactions across homogeneous and heterogeneous environments in real-time with no distance limitations.

We all know the architecture of Oracle GoldenGate which is pretty simple and easy to understand. It has components like,

1. Manager Process
2. Extract Process
3. DataPump Process
4. Server Collector Process
5. Replicat Process
6. Trail Files
7. Checkpoint Files

Each and every process has it’s own roles and responsibilities to play during the course of replication. The more details about this architecture is explained in the below link,

From OGG 12.3, a new kind of architecture has been introduced in Oracle GoldenGate known as “Microservices Architecture”. In this article, I will be explaining you the various components involved in it and their roles and responsibilities.

Oracle GoldenGate Microservices Architecture can be used to manager the data replication using an HTML interface. The replication processes works with the secure REST API interfaces.

In the Classic Architecture of OGG, we need separate tools to monitor the OGG instances. But Oracle GoldenGateMA provides all the tools you need to configure, monitor, and administer your deployments.

There are five major components introduced in Oracle GoldenGate MA. Each component has it’s own login console page. They are as below,

SERVICE MANAGER
ADMINISTRATION SERVER
DISTRIBUTION SERVER
RECEIVER SERVER
PERFORMANCE METRICS SERVER

One of the difference between the Classic Architecture and OGG MA is, in the Classic Architecture you have DataPump process and Server Collector process. But in OGG MA it has been replaced by Distribution Server and Receiver Server. The Extracts and Replicats does not have any change in it. It is the same as the Classic Architecture.

Let us see what are the roles and responsibilites of each component of OGG MA.

SERVICE MANAGER

1. A Service Manager acts as a watchdog for other services available with Microservices Architecture and allows you to manage one or multiple Oracle GoldenGate deployments on a local host.

2. It runs as a service and maintains inventory and configuration information about the deployments. From the Service Manager console, you can start and stop instances, and query deployments and other services.

ADMINISTRATION SERVER

1. An Administration Server supervises, administers, manages, and monitors processes operating within an Oracle GoldenGate deployment for both active and inactive processes.

2. The Administration Server operates as the central control entity for managing the replication components in your Oracle GoldenGate deployments. In Classic Architecture, all these tasks will be taken care by the Manager process where as in OGG MA, the Administration Server will taking care of these tasks. As I mentioned earlier, the Service Manager is like a real manager in your premises 😛 is just a Watch Dog.

3. The key feature of the Administration Server is the REST API Service Interface that can be accessed from any HTTP or HTTPS client, such as the Microservices Architecture service interfaces or other clients like Perl and Python.

4. You really do not want to login to the OGG server where it is installed. From the Administration server, you can create and manager your local Extract and Replicat processes. Below are the tasks which are done using the Administration Server.

Extract and Replicat processes
          Add, alter, and delete
          Register and unregister
          Start and stop
          Review process information, statistics, reports, and status including LAG and checkpoints
          Retrieve the report and discard files

Configuration (parameter) files
Checkpoint, trace, and heartbeat tables
Supplemental logging for procedural replication, schema, and tables
Tasks both custom and standard, such as auto-restart and purge trails
Credential stores
Encryption keys (MASTERKEY)
Add users and assign their roles

DISTRIBUTION SERVER

1. Distribution Server replaces the classic multiple source-side data pumps with a single instance service. This server distributes one or more trails to one or more destinations and provides lightweight filtering only (no transformations).

2. Unlike Data Pump process, it is a high performance application that is able to handle multiple commands and data streams from multiple source trail files, concurrently.

RECEIVER SERVER

1. A Receiver Server is a replacement of the multiple discrete target-side Collectors with a single instance service.

2. It is the central control service that handles all incoming trail files.

3. It coordinates with the Distribution Server and provides compatibility with the classic architecture pump for remote classic deployments.

PERFORMANCE METRICS SERVER

1. The Performance Metrics Server uses the metrics service to collect and store instance deployment performance results.

2. This metrics collection and repository is separate from the administration layer information collection.

3. All Oracle GoldenGate processes send metrics to the Performance Metrics Server. It can be used in both Microservices Architecture and Classic Architecture. Hence from OGG 12.3, you don’t have DATASTORE for monitoring purposes.

4. Use the Performance Metrics Server to:

Query for various metrics and receive responses in the services JSON format or the classic XML format
Integrate third party metrics tools
View error logs
View active process status
Monitor system resource utilization

ADMIN CLIENT

1. There is one more component with this architecture. You might have noted in the Microservices Architecture diagram which is nothing but the ADMIN CLIENT.

2. The Admin Client is a command line utility (similar to the classic GGSCI utility). It uses the REST API published by the Microservices Architecture(MA) Servers to accomplish control and configuration tasks in an Oracle GoldenGate deployment.

3. The Admin Client can be used to create and manage processes instead of doing it graphically using OGG MA.

4. You can use the Admin Client to execute all the commands necessary to create a new Extract, create a custom Extract application using the REST API, or use the Administration Server available with MA to configure an Extract.

So, now we would think what is the difference between the GGSCI and ADMIN CLIENT. Below are the differences.

Installation and configuration of the OGG Microservices are explained in the below link,

Thanks for the support.

Cheers 🙂