BizTalk 2006: Performance comparision – To debatch or not ?

At a client site, our BizTalk 06 was running into performance problems (something very common with BizTalk ). And I jumped into the party to generate some statistics to find out what was really happening. To make matters GREAT – the guy who actually developed the BizTalk application had moved on to another company!

After digging into the application, it was clear that a pipeline was debatching the incoming message and submitting these messages into the BizTalk message box. Another orchestation picked up these messages, performed some business logic and resubmitted them into the message box. This step was repeated several times within the flow of the architecture. In this case, I was dealing with one pipeline component (PipelineComponent) and three BizTalk orchestrations (Enhance,Categorize and Attest).

The flow was something as such

 

Message –> Pipeline –> [MessageBox] –> Enhance –> [MessageBox] –> Categorize –> [MessageBox] –> Attest –> Send.

Note that there would be atleast 3n instances of orchestrations running for every record submitted.

To generate some stats, the first thing I did was to insert debugging code at the entry and exit points of the components to generate timestamps. This would give me an indication of the total execution time each component undertook. Then, I had to view some performance counters to derive some results and pin point what needs to be changed.

So, with the background almost covered – we are ready to roll

 

 

 

 

 

 

 

 

 

 

Case A) File with 02 records

 ** Remember this would be debatched and two instances of each orchestation would be triggered.

 

Pipelines
Name	Instances	Start Time	End Time	Total
PipelineComponent	01	10:47:44.135	10:47:44.385	00:00:00.250

    

         Number of pipeline instances = 01

        Average Execution time 00:00:00.250    

 

Orchestration Process
Name	Instances	Start Time	End Time	Total
Enhance	02	10:47:44.854	10:47:45.291	00:00:00.437
Categorize	02	10:47:45.619	10:47:45.697	00:00:00.078
Attest	02	10:47:45.869	10:47:45.994	00:00:00.125

 

        Number of instances = 2x 3 = 6

        Average Execution time = 00:00:00.640

 

Execution –  Pipeline à Orchestration = 00:00:00.890

 

 

è         
Graph scale = 20

                          

 

Counter	Units (~Max)
Documents received per second	1
Orchestrations created per second	6
Persistence points per second	16
Running orchestrations	2
Orchestrations completed	6
Inbound Latency	0.39
Spool/Queue Size	2

====================================================================================== 

 Case B) File with 18 records

 

Pipelines
Name	Instances	Start Time	End Time	Total
PipelineComponent	1	11:00:23.245	11:00:23.542	00:00:00.297

       

        Number of pipeline instances = 1

        Average Execution time =     00:00:00.297

 

Orchestration Process
Name	Instances	Start Time	End Time	Total
Enhance	18	11:00:23.918	11:00:26.450	00:00:02.532
Categorize	18	11:00:27.060	00:00:01.032	00:00:01.032
Attest	18	11:00:26.716	11:00:28.216	00:00:01.500

 

        Number of instances = 18 x 3 = 36

        Average Execution time = 00:00:05.064

 

Execution -  Pipeline à Orchestration = 00:00:05.361

  Detailed Analysis

 

 

è Graph scale = 100

 

 

Counter	Units (Approx)
Documents received per second	1
Orchestrations created per second	26
Persistence points	75
Running orchestrations	5
Orchestrations completed	24
Inbound Latency	0.39
Spool/Queue Size	38

 

    =========================================================================

Case C) File with 400 records

Pipelines
Name	Instances	Start Time	End Time	Total
PipelineComponent	1	11:38:38.427	11:38:39.661	00:00:01.234

 

        Number of pipeline instances = 1

        Average Execution time =      00:00:01.234

 

Orchestration Process
Name	Instance Id	Start Time	End Time	Total
Enhance	400	11:38:44.537	11:39:38.757	00:00:54.220
Categorize	400	11:39:31.132	11:40:00.414	00:00:29.282
Attest	400	11:39:52.914	11:40:21.712	00:00:28.798

 

        Number of instances = 3 x 400 = 1200

        Average Execution time = 00:01:52.300

 

Execution -  Pipeline à Orchestration = 00:03:44.600

 

 Detailed Analysis

 

 è Graph scale = 1000

  

Counter	Units (Approx)
Documents received per second	1
Orchestrations created per second	200
Persistence points	83
Running orchestrations	20
Orchestrations completed	29
Inbound Latency	6
Spool Size	830

 ==============================================================

Case D) File with 1600 records

 

Pipelines
Name	Instance Id	Start Time	End Time	Total
PipelineComponent	1	11:59:13.318	11:59:36.647	00:00:23.329

       

        Number of pipeline instances = 1

        Average Execution time =      00:00:23.329

 

Orchestration Process
Name	Instances	Start Time	End Time	Total
Enhance	1600	11:59:43.366	12:03:15.546	00:03:32.180
Categorize	1600	12:03:06.796	12:04:31.190	00:01:24.394
Attest	1600	12:04:24.689	12:06:05.850	00:01:41.161

 

        Number of instances = 1600 x 3 = 4800

        Average Execution time = 00:06:37.735

 

 

Execution -  Pipeline à Orchestration = 00:13:15.470

 

 Detailed Analysis

 

èGraph scale = 1000

  

Counter	Units (Approx)
Documents received per second	1
Orchestrations created per second	240
Persistence points	83
Running orchestrations	25
Orchestrations completed	11
Inbound Latency	33
Spool Size	2333

 

Case D) File with 25000 records (Note physical file size is approx 2 MB)

 – Took too long, therefore had to terminate the enitre process.

 

Pipelines
Name	Instances	Start Time	End Time	Total
PipelineComponent	1	14:40:21.580	14:49:48.180	00:09:26.600

        Number of pipeline instances = 01

    Average Execution time = 00:09:26.600 (Pipeline ONLY)

 Detailed Analysis

 

 

Total percentage covered by each artefact within the execution cycle. As each component is called in the order : Pipeline — > Enhance –> Categorize –> Attest

 Artefact Percentage Execution vs Total execution records

 

 

———————————————

The shape statistics generated as a collective for each orchestration is as follows:

 

———————————————

———————————————

 

 

==================================================================

 

   

In summary, carefully decide while architecting your BizTalk solution for persistence points. As I’ve proved with this particular solution, the receiving file was debatched in the pipeline and this in turn created multiple instances of the BizTalk artifacts. As volume grew, so did the bottle neck – which directly affected the performance of the solution. Luckily, for this scenario the solution is pretty straight – but would it really be for every solution? Likely, not. So it is always better to performance tune your application before telling your boss  - it works (to a certain degree.. )

 

Hope it helps… Happy coding.