Performance tuning with BizTalk 2006

 BizTalk tuning is in itself a HUGE topic and there are various resources available on the internet. The aim of this article is to point out several factors one should consider in a BizTalk implementation from a hardware and software perspective also touching the concept of throttling. I’ve often found out that developers and infrastructure people have skipped the offerings on the table by BizTalk and hence – BizTalk becomes the blame box of the enterprise. It’s not a bad thing – as we consultants get invited to the party J

 ============================================================

 §   Functional testing + stress and performance testing is the key to deliver a solution

9     Understanding all the orchestration persistence points in your solution

9     If you are unsure, mock up a part of your solution and performance test it

 §      Don’t make an assumption – It is your enemy and would make you look bad

 ============================================================

I) Hardware

 §         If BizTalk + SQL is running on the same box è this box is only good for functional testing as the message box would become the bottle neck for any performance tests that you would do. One other reason is that SQL would try using as much memory available on the box to improve its performance

 §         When running on a 32 bit machine remember to set the /3GB and the /PAE switches on the SQL server

9     PAE: This enables SQL server to use the entire available memory (Is it good – u think about it?). This was a bug (fix) reported by Microsoft

 §         Counters to monitor – SQL

(i)           % Processor time

(ii)         Lock timeouts/sec

(iii)       Average wait time

(iv)        Lock wait/sec

 §         Two options are available to scaling a message box

(i)           Scale up

9     Run SQL server on a machine with more CPUs

 (ii)         Scale out

9     Message Boxes increased to use more servers

§         Ensure that the message publication is turned off on the master message box. This option is used to evaluate subscriptions and routes new messages to other message boxes.

§         Ensure that each SQL server is stored in its own drive to separate out the I/O

§         Ensure any extra SQL server used by BTS will also need to be clustered for availability

§         Large messages submitted into BizTalk can adversely affect performance (forward chaining)

9     Can overcome this problem by using virtual streams. This allows BizTalk to spool large messages (say 100 MB and over) to the disk system (sing virtual streams)

9     Advantageous as BizTalk won’t host the entire process in memory

9     Virtual Streams: The concept is that data is not loaded into the memory but instead persisted on the disk (Sample available in the Microsoft BizTalk SDK under …\Program Files\Microsoft BizTalk Server 2006\SDK\Samples\Pipelines\ArbitraryXPathPropertyHandler\VirtualStream.cs

9     BizTalk 2006 also introduced the concept of ScalableXPathDocument class that implements the IXPathNavigable interface.

9     In turn the XSL transforms the data by buffering the data to the disk

 

§         Each BizTalk host process runs under a configured service account

9     Each host process uses its temporary folder for a virtual stream

9     By default – this is not done for messages under 1 MB

 

§         For a large message processing scenario, attach a SCSI or SAN disk to the BizTalk server and the service account for the temp directory (used as the virtual stream)

 

§         Counters to monitor – BizTalk

(i)           Disk Queue length (often misleading L )

(ii)         % Disk Idle time

(iii)       Average Disk sec/read

(iv)        Avg

 

============================================================

 

II)      Software

 

§         Each BizTalk message box and logs should be separated out to a different volume

 

§         BizTalk tracking Db (BizTalkDtaDb) should be moved to a separate drive letter

 

§         BizTalk BAM Db (BAMPrimaryImport), if used,  should be moved to a separate drive letter

 

§         Separate out the host instances for receive, processing and send. (Refer to my blog entry comparing Single vs Multiple hosts in BizTalk 2006)

 

§         Use the database auto-growth feature

 

§         Ensure that BizTalk SQL server agent jobs for purge and archive are running successfully (important – as it directly affects the performance of your BizTalk databases)

 

§         Concurrent Web service calls

9     By default only two connections are allowed (Http 1.1 specification) – I hate to admit but it does make sense!

9     The effect is that the requests are queued after two subsequent active connection to the webservice

9     Options are to scale up using the BTSNTSvc config file

<system.net>

            <connectionManagement>

                        <add address = “*” maxConnections = “300” />

            </connectionManagement>

</system.net>

9     For the HTTP adapter there is registry setting to enable the number of outbound Http connections

9     Counters to use – Webservice performance counters

 

§         Tracking

9     Advisable to turn off tracking if it is not going to be used

9     Disable global tracking. This is done by opening up the BizTalkMgmtDb à adm_Group table à GlobalTrackingOption column. Set it to zero. (Caution: Please don’t play with these option on your live server directly. The logical approach would be analysing the effect it could have by running it on your test box)

9      If you are new to tracking, this would give you a simple start on this topic (How to track a message in BizTalk 2006)

 

 

 

 

 

§         Service Class Settings

9     This section referes to BizTalkMgmtDb à adm_ServiceClass class table

9     This is split into four different classes

(i)                 XLANGs (orchestrations)

(ii)               MessagingInProcess (artefacts like pipelines, adapters, etc)

(iii)             MSMQT

(iv)              Messaging Isolated

9     The two things of importance in this table are:

(i)                 Low/High Watermark: Explains the number of instances that can run on your server at a given time. When it touches the high scale à throttling happens to keep the instance numbers to the minimum low.

(ii)               Low/High Sessionmark: Specifies the concurrent Db sessions that can be persisted at any given time.

9     The above two settings are per server /per processor based visible to the OS

 

§         Cleaning the message box

9     Script available in the BizTalkMsgBoxDb called bts_CleanupMsgbox
       Remember to use exec bts_CleanupMsgBox and not execute (default parameter 0 will clear your BizTalk subscriptions as well)

9     If the script is empty, it can be opened from ..\ Program Files\Microsoft BizTalk Server 2006\Schema à  msgbox_cleanup_logic.sql

9     Ensure that all the host instances and dependencies are STOPPED including iis

9     Caution: Never run them on your live box before knowing about the consequences.

 

§         Monitoring

9     BizTalk sever MOM pack monitors a whole range of counters and raises alerts when certain counters exceed the threshold. (This is a MASSIVE topic in its own à and for simplicity purposes, lets talk about it some other day J)

 

§         Performance counters

9     Totally depends on what you would like to test (I’m in the process of writing another article about this area – so stay tuned)

9     One technique that is recommended is to test your solution under minimal load. Then perform a stress test on your solution to observe the changes and monitor familiar patterns to address issues that may arise.

============================================================

III)    Throttling

 

§         Consider a scenario where BizTalk manages an internal (spool or application) queue to process load because it cannot processes messages on the receive, processing or send side. This formation of the internal queue to handle load is called throttling.

 

§         Receive side throttling happens when the publication to the message box is being restricted.

9     Remember how we talked about the low/high watermark settings inside the BizTalkMgmtDb à adm_ServiceClass table?

 

§         Outcome depends on the adapter being used like MSMQ and SOAP, etc

9     There are several throttling states and can be monitored by looking at the message publishing state duration counter

(i)                 0: No throttling – This is the default level and means no throttling is happening.

 

(ii)               1: Rate – Messages are being received faster than they are published (receive throttling). This isn’t a BAD thing at all. Only raise a concern when this has been happening for long. Comparing the message publishing incoming rate vs the message publishing outgoing rate would provide a better picture.

 

Happens when: Incoming Rate > (Publishing rate) * (Overdrive factor)

The overdrive factor is set to 125 by default; meaning to allow 25% of what ever is being physically published

 

 

 

(iii)             3: Unprocessed message – The number of incoming messages delivered has reached the maximum configured threshold. This is generally the case when the orchestration engine is taking too long. Have a look at the In-process message count counter.

 

(iv)              4: Process Memory – Memory being utilized by the BizTalk host process has exceeded the configured limit

 Happens when: Memory of the host process > Process memory usage

 Counter to watch = process memory (MB)

 The 32-bit OS can address a maximum of 2GB memory regardless of the overall memory installed on the physical box. This memory in technical terms on the BizTalk box is being used by the underlying .NET framework to increase its footprint. Use the /3Gb switch in the boot.ini file to address 3 GB of memory. The default option in the admin console sets the process memory usage to 25. What this means, is that for a 1 GB RAM when the BizTalk process reaches 256 MB, memory throttling would be triggered.

 

 

                                                                 

 

 

 

 

 

 

(v)                5: System Memory – This represents the memory of the entire machine. This is set to 0 (disabled) by default. A value greater than 0 would mean detection of the value and a respective implementation for it.

 

 

(vi)             6: Database Size– Messages being published to the database are not being processed and a backlog is building up.

Happens when: DB Size > message count

Counter to watch for is the database size located in the Message Agent.

 

(vii)           8: Database Session Count – Indicates that too many database concurrent connections have been opened by the BizTalk host.
Counter to watch for is the database session in the Message Agent.
This is set to 0 (disabled) by default. A value greater than 0 would mean detection of the value and a respective implementation for it

 

(viii)           9: Thread Count – indicates the number of threads running.
This is set to 0 (disabled) by default. A value greater than 0 would mean restriction on the number of thread counts by the host instances.

 

(ix)            10 or 11: User Override – Indicates that the receive/publishing throttling has been overridden in the registry. 

============================

 Hope that gives you a starting point… Happy tuning J