Consulting with one of my colleague on a BizTalk assignment, we came across a performance question while calling web-services from a BizTalk application. The application was quite simple in BizTalk terms doing the following:
1) Input is an xml file (around 1KB in size)
2) BizTalk consumes the file and sends it to a web-service (SOAP Adapter)
3) The web-service is a dummy service that just returns back “Hello world”
We measured the overall response time for a load of 25, 50 and 100 concurrent users. The BizTalk solution consists of a simple orchestration that receives the message à calls the web-service and logs the response. To our surprise, for a load of about 100 users the web-service response time increased to about 2.42 seconds – which is unacceptable!
After looking on to several fronts and performance tuning with the recommendations the response time was no better! Some of the major steps we under took during the process are as follows:
· Increase the concurrent connection from 80 to 300 concurrent
· Increased the SQL Max. Worker thread from 750 to 1600
· Added the scope in the BizTalk Orchestration to reduce the no. of persistence points.
· Distribute the host instances (Receive, Processing and Send)
· All other points as mentioned in the blog entry (Click here)
Now the average response time decreased from 2.42 seconds to 2.32 (an improvement of 4% – still unacceptable!).
Finally, we thought about removing the SOAP adapterand calling the services inline from C# code. This time around we could process 100 concurrent users with an average response time of about 1.1 seconds (improving 54% from our original time). Here is the screen shot using load runner.
The problem is that both the Orchestration Engine and the SOAP/HTTP stack share the same .NET process, and hence the same threadpool. For many scenarios, this might not be an issue. For inline Web service/HTTP sends, however, it often is. Both the Orchestration Engine and the SOAP/HTTP stacks make aggressive use of the .NET threadpool.
Any threads that have not been used for 40 seconds will be deleted from the pool. When adding threads to the threadpool, it tries to avoid creating threads. The threadpool will try to create only one thread every 500 milliseconds. Under high-load conditions there is a good chance that you will run into this issue – say incase the thread pool has about 100 requests lined up.
Control the number of thread in the thread pool by using the following registry key – typicall setting the minimum value to the maximum.
Registry Key: HKLM\SYSTEM\CurrentControlSet\Services\BTSSvc$BizTalkServerApplication\CLR Hosting
Look into process isolation – this would using a different instance of the .NET threadpool executed in a separate address space from the BizTalk NT service. This approach provides a high level of isolation. Because you are in complete control of the work loaded onto the threadpool, its behavior is more predictable. This approach uses the COM+ application hosting model to avoid the issue of few threads. The other benefit it provides is disposing the transport component and the message after the method returns.
With a short time on our hand (about half a day!) – Option 1 served our purpose plus replacing the SOAP adapter with the inline webservice calls.
I can hear some of you saying, “Use the WCF adapter” – yeah – that’s another food for thought.
Thats all folks 🙂
Other Useful Posts
Happy coding J
1) Webservices and the 15 second delay
2) Microsoft BizTalk Server Performance Optimization Guide
Link: Click here
3) Troubleshooting MessageBox Latency Issues
Link: Click here
4) Identifying BizTalk Bottlenecks
Link: Click here
Read Full Post »