web stats
Mirth stops responding to LLP messages - Mirth Community

Go Back   Mirth Community > Mirth Connect > Development

Reply
 
Thread Tools Display Modes
  #1  
Old 11-25-2006, 10:58 PM
mnowlin mnowlin is offline
OBX.3 Kenobi
 
Join Date: Sep 2006
Posts: 142
mnowlin is an unknown quantity at this point
Default Mirth stops responding to LLP messages

Interesting problem I'm scratching my head over...

I have a Mirth LLP channel that receives lab results from our LIS, does some voodoo (nothing really strange), and stores the messages into a database. Most of the time, it works. Sometimes, it breaks and doesn't acknowledge messages, causing a timeout on the LIS side.

As far as I know, this only happens when Mirth is receiving several messages quickly, at least 3-4 per second. The first few messages in the TCP connection are ACK'd correctly, then it suddenly stops. The LIS holds the TCP conn open, resends after 30 seconds (still no ACK), and repeats two more times before closing the connection.

I frequently see "no start of message indicator received" exceptions - not entirely sure these are triggered on this channel, but I suspect it.

I have tcpdump recordings that show that the packets/messages are correctly formatted, and you can see where Mirth stops responding.

Running in super-duper debug mode, I spotted the following Java exception about the same time (the same HH:MM:SS timestamp) it failed to ACK a message:

Code:
ERROR 2006-11-25 11:34:50,632 [9fe8a1c4-58a8-425c-8dc2-bb453b9a8791_destination_4_connector.dispatcher.10] com.webreach.mirth.server.controllers.MessageObjectController: could not log message: id=3d05ad9c-b66e-4690-940d-6275a5fe6d26
java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextEntry(HashMap.java:841)
        at java.util.HashMap$EntryIterator.next(HashMap.java:883)
        at java.util.HashMap$EntryIterator.next(HashMap.java:881)
        at com.webreach.mirth.server.controllers.MessageObjectController.updateMessage(MessageObjectController.java:121)
        at com.webreach.mirth.server.mule.transformers.JavaScriptTransformer.evaluateFilterScript(JavaScriptTransformer.java:229)
        at com.webreach.mirth.server.mule.transformers.JavaScriptTransformer.doTransform(JavaScriptTransformer.java:173)
        at org.mule.transformers.AbstractTransformer.transform(AbstractTransformer.java:197)
        at org.mule.impl.MuleEvent.getTransformedMessage(MuleEvent.java:251)
        at org.mule.providers.file.FileMessageDispatcher.doDispatch(FileMessageDispatcher.java:68)
        at org.mule.providers.AbstractMessageDispatcher$Worker.run(AbstractMessageDispatcher.java:257)
        at org.mule.impl.work.WorkerContext.run(WorkerContext.java:290)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:595)

I've found this exception seven other times in today's log, but still need to figure out if these correspond to Mirth failing to ACK messages at the same time.

I'm running Mirth 1.2.0 (also happened on 1.1.0) under Java 1.5.0_08 on a Linux 2.6.9 kernel.

I'm wondering if the java.util.ConcurrentModificationException might be caused by something in my transformer being non-thread-friendly, for lack of a better description.

Any ideas?

thanks - mike
Reply With Quote
  #2  
Old 11-26-2006, 10:40 AM
chrisl chrisl is offline
Mirth Employee
 
Join Date: Aug 2006
Posts: 967
chrisl is an unknown quantity at this point
Default Re: Mirth stops responding to LLP messages

Thanks for reporting this Mike!

This is due to the status of a message being updated in the database when the transformer is running. This introduces threading issues with multiple messages.


We have resolved this issue by removing the database updates during transformation and only updating at the end of the message chain (or on error). Additionally, we have refined the threading and database locking that we do in the controllers to ensure this doesn't happen elsewhere.

These fixes will be available in 1.2.2 which is scheduled for next week.

Thanks!
-Chris
__________________
Chris Lang
Reply With Quote
  #3  
Old 11-26-2006, 08:51 PM
mnowlin mnowlin is offline
OBX.3 Kenobi
 
Join Date: Sep 2006
Posts: 142
mnowlin is an unknown quantity at this point
Default Re: Mirth stops responding to LLP messages

If I have the vendor insert a two-second pause after receiving an ACK before sending the next message, you think that might work around the problem for now?

thanks - mike
Reply With Quote
  #4  
Old 11-27-2006, 10:03 AM
chrisl chrisl is offline
Mirth Employee
 
Join Date: Aug 2006
Posts: 967
chrisl is an unknown quantity at this point
Default Re: Mirth stops responding to LLP messages

Yes, that MIGHT work, however we are working hard to have a new release by the end of this week that will address the DB problems.


Thanks,
-Chris
__________________
Chris Lang
Reply With Quote
  #5  
Old 12-01-2006, 10:24 PM
mnowlin mnowlin is offline
OBX.3 Kenobi
 
Join Date: Sep 2006
Posts: 142
mnowlin is an unknown quantity at this point
Default Re: Mirth stops responding to LLP messages

Just an update - I had the vendor insert a two-second delay as described above. It improved things significantly, but we still get 1-2 errors a day (instead of 50+) - these are manually requeued from an error report. At least people aren't panicked any more...

Now I just need to wait for 1.2.2 and try to calm my boss - he's wants to stick with the current situation since he's worried that upgrading might introduce new problems....

mike
Reply With Quote
  #6  
Old 12-02-2006, 09:34 AM
chrisl chrisl is offline
Mirth Employee
 
Join Date: Aug 2006
Posts: 967
chrisl is an unknown quantity at this point
Default Re: Mirth stops responding to LLP messages

1.3 (1.2.2) is a huge improvement. It includes the same functionality as 1.2, but with faster and more stable performance.
__________________
Chris Lang
Reply With Quote
  #7  
Old 12-20-2006, 11:20 PM
mnowlin mnowlin is offline
OBX.3 Kenobi
 
Join Date: Sep 2006
Posts: 142
mnowlin is an unknown quantity at this point
Default Re: Mirth stops responding to LLP messages

Update:

I have two copies of 1.3 running, set up to receive and process messages the same way the current 1.2 version is doing. One of them is a set of new, fast systems, the other is a set of relatively older hardware - something that should have been put out to pasture long ago. (The 1.2 instance has additional LLP sender destinations to each of the 1.3 instances added to the channel definition.) I'll let it run for a few days, then will compare the output (Postgres tables) of each Mirth instance to ensure that everything matches 1:1.

mike
Reply With Quote
  #8  
Old 12-29-2006, 10:39 PM
mnowlin mnowlin is offline
OBX.3 Kenobi
 
Join Date: Sep 2006
Posts: 142
mnowlin is an unknown quantity at this point
Default Re: Mirth stops responding to LLP messages

1.3 testing has been going well, but I did run across a somewhat similar error to the original 1.2 bug. Only happened once so far (out of 50+K messages), but should this be looked at further?

I'm using Postgres as the internal database now, if that makes any difference.

Error writing to the database
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.j ava:787)
at java.util.HashMap$EntryIterator.next(HashMap.java: 829)
at java.util.HashMap$EntryIterator.next(HashMap.java: 827)
at org.mule.providers.TemplateValueReplacer.loadConte xt(TemplateValueReplacer.java:55)
at org.mule.providers.TemplateValueReplacer.replaceVa lues(TemplateValueReplacer.java:29)
at org.mule.providers.jdbc.JdbcUtils.getParams(JdbcUt ils.java:106)
at org.mule.providers.jdbc.JdbcMessageDispatcher.doDi spatch(JdbcMessageDispatcher.java:113)
at org.mule.providers.AbstractMessageDispatcher$Worke r.run(AbstractMessageDispatcher.java:257)
at org.mule.impl.work.WorkerContext.run(WorkerContext .java:290)
at edu.emory.mathcs.backport.java.util.concurrent.Thr eadPoolExecutor$Worker.runTask(ThreadPoolExecutor. java:650)
at edu.emory.mathcs.backport.java.util.concurrent.Thr eadPoolExecutor$Worker.run(ThreadPoolExecutor.java :675)
at java.lang.Thread.run(Thread.java:595)


Thanks - mike

Reply With Quote
  #9  
Old 12-29-2006, 11:36 PM
chrisl chrisl is offline
Mirth Employee
 
Join Date: Aug 2006
Posts: 967
chrisl is an unknown quantity at this point
Default Re: Mirth stops responding to LLP messages

Ah very interesting...it appears that values are being modified in a hash-map while Mirth is trying to iterate through it. This points to an area where we need to synchronize the threads. Can you add this to the JIRA? I'll try to get a fix for this in 1.3.2 (coming withing a week).

Thanks!
-Chris
__________________
Chris Lang
Reply With Quote
  #10  
Old 12-30-2006, 08:37 PM
mnowlin mnowlin is offline
OBX.3 Kenobi
 
Join Date: Sep 2006
Posts: 142
mnowlin is an unknown quantity at this point
Default Re: Mirth stops responding to LLP messages

Added - Mirth-224.

Thanks for looking into this. Other than this one bug (and having to fix some of my sloppy transformers), 1.3.1 has performed admirably.

mike
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Mirth 1.5 WIndows XP.Channel stops Frequently rkedilaya Support 0 02-18-2009 08:33 PM
Responding to TCP listener with data cwinters Support 5 09-25-2008 09:02 AM
Bad MSH stops Mirth processing jeffkeller Support 5 06-18-2007 04:42 AM
Mirth Admin stops loading on Loading Components enzo Support 3 05-10-2007 02:34 PM
Mirt stops responding after an exception nshaik Support 2 09-14-2006 05:39 AM


All times are GMT -8. The time now is 01:34 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Mirth Corporation