web stats
File Writer in SFTP mode errors out reading/writing large files - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 08-21-2019, 03:30 PM
sfurman sfurman is offline
Mirth Newb
 
Join Date: Jan 2015
Posts: 12
sfurman is on a distinguished road
Default File Writer in SFTP mode errors out reading/writing large files

Hi All,

I have a very basic channel that reads a local file and then writes it out to an sftp remote. The file is about 200M and the error I get pretty consistently is shown below. Is there any way to deal with this? I've increased heap size in mirth configs to 2G, but that does not seem to help.

Thank you,
Stan

---
[2019-08-20 06:34:53,748] ERROR (com.mirth.connect.donkey.server.channel.Channel:1 288): Error processing message in channel largeFileTest (86583101-d0ee-4982-bc30-7c3075de3463).
com.mirth.connect.donkey.server.channel.ChannelExc eption:
at com.mirth.connect.donkey.server.channel.Channel.di spatchRawMessage(Channel.java:1243)
at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchRawMessage(SourceConnector.java:192)
at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchRawMessage(SourceConnector.java:170)
at com.mirth.connect.connectors.file.FileReceiver.pro cessFile(FileReceiver.java:418)
at com.mirth.connect.connectors.file.FileReceiver.pro cessFiles(FileReceiver.java:328)
at com.mirth.connect.connectors.file.FileReceiver.pol l(FileReceiver.java:239)
at com.mirth.connect.donkey.server.channel.PollConnec torJob.execute(PollConnectorJob.java:49)
at org.quartz.core.JobRunShell.run(JobRunShell.java:2 13)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run (SimpleThreadPool.java:557)Caused by: com.mirth.connect.donkey.server.data.DonkeyDaoExce ption: java.sql.BatchUpdateException: Java exception: 'Java heap space: java.lang.OutOfMemoryError'.
at com.mirth.connect.donkey.server.data.jdbc.JdbcDao. executeBatchInsertMessageContent(JdbcDao.java:254)
at com.mirth.connect.donkey.server.data.buffered.Buff eredDao.executeTasks(BufferedDao.java:112)
at com.mirth.connect.donkey.server.data.buffered.Buff eredDao.commit(BufferedDao.java:85)
at com.mirth.connect.donkey.server.data.buffered.Buff eredDao.commit(BufferedDao.java:67)
at com.mirth.connect.donkey.server.channel.Channel.pr ocess(Channel.java:1723)
at com.mirth.connect.donkey.server.channel.Channel.di spatchRawMessage(Channel.java:1221)
... 8 moreCaused by: java.sql.BatchUpdateException: Java exception: 'Java heap space: java.lang.OutOfMemoryError'.
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl .newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl .newInstance(NativeConstructorAccessorImpl.java:62 )
at java.base/jdk.internal.reflect.DelegatingConstructorAccessor Impl.newInstance(DelegatingConstructorAccessorImpl .java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCalle r(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Construc tor.java:481)
at org.apache.derby.impl.jdbc.Util.newBatchUpdateExce ption(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeL argeBatch(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeB atch(Unknown Source)
at com.mirth.connect.donkey.server.data.jdbc.JdbcDao. executeBatchInsertMessageContent(JdbcDao.java:251)
... 13 moreCaused by: java.sql.SQLException: Java exception: 'Java heap space: java.lang.OutOfMemoryError'.
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.g etSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.newEmbedSQLExcepti on(Unknown Source)
at org.apache.derby.impl.jdbc.Util.javaException(Unkn own Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl .wrapInSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl .handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleE xception(Unknown Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleE xception(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeS tatement(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement. executeBatchElement(Unknown Source)
... 16 moreCaused by: java.sql.SQLException: Java exception: 'Java heap space: java.lang.OutOfMemoryError'.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.get SQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.w rapArgsForTransportAcrossDRDA(Unknown Source)
... 25 moreCaused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.derby.iapi.services.io.DynamicByteArray OutputStream.<init>(Unknown Source)
at org.apache.derby.impl.store.raw.data.BasePage.inse rtAllowOverflow(Unknown Source)
at org.apache.derby.impl.store.raw.data.BasePage.inse rt(Unknown Source)
at org.apache.derby.impl.store.access.heap.HeapContro ller.doInsert(Unknown Source)
at org.apache.derby.impl.store.access.heap.HeapContro ller.insertAndFetchLocation(Unknown Source)
at org.apache.derby.impl.sql.execute.RowChangerImpl.i nsertRow(Unknown Source)
at org.apache.derby.impl.sql.execute.InsertResultSet. normalInsertCore(Unknown Source)
at org.apache.derby.impl.sql.execute.InsertResultSet. open(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement .executeStmt(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement .execute(Unknown Source)
... 18 more
---
Reply With Quote
  #2  
Old 08-22-2019, 09:25 AM
agermano agermano is offline
Mirth Guru
 
Join Date: Apr 2017
Location: Indiana, USA
Posts: 1,095
agermano is on a distinguished road
Default

If you are only moving and not processing files, have you tried using an attachment?
  • Set your attachment option on the channel summary tab to "entire message." You can set the mime type in the attachment options.
  • Set your inbound and outbound types for all of your connectors to Raw.
  • Set your File Writer template to ${message.encodedData}
  • Make sure you have "Reattach Attachments" turned on (it should already be on by default.)

When you view your message in the browser, all you should see is the replacement token, but there should also be an "Attachments" tab where you can see your attachment.

With files this size, you may also want to check "Remove Attachments on Completion" under the storage options on your channel summary tab. In that case, you probably won't see the attachment tab in the message browser.

An alternate option that could be even faster, but more work for you, is to do it entirely in javascript. If you directly access the file and start your own sftp session with jsch (the library mirth uses) you could stream data from the file to the remote without needing to read the entire file into memory in the first place or store it in the mirth database. The only case in which a File Reader does not read the entire file into memory at once is if you have your inbound data type set to Delimited and are splitting the batch line by line.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 07:43 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Mirth Corporation