Any experience with large attachment in an SOA platform ?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Any experience with large attachment in an SOA platform ?

Jérémie Doucy
Hello,
We actually need to transfer large attachment through PEtALS, do you have any experience with this ?
We thought about using WS-ATTACHMENT, but how BC-SOAP deals with it ?

Thanks in advance.




Information Processing, Control and Cognition (IPCC)
EADS Defence and Security Systems
http://weblab-project.org




-------------------- m2f --------------------

Subscribe/Unsubscribe emails notifications.

Response to this email will be posted on the Petals forum.
Please delete the existing text before responding :)

Read the topic online:
http://forum.petalslink.com/viewtopic.php?p=30898#30898

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: Any experience with large attachment in an SOA platform ?

chamerling
The SOAP BC provides WS-Attachement and MTOM features. We did some tests months (years Smile) ago and I think that something like 100 Mb files were OK (need JVM tuning for such size!).
What is large file in your case?




-------------------- m2f --------------------

Subscribe/Unsubscribe emails notifications.

Response to this email will be posted on the Petals forum.
Please delete the existing text before responding :)

Read the topic online:
http://forum.petalslink.com/viewtopic.php?p=30899#30899

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: Any experience with large attachment in an SOA platform ?

Jérémie Doucy
In reply to this post by Jérémie Doucy
Could be as large as possible (videos for example) but we can image that 100 mo could be ok, I'm very interested about the JVM tuning you cite.
We can start with some test with 100 mo. But I think that ws-attachement is not the good solution for larger files, what do you think about it ?
If we need larger files, do you have any idea about how to manage this in a SOA platform ?




Information Processing, Control and Cognition (IPCC)
EADS Defence and Security Systems
http://weblab-project.org




-------------------- m2f --------------------

Subscribe/Unsubscribe emails notifications.

Response to this email will be posted on the Petals forum.
Please delete the existing text before responding :)

Read the topic online:
http://forum.petalslink.com/viewtopic.php?p=30900#30900

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: Any experience with large attachment in an SOA platform ?

chamerling
In reply to this post by Jérémie Doucy
In case of large files, I think that it is quite complicated to tune the JVM, petals and the components to handle all the files sizes... Not sure that is the petals' role to handle such data amount... There are some alternatives :
1. Just exchange URLs with petals between service consumers and providers and add something which deals with remote files... This 'breaks' the SOA approach but using a standard FTP client of some HTTP based commands will be more reliable, performant for such data.
2. Same as 1 but use the FTP component for example...
3. ...




-------------------- m2f --------------------

Subscribe/Unsubscribe emails notifications.

Response to this email will be posted on the Petals forum.
Please delete the existing text before responding :)

Read the topic online:
http://forum.petalslink.com/viewtopic.php?p=30901#30901

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: Any experience with large attachment in an SOA platform ?

Jérémie Doucy
In reply to this post by Jérémie Doucy
Thanks for the answers. We already use something like echanging URL between services, but as you said, it brokes the SOA approach.
I'm going to look at bc-ftp and try to fing a use case using it.

Have a nice day Wink.




Information Processing, Control and Cognition (IPCC)
EADS Defence and Security Systems
http://weblab-project.org




-------------------- m2f --------------------

Subscribe/Unsubscribe emails notifications.

Response to this email will be posted on the Petals forum.
Please delete the existing text before responding :)

Read the topic online:
http://forum.petalslink.com/viewtopic.php?p=30903#30903

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: Any experience with large attachment in an SOA platform ?

cdeneux-2
In reply to this post by Jérémie Doucy
I think that it's possible to porcess large file using stream capabilities. It seems to me that all files are processed as stream. So if we use a specific stream implemenatation that flushes the file on disk if its size is upper that a configurable value, the large files will be correctly supported without introducing other things.

On other optimisation point is: how to avoid several flushs of the same large file if the attachment is processed several times by different components on the same PEtALS instance ?


Christophe

2010/4/22 chamerling [hidden email])>
:
In case of large files, I think that it is quite complicated to tune the JVM, petals and the components to handle all the files sizes... Not sure that is the petals' role to handle such data amount... There are some alternatives :
1. Just exchange URLs with petals between service consumers and providers and add something which deals with remote files... This 'breaks' the SOA approach but using a standard FTP client of some HTTP based commands will be more reliable, performant for such data.
2. Same as 1 but use the FTP component for example...
3. ...







--Posted via petals mailing-list--




Christophe DENEUX / Capgemini Sud / M�diterran�e
Integration Architect / OW2 PEtALS Comitter
www.capgemini.com
Porte de l'Ar�nas - Entr�e B / 455 Promenade des Anglais / 06200 Nice / FRANCE
Join the Collaborative Business Experience
_______________________________________________________________________________




-------------------- m2f --------------------

Subscribe/Unsubscribe emails notifications.

Response to this email will be posted on the Petals forum.
Please delete the existing text before responding :)

Read the topic online:
http://forum.petalslink.com/viewtopic.php?p=30914#30914

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: Any experience with large attachment in an SOA platform ?

Charles.w
In reply to this post by Jérémie Doucy
Last year when I was writing the BC-SFTP component, I tested it with huge files attachements such a VM image (a few Gb of data if I remember well), in distributed mode (between a mac acting as SFTP server with a Petals ESB node on it and a Linux station with another Petals ESB node with a Filetransfer consume), and I never encountered any problem. The only big problem with file attachements is connection loss between the two petals node...




-------------------- m2f --------------------

Subscribe/Unsubscribe emails notifications.

Response to this email will be posted on the Petals forum.
Please delete the existing text before responding :)

Read the topic online:
http://forum.petalslink.com/viewtopic.php?p=30915#30915

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: Any experience with large attachment in an SOA platform ?

chamerling
In reply to this post by Jérémie Doucy
cdeneux wrote:


On other optimisation point is: how to avoid several flushs of the same large file if the attachment is processed several times by different components on the same PEtALS instance ?



This should not be a problem to implement and share things between container and components... Not so simple but YES WE CAN Smile




SOA Research Engineer (& more)
Petals Link Labs




-------------------- m2f --------------------

Subscribe/Unsubscribe emails notifications.

Response to this email will be posted on the Petals forum.
Please delete the existing text before responding :)

Read the topic online:
http://forum.petalslink.com/viewtopic.php?p=30916#30916

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: Any experience with large attachment in an SOA platform ?

cdeneux-2
In reply to this post by Jérémie Doucy
Hi,

Are you sure ? It seems to me that the attachments are stored only memory when they are exchanged betwwen containers. So you need to set your heap max size to a very big value to be able to exchange  few Gb. And on 32bits systems you it should not be possible to set the heap over 2Gb ?

Christophe



2010/4/24 ccasadei [hidden email])>
:
Last year when I was writing the BC-SFTP component, I tested it with huge files attachements such a VM image (a few Gb of data if I remember well), in distributed mode (between a mac acting as SFTP server with a Petals ESB node on it and a Linux station with another Petals ESB node with a Filetransfer consume), and I never encountered any problem. The only big problem with file attachements is connection loss between the two petals node...







--Posted via petals mailing-list--




Christophe DENEUX / Capgemini Sud / M�diterran�e
Integration Architect / OW2 PEtALS Comitter
www.capgemini.com
Porte de l'Ar�nas - Entr�e B / 455 Promenade des Anglais / 06200 Nice / FRANCE
Join the Collaborative Business Experience
_______________________________________________________________________________




-------------------- m2f --------------------

Subscribe/Unsubscribe emails notifications.

Response to this email will be posted on the Petals forum.
Please delete the existing text before responding :)

Read the topic online:
http://forum.petalslink.com/viewtopic.php?p=30917#30917

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: Any experience with large attachment in an SOA platform ?

cdeneux-2
In reply to this post by Jérémie Doucy
It's not very difficult.
On the reception side, just create an InputStream as Wrapper of 2 "buffers":
    - one base on the existing one, holding all bytes in memory,
    - another one based on a FileOutputStream, saving all bytes on disk. This buffer is used in combination with a size threshold.
On the sending side, we can use same concepts


Christophe




2010/4/26 chamerling [hidden email])>
:
cdeneux wrote:

On other optimisation point is: how to avoid several flushs of the same large file if the attachment is processed several times by different components on the same PEtALS instance ?




This should not be a problem to implement and share things between container and components... Not so simple but YES WE CAN [img][/img]



SOA Research Engineer (& more)
Petals Link Labs







--Posted via petals mailing-list--




Christophe DENEUX / Capgemini Sud / M�diterran�e
Integration Architect / OW2 PEtALS Comitter
www.capgemini.com
Porte de l'Ar�nas - Entr�e B / 455 Promenade des Anglais / 06200 Nice / FRANCE
Join the Collaborative Business Experience
_______________________________________________________________________________




-------------------- m2f --------------------

Subscribe/Unsubscribe emails notifications.

Response to this email will be posted on the Petals forum.
Please delete the existing text before responding :)

Read the topic online:
http://forum.petalslink.com/viewtopic.php?p=30918#30918

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: Any experience with large attachment in an SOA platform ?

rnaudin
In reply to this post by Jérémie Doucy
Hi,

You're right Christophe, the attachments or sources in the Petals components are handled mostly 'in memory'. There is only the File Transfer that use the 'FileInputStream' Stream handler.
It's a good idea to introduce a Stream that mix a memory buffer backed by a file once full.

About several component that process the resources, it is the same as actually, the Stream is consumed and then copied if required.




-------------------- m2f --------------------

Subscribe/Unsubscribe emails notifications.

Response to this email will be posted on the Petals forum.
Please delete the existing text before responding :)

Read the topic online:
http://forum.petalslink.com/viewtopic.php?p=30919#30919

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: Any experience with large attachment in an SOA platform ?

Charles.w
In reply to this post by Jérémie Doucy
Hi,

Actually, I didn't remember the configuration I was using at the time...the only thing I can remember is that I was working with 64bits VM on the server side. I should retry this, and in fact I will, since, for a client project, we need to know what is the maximum amount of data we can pass per request with an acceptable delay for *almost* real time user manipulation.

cdeneux wrote:
Hi,

Are you sure ? It seems to me that the attachments are stored only memory when they are exchanged betwwen containers. So you need to set your heap max size to a very big value to be able to exchange� few Gb. And on 32bits systems you it should not be possible to set the heap over 2Gb ?

Christophe



2010/4/24 ccasadei [hidden email])>
Quote:
Last year when I was writing the BC-SFTP component, I tested it with huge files attachements such a VM image (a few Gb of data if I remember well), in distributed mode (between a mac acting as SFTP server with a Petals ESB node on it and a Linux station with another Petals ESB node with a Filetransfer consume), and I never encountered any problem. The only big problem with file attachements is connection loss between the two petals node...







--Posted via petals mailing-list--




-------------------- m2f --------------------

Subscribe/Unsubscribe emails notifications.

Response to this email will be posted on the Petals forum.
Please delete the existing text before responding :)

Read the topic online:
http://forum.petalslink.com/viewtopic.php?p=30920#30920

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general