PEtALS monitoring & Administration

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

PEtALS monitoring & Administration

Mathieu Le Breton
Administrator
This post has NOT been accepted by the mailing list yet.
==Posted by: Christophe Deneux==
This first message was previously posted in an other topic.

About monitoring, we are thinking about an agile solution to monitor PEtALS.

Our production context is a national distributed PEtALS network with 10 PEtALS nodes today and 20 in few months, over a WAN with a lot of firewalls.

So our first solution is to use PEtALS to monitor itself:

    A new SE (as petals-sample-client) will provide a GUI, so it's not needed to change network box configuration to access all PEtALS nodes. The accesses will be done through the NMR,
    A timer in the CDK will send InRobust message exchange containing statitistic about services/endpoints/... to the GUI component,
    A new BC (petals-bc-kernel) will provide some services to interact with kernel layer. This BC will interact with the kernel of the PEtALS node on which it is installed through JMX.


The main benefits of the solution are:

    no need to change network box configuration betwwen GUI and PEtALS,
    the monitoring application is deployed as all JBI artifacts,
    high availibity using PEtALS HA feature,
    monitoring infrastructure very agile.

==Posted by: cdeneux==
PostPosted: 30 Sep 2008 03:14 pm - email : christophe.deneux@petalslink.com   Reply with quote Edit/Delete this post Delete this post View IP address of poster
According to a phone call with Christophe Hamerling, it seems interresting to use WSDM (Web Service Distributed Management) to monitor/administrate the services deployed in the container:

    WSDM-MOWS (Management Of Web service) should be used to manage services,
    WSDM-MUWS (Management Using Web service) should be used to manage resources connected to/embedded in BC.

==Posted by: chamerling==
Our first idea was to use WSDM to manage PEtALS in the same way than managing it with JMX :
- Expose 'the same things' than JMX (in my opinion, extend the management features even if not JBI compliant)
- Use a standard API with SOAP messaging in order to avoid dealing with JMX which is not very 'developer-friendly'
- Possibly agregate management/monitoring data coming from different domains withtout any firewall issues (SOAP/HTTP should be nice)...
- ...

One other aspect is to introduce WS-notification instead of JMX notification for the same reasons.

Last point, for the messages you want to transmit periodically from a component to a 'supervisor' (your GUI), we are planning to develop the notifications in the bus so that endpoints can publish/subscribe to notifications based on rules.


==Posted by: cdeneux===
So you need to include into PEtALS something like an HTTP server to be able to join these web-services. I think that:

    it is not a good idea to add other server into the container,
    in a customer point of view, the JMX API is the most interesting because it is the same channel as the JBI administration API.

If you want a WSDM API to manage PEtALS, imagine that my petals-bc-kernel implements WSDM, you need only to deploy a SA for petals-bc-soap to expose the services of the petals-bc-kernel. So PEtALS will remain a light container.
Not only the container should be manageable using WSDM, but also service deployed on each nodes. Perhaps it should be interesting to have a WSDM API in my GUI component.
Quote:
Our first idea was to use WSDM to manage PEtALS in the same way than managing it with JMX

I agree with publish/subscribe.
Back to top
View user's profile Send private message Send e-mail Visit poster's website

==Posted by: alouis==
Guest
I think that the administration of the container should remains in the hand of a JMX administrator.
I don’t see any use case (in the real life) where this “http” management protocol could be used?


If you want a WSDM API to manage PEtALS, imagine that my petals-bc-kernel implements WSDM, you need only to deploy a SA for petals-bc-soap to expose the services of the petals-bc-kernel. So PEtALS will remain a light container.
Not only the container should be manageable using WSDM, but also service deployed on each nodes. Perhaps it should be interesting to have a WSDM API in my GUI component.

Quote:
Our first idea was to use WSDM to manage PEtALS in the same way than managing it with JMX


I agree with publish/subscribe.
Petals Link
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

fgardes
Maybe implements it into the open source project wsstar, and use it into the CDK. No?




-------------------- m2f --------------------

Subscribe/Unsubscribe emails notifications.

Response to this email will be posted on the Petals forum.
Please delete the existing text before responding :)

Read the topic online:
http://forum.petalslink.com/viewtopic.php?p=31158#31158

-------------------- m2f --------------------




_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
A quick update on WS-Adressing support : IMO it is a little boring to handle WSA data in DocumentFragment. I suggest to use, for the WSA support, the WSA QNAME=String pattern. CocumentFragments will be used for everything else.

chamerling wrote:
cdeneux wrote:
According to JBI specifications, page 40, §5.5.1.1.3, the SOAP header must be saved into the property named javax.jbi.messaging.protocol.headers as a Map. Each SOAP header child is an element of the Map as org.w3c.dom.DocumentFragment. JBI specifications does not defined how to name the map elements. I propose you to use the qualified name of the SOAP header child element.


Maybe an evolution of the CDK will be to provide an API based on the 5.5.1.1.3 section :
- message.setSecuritySubject(Subject s) will set the javax.jbi.security.subject property
- message.setProtocolType(String s) will set the protocol type property
- message.setHeaders(Map m) will set the headers properties.




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=237#237

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
I propose you to use this post to specify the Monitoring a Administration features. It will be updated according our thought maturity for these features. Don't hesitate to contribute replying to this post !

About Administration feature

  1. About exposing PEtALS kernel JMX API using other protocols

    • For network configuration/security reasons or management console usage, it could be interesting to expose the PEtALS Kernel JMX API using protocols as HTTP, SNMP, WSDM.
    • These protocols don't have to be included in the PEtALS Kernel because of:

      • to avoid a source code complexity increase of the PEtALS kernel,
      • to have PEtALS a light process,
      • to avoid configuration conflict between components deployed into the container (ie: TCP port, ...)


    Proposed solution wrote:


    • A new BC (petals-bc-kernel) could be created to expose the PEtALS kernel JMX API as a service on NMR (each API method will be an operation of the service). So this service could be exposed outside PEtALS using petals-bc-soap for example, or a new compodent "petals-bc-snmp",
    • petals-bc-kernel services will be compliant with WSDM, so in combination with petals-bc-soap we will have a support of WSDM managers (a SA can be delivered to expose theses services),
    • To be reusable with other JBI container, petals-bc-kernel will expose the JBI management API if it runs with a JBI container that is not PEtALS, and expose the PEtALS specific API if it runs with PEtALS.



About Monitoring feature
  1. The monitoring feature includes a statistics part. Two kinds of statistics:

    • common statistics:

      • number of messages (failed or sucessfull) per container, per exchange pattern, per interface, per service, per endpoint, during a given time (10 minutes),
      • minimum response time, maximum response time, average response time, percentile 90, percentile 10, during a given time (10 minutes),
      • ...

    • component specific statistics:

      • petals-se-orchestra: number of pending processes, ...
      • petals-bc-soap: number of requests rejected by Rampart for security reason, ...
      • ...

      Any statistics information must be lost, particularly statistics available per duration.

  2. Monitoring API should be based on WSDM.


    Proposed solution wrote:


    • Common statistics should be computed by the PEtALS kernel and exposed using its JMX API (and so petals-bc-kernel). A solution should be found concerning the storage of periodic statistics into the kernel waiting their consumption, perhaps using MEP "InRobust ",
    • Component specific statistics should be included in the CDK using WSDM.



About Administration & Monitoring GUI feature

  • A centralized GUI is needed to avoid switching between each PEtALS of a PEtALS network,
  • The GUI requires minimal network configuration (only one TCP port with one PEtALS node for example)

Proposed solution wrote:


  • To have the minimal network configuration required by the GUI, it will be interesting to propose the GUI inside a SE (as the petals-sample-client), so all communication needs will be assumed by the NMR. The customer will only configure the link between the SE and its machine.
  • The GUI could be an embedded web-application, based on the current web-console.





-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=209#209

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
In reply to this post by cdeneux-4
I am totally agree with you on using WSDM for monitoring and management.
Here are the links on OASIS WSDM working group (http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=wsdm):

Management Of Web Services

    * Management Using Web Services
    [url=http://www.oasis-open.org/committees/download.php/20576/wsdm-muws1-1.1-spec-os-01.pdf ]http://www.oasis-open.org/committees/download.php/20576/wsdm-muws1-1.1-spec-os-01.pdf [/url]
    [url=http://www.oasis-open.org/committees/download.php/20575/wsdm-muws2-1.1-spec-os-01.pdf ]http://www.oasis-open.org/committees/download.php/20575/wsdm-muws2-1.1-spec-os-01.pdf [/url]

    * Management Of Web Services
    [url=http://www.oasis-open.org/committees/download.php/20574/wsdm-mows-1.1-spec-os-01.pdf ]http://www.oasis-open.org/committees/download.php/20574/wsdm-mows-1.1-spec-os-01.pdf [/url]


For the monitoring and management console, my first impression will be a web based application embedded in a service engine. We need to see if it is an extension of the current WebConsole or a new one...




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=210#210

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
In reply to this post by cdeneux-4
Can you elaborate more ?
chamerling wrote:
For the monitoring and management console, my first impression will be a web based application embedded in a service engine. We need to see if it is an extension of the current WebConsole or a new one...




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=211#211

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
1. A do not think that a Swing based application is a good thing to start in a Service Engine. The main reason is that the X server could not be available on some hosts and if it is you need to forward the X display from the server to the host where you are monitoring the system -> IMO this is an unnecessary network traffic.
So I am for a web based service engine even if it will use a new port to start a Jetty server or whatever...

2. The current webconsole is not sufficient, we need to work on it and so I am not sure that the new monitoring tool will be a WebConsole extension...




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=212#212

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
In reply to this post by cdeneux-4
I agree with you. I add these explanations to the specification post.
What do you think to regard the SE as a web-application container, on which web-applications are deployed as SU ?.
chamerling wrote:
1. A do not think that a Swing based application is a good thing to start in a Service Engine. The main reason is that the X server could not be available on some hosts and if it is you need to forward the X display from the server to the host where you are monitoring the system -> IMO this is an unnecessary network traffic.
So I am for a web based service engine even if it will use a new port to start a Jetty server or whatever...

If the new monitoring tool is a extension of the WebConsole, it will need to deploy a SA on a BC to access all WSDM services available on the NMR. I think that it is possible to update the current webconsole to use NMR calls instead of JMX calls.
Perhaps could we:
  • define an interface between presentation layer and business layer of the web-console,
  • create a logical layer based on NMR calls and a other logical layer based on JMX calls,
  • keep the current presentation layer (completed with a new part)
  • package both web-console (one per logical layer).

chamerling wrote:
2. The current webconsole is not sufficient, we need to work on it and so I am not sure that the new monitoring tool will be a WebConsole extension...




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=213#213

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
cdeneux wrote:
I agree with you. I add these explanations to the specification post.
What do you think to regard the SE as a web-application container, on which web-applications are deployed as SU ?.


I think that it is a quite easy component to do since it is quite easy to do with Jetty Wink
We just need to inject some JBI contexts (such as the component context) in the Web Application one to be able to use them (not so complicated too).




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=214#214

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
In reply to this post by cdeneux-4
I have just updated specification post about how to handle WSDM into normalized messages. What are you thinking about it ?




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=215#215

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
You are right when you said that the SOAP header must be splited into several properties. The current mechanism is not very useful and your proposition is.

Note that I have submitted a feature request about Ws-Addressing handling (http://forge.objectweb.org/tracker/index.php?func=detail&aid=311338&group_id=213&atid=350260) which is 'linked' to what you describe here. I want to add the capability to use the wsa:To element to choose the destination service.




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=216#216

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
What do you think to use Apache MUSE as WSDM framework ?

On other interesting link about WSDM inside Eclipse: http://www.eclipse.org/tptp/monitoring/documents/tutorials/tptp_wsdm_setup_4.3.html




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=222#222

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
It seems to me that Apache MUSE is not usable because it works at SOAP level.

I added some information returned by monitoring: state information.
I proposed an implementation of WSDM into the CDK. What do you think about it ?




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=224#224

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
In reply to this post by cdeneux-4
cdeneux wrote:
It seems to me that Apache MUSE is not usable because it works at SOAP level.


Sorry, no time to check this, let's assume that it is true Wink

cdeneux wrote:
I added some information returned by monitoring: state information.
I proposed an implementation of WSDM into the CDK. What do you think about it ?


You have proposed to create a new worker pool for WSDM workers : Since this is a monitoring and management feature I also think that it is a good choice.
I just want to add that maybe we can have a default WSDM worker implementation provided by the CDK which is engaged or not. It can for example return the component statistics (nb of requests, fault, ...).
An abstract class can also be specified to extend the features.




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=225#225

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
Yes, instead of an abstract class, a basic implementation will be provided by the CDK. This implementation could be superseded by the component. Why do you want to be able to engage/disengage it ? I think it should be always engaged (the default implementation if not implementation is configured in the component JBI descriptor)

In my mind, statistics as you have mentioned should be provided by the container. So you can have stats even if the component is not able to provide them.

chamerling wrote:
I just want to add that maybe we can have a default WSDM worker implementation provided by the CDK which is engaged or not. It can for example return the component statistics (nb of requests, fault, ...).
An abstract class can also be specified to extend the features.




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=226#226

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
In reply to this post by cdeneux-4
According to JBI specifications, page 40, §5.5.1.1.3, the SOAP header must be saved into the property named javax.jbi.messaging.protocol.headers as a Map. Each SOAP header child is an element of the Map as org.w3c.dom.DocumentFragment. JBI specifications does not defined how to name the map elements. I propose you to use the qualified name of the SOAP header child element.

chamerling wrote:
You are right when you said that the SOAP header must be splited into several properties. The current mechanism is not very useful and your proposition is.

Note that I have submitted a feature request about Ws-Addressing handling (http://forge.objectweb.org/tracker/index.php?func=detail&aid=311338&group_id=213&atid=350260) which is 'linked' to what you describe here. I want to add the capability to use the wsa:To element to choose the destination service.




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=228#228

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
cdeneux wrote:
According to JBI specifications, page 40, §5.5.1.1.3, the SOAP header must be saved into the property named javax.jbi.messaging.protocol.headers as a Map. Each SOAP header child is an element of the Map as org.w3c.dom.DocumentFragment. JBI specifications does not defined how to name the map elements. I propose you to use the qualified name of the SOAP header child element.


Maybe an evolution of the CDK will be to provide an API based on the 5.5.1.1.3 section :
- message.setSecuritySubject(Subject s) will set the javax.jbi.security.subject property
- message.setProtocolType(String s) will set the protocol type property
- message.setHeaders(Map m) will set the headers properties.




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=230#230

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
cdeneux wrote:

In my mind, statistics as you have mentioned should be provided by the container. So you can have stats even if the component is not able to provide them.


Right but imagine you want to deploy the component on another JBI container? (Same as the WSDM in a component).
I do not think that some stats at the component level (CDK) are bad.




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=231#231

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
Yes, but no class wrapping a normalized message exists, only a message exchange wrapper. So, perhaps this CDK update can be independent of this feature. If you agree, can you add an entry to the feature request tarcker ?
chamerling wrote:
Maybe an evolution of the CDK will be to provide an API based on the 5.5.1.1.3 section :
- message.setSecuritySubject(Subject s) will set the javax.jbi.security.subject property
- message.setProtocolType(String s) will set the protocol type property
- message.setHeaders(Map m) will set the headers properties.




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=233#233

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
Reply | Threaded
Open this post in threaded view
|

Re: PEtALS monitoring & Administration

cdeneux-4
In reply to this post by cdeneux-4
I agree with you if we can consider to compute statistics about message exchanges into the kernel, and to compute statistics about service requests in components (CDK). Do you agree ?
chamerling wrote:
Right but imagine you want to deploy the component on another JBI container? (Same as the WSDM in a component).
I do not think that some stats at the component level (CDK) are bad.




-------------------- m2f --------------------

Read this forum topic online here:
http://petals.ebmwebsourcing.com/forum/viewtopic.php?p=234#234

-------------------- m2f --------------------

_______________________________________________
General mailing list
[hidden email]
http://forum-list.petalslink.org/cgi-bin/mailman/listinfo/general
12