<meta name='google-adsense-platform-account' content='ca-host-pub-1556223355139109'/> <meta name='google-adsense-platform-domain' content='blogspot.com'/> <!-- data-ad-client=ca-pub-4320963827702032 --> <!-- --><style type="text/css">@import url(https://www.blogger.com/static/v1/v-css/navbar/3334278262-classic.css); div.b-mobile {display:none;} </style> </head><body><script type="text/javascript"> function setAttributeOnload(object, attribute, val) { if(window.addEventListener) { window.addEventListener('load', function(){ object[attribute] = val; }, false); } else { window.attachEvent('onload', function(){ object[attribute] = val; }); } } </script> <div id="navbar-iframe-container"></div> <script type="text/javascript" src="https://apis.google.com/js/platform.js"></script> <script type="text/javascript"> gapi.load("gapi.iframes:gapi.iframes.style.bubble", function() { if (gapi.iframes && gapi.iframes.getContext) { gapi.iframes.getContext().openChild({ url: 'https://www.blogger.com/navbar.g?targetBlogID\x3d7256432\x26blogName\x3dThe+Frustrated+Programmer\x26publishMode\x3dPUBLISH_MODE_BLOGSPOT\x26navbarType\x3dBLACK\x26layoutType\x3dCLASSIC\x26searchRoot\x3dhttps://frustratedprogrammer.blogspot.com/search\x26blogLocale\x3den_US\x26v\x3d2\x26homepageUrl\x3dhttp://frustratedprogrammer.blogspot.com/\x26vt\x3d4213664491834773269', where: document.getElementById("navbar-iframe-container"), id: "navbar-iframe", messageHandlersFilter: gapi.iframes.CROSS_ORIGIN_IFRAMES_FILTER, messageHandlers: { 'blogger-ping': function() {} } }); } }); </script>

Promiscuous Performance Monitoring

| Sunday, November 14, 2004

At work we have a lot of issues with monitoring and logging. We run a lot of different services as EJBs in Weblogic clusters. They get a lot of use, so much so, that logging can actually make a real performance problem. The main issues are:
  • The logs are not aggregated in any one place.
  • There are a lot of ad-hoc scripts to do remote greps via ssh.
  • Any type of central aggregation will cause increased network load and increased CPU load on the servers.
While I was thinking about this, I remembered IBM used to have a HTTP log analyzer that, instead of pulling log files like most do, listened on a promiscuous ethernet device and built the logs itself, in real time. These logs could be analyzed and poked through at any time, without fear of bogging down the servers.

I'd like something like this for Java Applications. Something that could listen for known protocols like RMI, SOAP-RMI, etc. It could do performance montoring (avg., min, max, std. dev.) and also track error/exception rates. In my head these seems very possible, but maybe I'm missing something.

There are several products that use byte-code modification to instrument code, but this adds unnessesary overhead. I was quoted something like a 7% increase, and its still not doing the other half I need -- error analysis.

posted by Paul

at 11/14/2004 02:27:00 PM Permalink