Tuesday, 27 May 2008
Log4j SMTPAppender for email alerts
« Hello World | Main | Dont put a lot in IN. »We recently put a new application in production under a tight schedule. One of things we wanted to track was the number of errors which were happening at runtime,so we could get some feedback on the frequency with which they happened.
A couple of other members in the team had used log4j's smtpappender for this before. So we hooked it up to send us email alerts for each log.error statement in our code (Application errors). It was very interesting to see how our dependencies failed us and what bugs had gotten past QA.
Sample log4j configuration for the SMTP appender -
<appender name="SMTP" class="org.apache.log4j.net.SMTPAppender">
<errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
<param name="Threshold" value="${log4j.email.threshold}"/>
<param name="To" value="${log4j.email.to}"/>
<param name="From" value="${log4j.email.from}"/>
<param name="Subject" value="Service Error - ${machineid}"/>
<param name="SMTPHost" value="hqintmail.org.com"/>
<param name="BufferSize" value="10"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{ABSOLUTE} %-5p [%c] %m%n"/>
</layout>
</appender>
For example one of things we noticed was a couple of the webservices we depended on was not very reliable and crashed (returned no data) for atleast 10-15 times in the day !Since this was a customer facing webapp the errors were appropriately handled on the UI but having this alerting turned on, enabled us to proactively track issues before customers actually reported them. There were a couple of situations in which it didn't exactly do what we wanted. For example, when one of our enterprise databases went down for a couple of minutes, our inbox got flooded with alerts within minutes.
But on the whole, things were going well until our mail server changed. Log4j configuration still had the old mail server name. And since the appender could not send out the email alert, it relentlessly kept trying to do so making the box totally unresponsive and even dropping existing client connections Googling further on the problem I was able to find this (just thought I would pass it on) – java forum thread discussing log4j
So if you worry about high availability then this is one of the things which could expose a chink in your design and eventually we are moving away from using this as our alerting mechanism( though using asyncappender could mitigate some of the risk) as we wanted more features from our alerting.A couple of features which would be nice to have -
- Cumulative alerting - Tell me when a problem(database down) starts and when it subsides instead of each failed attempt.
- Selective alerting - Ability to ignore some alerts without requiring a code change
- Routing - Ability to route different kinds of alerts to different emails (out of the box).
- Performance and Availability - It should not affect the performance or the availability of the application.
- Statisitcal Analysis of Alerts - Basically fancy charts and historical analysis.
Technorati Tags: log4j smtpappender alerting mechanism
[Trackback URL for this entry]
