Friday 27 May 2011

System.Messaging.MessageQueueException: Insufficient resources to perform operation.

I came across this error today in one our web service applications:-

System.Messaging.MessageQueueException: Insufficient resources to perform operation.
   at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
   at System.Messaging.MessageQueue.Send(Object obj, String label, MessageQueueTransaction transaction, MessageQueueTransactionType transactionType)
   at System.Messaging.MessageQueue.Send(Object obj, String label)

The service causing the error is a data import service.  It collects requests received during the day into an MSMQ and then processes it overnight.

The queue of requests only had about 10 messages in it, so was unlikely to be causing any resource issues.

However, when the data import service fails to process a message overnight, it sends it to a poison queue.  That queue had lots of messages in it.  At the moment the poison queue is left to fill up, we only use it when we’re fixing a specific problem with the import service.  Neither queue had a value set for “Limit message storage”.

It turns out that there’s a machine setting for limiting the amount of disk space given to MSMQ for storing messages.  It defaults to 1GB and sure enough the C:\WINDOWS\system32\msmq\storage had 1GB of data in it and no more. 

To solve it I backed up the queues using mqbkup and then purged the poison queue.  I also upped the limit to 4GB so the problem occurs less often.  But that’s not a real solution.  Really I’d like some monitoring to tell me when the queue is approach full so that I can do something about it.

Having a dig around I found a performance counter called MSMQ Service\Total bytes in all queues.  Should be able to monitor that and raise an alert when it approaches the limit.

Thanks to Ayende’s post on this one.  Also there’s a more complete list of possible causes from this post (this is number 7).

No comments: