[ejabberd] ejabberd mod_muc bottleneck
karthik at socialvisioninc.com
Sat Jun 19 01:41:58 MSD 2010
I am load testing ejabberd MUC. I have a Tsung load test that generates
2500 users, puts them in 250 rooms, and has each one send 1 msg/s (for
2500msg/s overall being sent to ejabberd MUC).
I'm running ejabberd 2.1.4 on Erlang 13B03 on a 4-core Ubuntu Server 10.04
LTS VM w/ 16GB of RAM. This is my ejabberdctl.cfg:
http://pastebin.com/hs6Fijri and ejabberd.cfg: http://pastebin.com/deQ495gN
As the test is running, I notice that the message queue for the mod_muc
process grows and grows. From a crash dump I observed the message queue for
mod_muc is bottlenecking the whole system, it's the only process with a
sizable message queue (it's enormous, well over 100,000 queued messages).
Using process_info in a remote shell it is possible to see the queue length
growing very rapidly in real time (~1000msg/s).
I used fprof to do some profiling on the mod_muc process, see the output
file here: http://pastebin.com/QXchAKtM. From the log mod_muc:do_route was
called around 26,000 times over the course of the 38s of data collection.
This is ~1.4ms per packet, which translates to millions of clock cycles,
highly inefficient! It seems that nearly all the time is being used in
"suspend" calls (if you add up the ACC time for suspend, it becomes 100% of
the overall measured time). Indeed, when I call process_info on the mod_muc
process inside a remote shell, its status seems to always be "runnable".
However, my fprof can only use wallclock time, not the high resolution CPU
time, so I'm not sure how accurate all of this is.
It seems odd that a 4-core box can't even handle 2,500 messages/s of
incoming traffic. It seems like the bottleneck is a single process, which
if I understand correctly can only run on 1 core. Maybe if the process
could run at higher priority? Or if the routing in mod_muc can be
distributed to multiple processes?
Can anyone shed any light on what is going on?
Karthik Kailash | Product
SocialVision, Online Television Becomes a Social Experience
CELL . 408.768.7704 | WEB . www.socialvisioninc.com | FACEBOOK .
facebook.com/socialvision <http://www.facebook.com/socialvision> | TWITTER
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the ejabberd