[ejabberd] More virt host woes.

Brian Cully bcully at gmail.com
Thu Feb 7 21:02:02 MSK 2008

On 7-Feb-2008, at 09:58, Jesse Thompson wrote:
> Obviously you can run multiple domains in a single ejabberd cluster.
> We're running 15 currently.  There are limitations to running many
> domains, but is it really impossible to run thousands?  Maybe it would
> be better for everyone if we stick to addressing the issues, rather  
> than
> declaring the task impossible.

	It is not impossible. With a small amount of work (including patching  
ejabberd to pool odbc connections properly), I was able to get it / 
functional/ for ~7k domains.

	It is still not /acceptable/. It currently takes up ~700M of memory  
and burns 100% CPU for about 30 minutes to handle this load on  
startup. Memory usage stays flat at about ~650M after the CPU load  
dies down.

	The server, notably, is perfectly responsive during this period. But  
for what are hopefully obvious reasons, this isn't production ready.

	Our load going live will be ~8k domains. We want to support an order  
of magnitude more before we go live, so that we have plenty of time  
and room to grow before diving into it again.

> * You can't add domains without restarting ejabberd.  We schedule
> restarts during our weekly maintenance window to add new domains.

	Not acceptable for my uses. That this is an Erlang application that  
requires this kind of total hack only adds salt on the wounds.

> * You might run into file descriptor limits.  We had to compile
> erlang/ejabberd as 64-bit to keep from running into a file descriptor
> limit bug in 32-bit Solaris libc.

	I can deal with fd limits and have done so many times in the past,  
but so far I have not run into any. If they crop up I'm not too worried.

> * ODBC/LDAP connections aren't pooled across vhosts.  But you don't  
> need
> to use ODBC.  We use use an external authentication script (one  
> instance
> per vhost) that authenticates against LDAP and also has some  
> additional
> authnz logic specific to our environment.  We use Mnesia for rosters  
> and
> vcards.

	In my case it was either ODBC or data silos and synchronization and  
all the grief that brings. The total amount of work to get pooling  
properly was ~6 hours, so well worth it vs. silos.

> * Memory requirements.  We managed to get ejabberd to consume 4 GB of
> memory during a load test.  We worked around this by tuning Mnesia
> appropriately.

	I'm going to be rewriting the entire configuration engine to support  
larger domain loads. I hope to be able to do this while still  
supporting the (bullshit) objective of per-domain configuration. The  
only reason I intend to support such a nonsense use case is to try and  
get my patches integrated into mainline when I'm done.

> * Certificate management.  The XMPP specification requires that the  
> cert
> match the domain, not the server.  This makes it very difficult for
> hosting providers.

	I haven't tried this yet. I was worried from reading the code that  
this could be a problem. If you have any solutions that could save me  
time when I finally get there, I would appreciate it.

> I'm still new to ejabberd, so I'm sure that there are more issues
> lurking under the surface.  However, our load testing has shown that  
> it
> is possible to run at least a few hundred domains in single ejabberd
> instance, which meets our requirements for future growth.

	I have no doubt that my configuration changes won't be the last. They  
ought to be, but at this point I've been bitten too many times to hold  
out hope.


More information about the ejabberd mailing list