[ejabberd] More virt host woes.
bcully at gmail.com
Thu Feb 7 21:02:02 MSK 2008
On 7-Feb-2008, at 09:58, Jesse Thompson wrote:
> Obviously you can run multiple domains in a single ejabberd cluster.
> We're running 15 currently. There are limitations to running many
> domains, but is it really impossible to run thousands? Maybe it would
> be better for everyone if we stick to addressing the issues, rather
> declaring the task impossible.
It is not impossible. With a small amount of work (including patching
ejabberd to pool odbc connections properly), I was able to get it /
functional/ for ~7k domains.
It is still not /acceptable/. It currently takes up ~700M of memory
and burns 100% CPU for about 30 minutes to handle this load on
startup. Memory usage stays flat at about ~650M after the CPU load
The server, notably, is perfectly responsive during this period. But
for what are hopefully obvious reasons, this isn't production ready.
Our load going live will be ~8k domains. We want to support an order
of magnitude more before we go live, so that we have plenty of time
and room to grow before diving into it again.
> * You can't add domains without restarting ejabberd. We schedule
> restarts during our weekly maintenance window to add new domains.
Not acceptable for my uses. That this is an Erlang application that
requires this kind of total hack only adds salt on the wounds.
> * You might run into file descriptor limits. We had to compile
> erlang/ejabberd as 64-bit to keep from running into a file descriptor
> limit bug in 32-bit Solaris libc.
I can deal with fd limits and have done so many times in the past,
but so far I have not run into any. If they crop up I'm not too worried.
> * ODBC/LDAP connections aren't pooled across vhosts. But you don't
> to use ODBC. We use use an external authentication script (one
> per vhost) that authenticates against LDAP and also has some
> authnz logic specific to our environment. We use Mnesia for rosters
In my case it was either ODBC or data silos and synchronization and
all the grief that brings. The total amount of work to get pooling
properly was ~6 hours, so well worth it vs. silos.
> * Memory requirements. We managed to get ejabberd to consume 4 GB of
> memory during a load test. We worked around this by tuning Mnesia
I'm going to be rewriting the entire configuration engine to support
larger domain loads. I hope to be able to do this while still
supporting the (bullshit) objective of per-domain configuration. The
only reason I intend to support such a nonsense use case is to try and
get my patches integrated into mainline when I'm done.
> * Certificate management. The XMPP specification requires that the
> match the domain, not the server. This makes it very difficult for
> hosting providers.
I haven't tried this yet. I was worried from reading the code that
this could be a problem. If you have any solutions that could save me
time when I finally get there, I would appreciate it.
> I'm still new to ejabberd, so I'm sure that there are more issues
> lurking under the surface. However, our load testing has shown that
> is possible to run at least a few hundred domains in single ejabberd
> instance, which meets our requirements for future growth.
I have no doubt that my configuration changes won't be the last. They
ought to be, but at this point I've been bitten too many times to hold
More information about the ejabberd