[ejabberd] Multi node ejabberd cluster set up

Jesse Thompson jesse.thompson at doit.wisc.edu
Wed Dec 9 23:31:18 MSK 2009


Here is the result of our recent "trial and error" investigation into 
the best way to configure the database in a cluster.  Sorry for the 
formatting (pasted from a confluence table.)

Disclaimer: it may not be correct.  It would be nice if there was 
official documentation for this either provided by p1 or by the 
community (is there even a place for the community to contribute 
documentation?)

Table configuration

    1. Inference1: If the master is RAM&disc, then the slave should be
          1. RAM&disc if you want it available when slave starts up alone
          2. RAMcopy if you don't
    2. Inference2: If the master is RAMcopy, then the slave should be
          1. RAM copy (instead of Remote Copy) so that the feature works 
on the slave when the master is down.
          2. Remote copy: if the feature only works if the master is up
    3. Inference3: If the master is DiscOnly, then the slave should be:
          1. Remote copy: if the feature only works if the master is up
          2. There is a possible synchronization problem if we try to 
set the slave to DiscOnly (needs to be confirmed)
          3. Consider changing both master and slave to RAM&disc

table name 	master 	slave 	notes
acl
	RAM and disc copy
	RAM and disc copy
	changed slave from RAM copy to RAM and disc copy due to Inference1
caps_feature
	RAM copy
	RAM copy
	changed slave from Remote copy to RAM copy due to Inference2
config
	RAM and disc copy
	RAM and disc copy
	changed slave from RAM copy to RAM and disc copy due to Inference1
disco_publish
	Disc only copy
	Remote copy
	need to research changing both to RAM and disc copy
http_bind
	RAM copy
	RAM copy
	changed slave from Remote copy to RAM copy due to Inference2
http_poll
	RAM copy
	RAM copy
	changed slave from Remote copy to RAM copy due to Inference2
iq_response
	RAM copy
	RAM copy
	OK
last_activity
	Disc only copy
	Remote copy
	need to research changing both to RAM and disc copy
local_config
	RAM and disc copy
	RAM and disc copy
	changed slave from RAM copy to RAM and disc copy due to Inference1
motd
	Disc only copy
	Remote copy
	need to research changing both to RAM and disc copy
motd_users
	Disc only copy
	Remote copy
	need to research changing both to RAM and disc copy
muc_online_room
	RAM copy
	Remote copy
	leaving slave as Remote copy since muc features aren't clustered
need to research muc clustering
muc_registered
	Disc only copy
	Remote copy
	leaving slave as Remote copy since muc features aren't clustered
need to research muc clustering
muc_room
	Disc only copy
	Remote copy
	leaving slave as Remote copy since muc features aren't clustered
need to research muc clustering
offline_msg
	Disc only copy
	Disc only copy
	leave both as disc only since this database has a risk of growing very 
large
we should test to make sure that offline messages are delivered in 
outage scenarios
privacy
	Disc only copy
	Disc only copy
	need to research possible consistency issue due to Inference3
private_storage
	Disc only copy
	Remote copy
	need to research changing both to RAM and disc copy
pubsub_node
	Disc only copy
	Remote copy
	need to research changing both to RAM and disc copy
roster
	RAM and disc copy
	RAM and disc copy
	changed slave from Disc only copy to RAM and disc copy due to Inference1
route
	RAM copy
	RAM copy
	OK
s2s
	RAM copy
	RAM copy
	OK
schema
	RAM and disc copy
	RAM and disc copy
	OK
session
	RAM copy
	RAM copy
	OK
user_caps
	RAM and disc copy
	RAM and disc copy
	changed slave from Remote copy to RAM and disc copy due to Inference1
user_caps_resources
	RAM and disc copy
	RAM and disc copy
	changed slave from Remote copy to RAM and disc copy due to Inference1
vcard
	Disc only copy
	Remote copy
	leaving slave as Remote copy due to Inference3
we should retest the consistency issues
vcard_search
	Disc only copy
	Remote copy
	leaving slave as Remote copy due to Inference3
we should retest the consistency issues

On 12/4/2009 2:03 PM, Jonathan Tullett wrote:
> Hello.
>
> I'm looking at deploying a clustered ejabberd setup at my company which
> will have, initially, 4 nodes in various locations around the world, all
> serving the same domain.
>
> The idea is that users can connect to any node and will be able to
> communicate with any other user logged into any other node without issue.
>
> I've tested this set up with Parallels on my machine here, and that part
> of communication works as advertised.
>
> The questions I have are related to the mnesia database set up. I've
> spent many hours reading mailing lists and the majority (all?) of
> cluster related setup questions involve just 2 nodes.
>
> This is what we're looking for in an ideal set up:
>
> 1. all 4 nodes communicating and replicating.
> 2. the ability for any node (or 2 nodes) to drop out without it
> affecting the other nodes' communications (so if we lose 'B', people on
> 'A','C' and 'D' can all still see each other and communicate)
> 3. for any dropped out node to reintegrate into the cluster without
> causing too much of a headache.
>
> Following (slightly modified) instructions found at
> http://www.process-one.net/en/ejabberd/guide_en#htoc82, part 1 is
> working fine - with three nodes using one other as the master.
>
> re point 2, I've found if the master node (the one passed as the '
> -mnesia extra_db_nodes "['ejabberd at first']" \' param of the setup
> instructions) goes down, then the other nodes are unable to restart
> themselves.
>
> An apparent work around is to, on the database set up page of ejabberd,
> change all 'remote copy' databases to 'disk and local'. Once done,
> they're able to restart. I'm unable to find if this is the correct thing
> to do, or whether there's another solution (is it possible to specify
> more than a single master node? what happens if you specify 3, and one
> of those goes down?)
>
> If multi-master isn't viable, is it possible to switch an instance to
> look at another node for the master copy?
>
> Any ideas, pointers or help you can provide will be appreciated.
>
> Jonathan.
> _______________________________________________
> ejabberd mailing list
> ejabberd at jabber.ru
> http://lists.jabber.ru/mailman/listinfo/ejabberd

-- 
   Jesse Thompson
   Division of Information Technology, University of Wisconsin-Madison
   Email/IM: jesse.thompson at doit.wisc.edu

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3317 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.jabber.ru/pipermail/ejabberd/attachments/20091209/56abe788/attachment.bin>


More information about the ejabberd mailing list