[ejabberd] Node Clustering issue

Matthew Reilly matthew.reilly at sipphone.com
Fri Jun 13 02:16:17 MSD 2008


Solution summary:

I communicated offlist with Florian, an the issue did turn out to be a
directory issue with the location of the mnesia directory. 

The ejabberdctl command (provided with ejabberd-2.0.1) passes the
erlexec option "-mnesia dir $ROOTDIR/database/$NODE". This forces the
mnesia database in a non-default location.

There is out-of-date infomation in the ejabberd guide at
http://www.process-one.net/docs/ejabberd/guide_en.html which says to set
up clustering by:
   ... in the working directory of ejabberd:
      erl -sname ejabberd \
          -mnesia extra_db_nodes "['ejabberd at first']" \
          -s mnesia

This has mnesia use the default directory name/location, which will be different than an ejabberd-2.0.1 server started with ejabberdctl.

For people using ejabberctl to start their servers, when setting up clustering, make sure to add the non-standard directory name. e.g.:
      erl -sname ejabberd \
          -mnesia extra_db_nodes "['ejabberd at first']" \
          -mnesia dir "/home/jabber/database/ejabberd" \          
          -s mnesia 

Of course, this'll be different for each system.
If you're using fully qualified node names, use "-name" instead of
"-sname".



> Florian Jensen wrote:
> > Hi,
> >
> > thanks a million for the detailed commands.
> >
> > For 1: I installed erlang-nox.
> >
> > I got my nodes setup to copy. I can see the node connected in the 
> > web-admin panel, when I am using the console command. But when I then 
> > start ejabberd, it doesn't connect to the node.
> > ds2581 only sees himself.
> > ds2248 sees the second node as stopped.
> >
> > What am I missing?
> >
> >   
> 
> My best guess:
> 
> Mnesia puts it's disc store in your current directory (by default). You 
> need to make sure that when you set up clustering, you did so in the 
> same directory as the ejabberd server runs in (perhaps /home/jabber ?). 
> Also make sure you did it as the same Linux user name (e.g. "jabber") 
> and with the same node name (the option to -sname) as the server (almost 
> certainly "ejabberd").
> 
> You can check the cwd working directory of the current running beam 
> process with:
> # lsof -c beam -a -d cwd -n
> (You may need to be the root or jabber user for permissions for this).
> 
> If it is a directory issue, you'll see a directory with a name like 
> "Mnesia.ejabberd at ds2581" in two places -- the directory you did the 
> clustering to begin with, and the directory ejabberd runs from. You can 
> stop ejabberd (on the second node), and move the Mnesia tables around so 
> that the one you set up for clustering is now in the current directory 
> of the ejabberd process. If you created this as a different Linux user, 
> make sure to do a "chown -R jabber Mnesia.ejabberd at ds2581" so the 
> ejabberd server will have the proper permissions.
> 
> 
> 
> 
> > Greets,
> >
> > Florian Jensen
> >
> > Matthew Reilly wrote:
> >   
> >> Florian Jensen wrote:
> >>     
> >>> Ok, I got it running. I now can access the erlang console.
> >>>
> >>>
> >>>       
> >> Out of curiosity, how did you get it fixed?
> >>
> >>     
> >>> So, now, how can I let it sync all? And how can I let it start
> >>> automatically in cluster mode?
> >>>
> >>>
> >>>       
> >> I'm doing this from memory, so I apologize if I get something wrong.
> >>
> >> Now that you have connected, you should be able to do (from the erlang
> >> console):
> >> erl>  mnesia:info().
> >>
> >> This should show both your main node and your new node.
> >>
> >> You want to have it set up so that the next time you start your new
> >> node, it remembers to connect to the original node.
> >>
> >> To do this:
> >>
> >> erl>  mnesia:change_table_copy_type(schema,node(),disc_copies).
> >>
> >> This creates a copy of the mnesia schema on the local drive, so that
> >> when you restart the local node, it remebers to connect to the main node.
> >>
> >> What that means is that when you kill your console (with
> >> control-C/control-C), and start ejabberd on the second node (with the
> >> same ./start script), it will come up in clustered mode -- you don't
> >> need to do anything else for (basic) clustering! (on the ejabberd side
> >> -- you still need to set up DNS entries pointing to the second node, or
> >> no clients will find it).
> >>
> >> Once you've done this, the first node stores all data, and the second
> >> node stores none of the data (except for the schema itself). This means
> >> that every mnesia lookup (which is done e.g. everytime a
> >> message/presence stanza is sent) works by having the second node send a
> >> network request to the first node, the first node lookups up the info in
> >> its mnesia table and sends it back to the second node.
> >>
> >> That is the default behavior, which may be acceptable --the mnesia
> >> traffic/CPU load is not very large. However, this only gives you
> >> scalability benefits of clustering, but not fault-tolerance. For
> >> example, if your first node went down, the second node would suddenly
> >> become unusable (since it has no copies of the mnesia tables locally).
> >>
> >> To add fault tolerance, you'll need to stay in the console mode (or
> >> reenter it if you've left it), and do a mnesia:add_table_copy() for each
> >> table. This'll be tedious, and if anyone else knows a more automated
> >> way, I'd like to hear it.
> >>
> >> To get a list of the tables, do:
> >> erl>  mnesia:info().
> >>
> >> You get output similar to:
> >> [{'ejabberd at ds2248',ram_copies}] = [session,route, ...]
> >> [{'ejabberd at ds2248',disc_copies}] = [privacy,...]
> >> [{'ejabberd at ds2248',disc_only_copies}] = [offline_msg,...]
> >>
> >> There are three types of tables: RAM resident (ram_copies), RAM&  disc
> >> resident (disc_copies), and just disc resident (disc_only_copies).
> >>
> >> For each table, you tell the mnesia system that you want a clustered
> >> copy on the second node by issuing these commands from the erlang
> >> console (one the second node):
> >>
> >> erl>  mnesia:add_table_copy(session,node(),ram_copies).
> >> erl>  mnesia:add_table_copy(route,node(),ram_copies).
> >> ...
> >> mnesia:add_table_copy(privacy,node(),disc_copies).
> >> ...
> >> erl>  mnesia:add_table_copy(offline_msg,node(),disc_only_copies).
> >> ...
> >>
> >> If you do this for each table, the second node will be able to keep
> >> running if the first dies. You only have to do this once -- even if you
> >> restart the second node, it knows which tables to cluster.
> >>
> >> Note: The actual list of tables you'll need to cluster will differ
> >> depending on the modules you've configured in your ejabberd.cfg as well
> >> as whether you're using a SQL backend. If you add a new module to
> >> ejabberd.cfg, it may add a new mnesia table, and you made need to do a
> >> mnesia:add_table_copy() for the new table.
> >>     
> >>> _______________________________________________
> >>> ejabberd mailing list
> >>> ejabberd at jabber.ru
> >>> http://lists.jabber.ru/mailman/listinfo/ejabberd
> >>>
> >>>
> >>>       
> >> _______________________________________________
> >> ejabberd mailing list
> >> ejabberd at jabber.ru
> >> http://lists.jabber.ru/mailman/listinfo/ejabberd
> >>     
> > _______________________________________________
> > ejabberd mailing list
> > ejabberd at jabber.ru
> > http://lists.jabber.ru/mailman/listinfo/ejabberd
> >
> >   
> 
> _______________________________________________
> ejabberd mailing list
> ejabberd at jabber.ru
> http://lists.jabber.ru/mailman/listinfo/ejabberd



More information about the ejabberd mailing list