14 Distribution Protocol

This description is far from complete. It will be updated if the protocol is updated. However, the protocols, both from Erlang nodes to the Erlang Port Mapper Daemon (EPMD) and between Erlang nodes are stable since many years.

The distribution protocol can be divided into four parts:

• Low-level socket connection (1)

• Handshake, interchange node name, and authenticate (2)

• Authentication (done by net_kernel(3)) (3)

• Connected (4)

A node fetches the port number of another node through the EPMD (at the other host) to initiate a connection request.

For each host, where a distributed Erlang node is running, also an EPMD is to be running. The EPMD can be started explicitly or automatically as a result of the Erlang node startup.

By default the EPMD listens on port 4369.

(3) and (4) above are performed at the same level but the net_kernel disconnects the other node if it communicates using an invalid cookie (after 1 second).

The integers in all multibyte fields are in big-endian order.

14.1 EPMD Protocol

The requests served by the EPMD are summarized in the following figure.

Each request *_REQ is preceded by a 2 byte length field. Thus, the overall request format is as follows:

Register a Node in EPMD

When a distributed node is started it registers itself in the EPMD. The message ALIVE2_REQ described below is sent from the node to the EPMD. The response from the EPMD is ALIVE2_RESP.

PortNo

The port number on which the node accept connection requests.

NodeType

77 = normal Erlang node, 72 = hidden node (C-node), ...

Protocol

0 = TCP/IPv4, ...

HighestVersion

The highest distribution version that this node can handle. The value in Erlang/OTP R6B and later is 5.

LowestVersion

The lowest distribution version that this node can handle. The value in Erlang/OTP R6B and later is 5.

Nlen

The length (in bytes) of field NodeName.

NodeName

The node name as an UTF-8 encoded string of Nlen bytes.

Elen

The length of field Extra.

Extra

Extra field of Elen bytes.

The connection created to the EPMD must be kept as long as the node is a distributed node. When the connection is closed, the node is automatically unregistered from the EPMD.

The response message ALIVE2_RESP is as follows:

Result = 0 -> ok, result > 0 -> error.

Unregister a Node from EPMD

A node unregisters itself from the EPMD by closing the TCP connection to EPMD established when the node was registered.

Get the Distribution Port of Another Node

When one node wants to connect to another node it starts with a PORT_PLEASE2_REQ request to the EPMD on the host where the node resides to get the distribution port that the node listens to.

where N = Length - 1.

or

If Result > 0, the packet only consists of [119, Result].

The EPMD closes the socket when it has sent the information.

Get All Registered Names from EPMD

This request is used through the Erlang function net_adm:names/1,2. A TCP connection is opened to the EPMD and this request is sent.

The response for a NAMES_REQ is as follows:

NodeInfo is a string written for each active node. When all NodeInfo has been written the connection is closed by the EPMD.

NodeInfo is, as expressed in Erlang:

io:format("name ~ts at port ~p~n", [NodeName, Port]).

Dump All Data from EPMD

This request is not really used, it is to be regarded as a debug feature.

The response for a DUMP_REQ is as follows:

NodeInfo is a string written for each node kept in the EPMD. When all NodeInfo has been written the connection is closed by the EPMD.

NodeInfo is, as expressed in Erlang:

io:format("active name     ~ts at port ~p, fd = ~p~n",
          [NodeName, Port, Fd]).

or

io:format("old/unused name ~ts at port ~p, fd = ~p ~n",
          [NodeName, Port, Fd]).

Kill EPMD

This request kills the running EPMD. It is almost never used.

The response for a KILL_REQ is as follows:

where OKString is "OK".

STOP_REQ (Not Used)

where n = Length - 1.

The current implementation of Erlang does not care if the connection to the EPMD is broken.

The response for a STOP_REQ is as follows:

where OKString is "STOPPED".

A negative response can look as follows:

where NOKString is "NOEXIST".

14.2 Distribution Handshake

This section describes the distribution handshake protocol introduced in Erlang/OTP R6. The handshake has remained almost the same since then.

General

The TCP/IP distribution uses a handshake that expects a connection-based protocol, that is, the protocol does not include any authentication after the handshake procedure.

This is not entirely safe, as it is vulnerable against takeover attacks, but it is a tradeoff between fair safety and performance.

The cookies are never sent in cleartext and the handshake procedure expects the client (called A) to be the first one to prove that it can generate a sufficient digest. The digest is generated with the MD5 message digest algorithm and the challenges are expected to be random numbers.

Definitions

A challenge is a 32-bit integer in big-endian order. Below the function gen_challenge() returns a random 32-bit integer used as a challenge.

A digest is a (16 bytes) MD5 hash of the challenge (as text) concatenated with the cookie (as text). Below, the function gen_digest(Challenge, Cookie) generates a digest as described above.

An out_cookie is the cookie used in outgoing communication to a certain node, so that A's out_cookie for B is to correspond with B's in_cookie for A and conversely. A's out_cookie for B and A's in_cookie for B need not be the same. Below the function out_cookie(Node) returns the current node's out_cookie for Node.

An in_cookie is the cookie expected to be used by another node when communicating with us, so that A's in_cookie for B corresponds with B's out_cookie for A. Below the function in_cookie(Node) returns the current node's in_cookie for Node.

The cookies are text strings that can be viewed as passwords.

Every message in the handshake starts with a 16-bit big-endian integer, which contains the message length (not counting the two initial bytes). In Erlang this corresponds to option {packet, 2} in gen_tcp(3). Notice that after the handshake, the distribution switches to 4 byte packet headers.

The Handshake in Detail

Imagine two nodes, A that initiates the handshake and B that accepts the connection.

1) connect/accept

A connects to B through TCP/IP and B accepts the connection.

2) send_name/receive_name

A sends an initial identification to B, which receives the message. The message looks as follows (every "square" is one byte and the packet header is removed):

+---+--------+--------+-----+-----+-----+-----+-----+-----+-...-+-----+
|'n'|Version0|Version1|Flag0|Flag1|Flag2|Flag3|Name0|Name1| ... |NameN|
+---+--------+--------+-----+-----+-----+-----+-----+-----+-... +-----+

'n' is the message tag. 'Version0' and 'Version1' is the distribution version selected by A, based on information from the EPMD. (16-bit big-endian) 'Flag0' ... 'Flag3' are capability flags, the capabilities are defined in $ERL_TOP/lib/kernel/include/dist.hrl. (32-bit big-endian) 'Name0' ... 'NameN' is the full node name of A, as a string of bytes (the packet length denotes how long it is).

3) recv_status/send_status

B sends a status message to A, which indicates if the connection is allowed. The following status codes are defined:

ok

The handshake will continue.

ok_simultaneous

The handshake will continue, but A is informed that B has another ongoing connection attempt that will be shut down (simultaneous connect where A's name is greater than B's name, compared literally).

nok

The handshake will not continue, as B already has an ongoing handshake, which it itself has initiated (simultaneous connect where B's name is greater than A's).

not_allowed

The connection is disallowed for some (unspecified) security reason.

alive

A connection to the node is already active, which either means that node A is confused or that the TCP connection breakdown of a previous node with this name has not yet reached node B. See step 3B below.

The format of the status message is as follows:

+---+-------+-------+-...-+-------+
|'s'|Status0|Status1| ... |StatusN|
+---+-------+-------+-...-+-------+

's' is the message tag. 'Status0' ... 'StatusN' is the status as a string (not terminated).

3B) send_status/recv_status

If status was alive, node A answers with another status message containing either true, which means that the connection is to continue (the old connection from this node is broken), or false, which means that the connection is to be closed (the connection attempt was a mistake.

4) recv_challenge/send_challenge

If the status was ok or ok_simultaneous, the handshake continues with B sending A another message, the challenge. The challenge contains the same type of information as the "name" message initially sent from A to B, plus a 32-bit challenge:

+---+--------+--------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-...-+-----+
|'n'|Version0|Version1|Flag0|Flag1|Flag2|Flag3|Chal0|Chal1|Chal2|Chal3|Name0|Name1| ... |NameN|
+---+--------+--------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-... +-----+

'Chal0' ... 'Chal3' is the challenge as a 32-bit big-endian integer and the other fields are B's version, flags, and full node name.

5) send_challenge_reply/recv_challenge_reply

Now A has generated a digest and its own challenge. Those are sent together in a package to B:

+---+-----+-----+-----+-----+-----+-----+-----+-----+-...-+------+
|'r'|Chal0|Chal1|Chal2|Chal3|Dige0|Dige1|Dige2|Dige3| ... |Dige15|
+---+-----+-----+-----+-----+-----+-----+-----+-----+-...-+------+

'r' is the tag. 'Chal0' ... 'Chal3' is A's challenge for B to handle. 'Dige0' ... 'Dige15' is the digest that A constructed from the challenge B sent in the previous step.

6) recv_challenge_ack/send_challenge_ack

B checks that the digest received from A is correct and generates a digest from the challenge received from A. The digest is then sent to A. The message is as follows:

+---+-----+-----+-----+-----+-...-+------+
|'a'|Dige0|Dige1|Dige2|Dige3| ... |Dige15|
+---+-----+-----+-----+-----+-...-+------+

'a' is the tag. 'Dige0' ... 'Dige15' is the digest calculated by B for A's challenge.

7) check

A checks the digest from B and the connection is up.

Semigraphic View

A (initiator)                                      B (acceptor)

TCP connect ------------------------------------>
                                                   TCP accept

send_name -------------------------------------->
                                                   recv_name

  <---------------------------------------------- send_status
recv_status
(if status was 'alive'
 send_status - - - - - - - - - - - - - - - - - ->
                                                   recv_status)
                                                   ChB = gen_challenge()
                          (ChB)
  <---------------------------------------------- send_challenge
recv_challenge
ChA = gen_challenge(),
OCA = out_cookie(B),
DiA = gen_digest(ChB, OCA)
                          (ChA, DiA)
send_challenge_reply --------------------------->
                                                   recv_challenge_reply
                                                   ICB = in_cookie(A),
                                                   check:
                                                   DiA == gen_digest (ChB, ICB)?
                                                   - if OK:
                                                    OCB = out_cookie(A),
                                                    DiB = gen_digest (ChA, OCB)
                          (DiB)
  <----------------------------------------------- send_challenge_ack
recv_challenge_ack                                  DONE
ICA = in_cookie(B),                                - else:
check:                                              CLOSE
DiB == gen_digest(ChA, ICA)?
- if OK:
 DONE
- else:
 CLOSE

Distribution Flags

The following capability flags are defined:

-define(DFLAG_PUBLISHED,16#1).

The node is to be published and part of the global namespace.

-define(DFLAG_ATOM_CACHE,16#2).

The node implements an atom cache (obsolete).

-define(DFLAG_EXTENDED_REFERENCES,16#4).

The node implements extended (3 × 32 bits) references. This is required today. If not present, the connection is refused.

-define(DFLAG_DIST_MONITOR,16#8).

The node implements distributed process monitoring.

-define(DFLAG_FUN_TAGS,16#10).

The node uses separate tag for funs (lambdas) in the distribution protocol.

-define(DFLAG_DIST_MONITOR_NAME,16#20).

The node implements distributed named process monitoring.

-define(DFLAG_HIDDEN_ATOM_CACHE,16#40).

The (hidden) node implements atom cache (obsolete).

-define(DFLAG_NEW_FUN_TAGS,16#80).

The node understand new fun tags.

-define(DFLAG_EXTENDED_PIDS_PORTS,16#100).

The node can handle extended pids and ports. This is required today. If not present, the connection is refused.

-define(DFLAG_EXPORT_PTR_TAG,16#200).

-define(DFLAG_BIT_BINARIES,16#400).

-define(DFLAG_NEW_FLOATS,16#800).

The node understands new float format.

-define(DFLAG_UNICODE_IO,16#1000).

-define(DFLAG_DIST_HDR_ATOM_CACHE,16#2000).

The node implements atom cache in distribution header.

-define(DFLAG_SMALL_ATOM_TAGS, 16#4000).

The node understand the SMALL_ATOM_EXT tag.

-define(DFLAG_UTF8_ATOMS, 16#10000).

The node understand UTF-8 encoded atoms.

-define(DFLAG_MAP_TAG, 16#20000).

The node understand the map tag.

-define(DFLAG_BIG_CREATION, 16#40000).

The node understand big node creation.

-define(DFLAG_SEND_SENDER, 16#80000).

Use the SEND_SENDER control message instead of the SEND control message and use the SEND_SENDER_TT control message instead of the SEND_TT control message.

-define(DFLAG_BIG_SEQTRACE_LABELS, 16#100000).

The node understands any term as the seqtrace label.

-define(DFLAG_EXIT_PAYLOAD, 16#400000).

Use the PAYLOAD_EXIT, PAYLOAD_EXIT_TT, PAYLOAD_EXIT2, PAYLOAD_EXIT2_TT and PAYLOAD_MONITOR_P_EXIT control messages instead of the non-PAYLOAD variants.

-define(DFLAG_FRAGMENTS, 16#800000).

Use fragmented distribution messages to send large messages.

There is also function dist_util:strict_order_flags/0 returning all flags (bitwise or:ed together) corresponding to features that require strict ordering of data over distribution channels.

14.3 Protocol between Connected Nodes

Since ERTS 5.7.2 (OTP R13B) the runtime system passes a distribution flag in the handshake stage that enables the use of a distribution header on all messages passed. Messages passed between nodes have in this case the following format:

Length

Equal to d + n + m.

DistributionHeader

Distribution header describing the atom cache and fragmented distribution messages.

ControlMessage

A tuple passed using the external format of Erlang.

Message

The message sent to another node using the '!' or the reason for a EXIT, EXIT2 or DOWN signal using the external term format.

Notice that the version number is omitted from the terms that follow a distribution header.

Nodes with an ERTS version earlier than 5.7.2 (OTP R13B) does not pass the distribution flag that enables the distribution header. Messages passed between nodes have in this case the following format:

Length

Equal to 1 + n + m.

Type

Equal to 112 (pass through).

ControlMessage

A tuple passed using the external format of Erlang.

Message

The message sent to another node using the '!' (in external format). Notice that Message is only passed in combination with a ControlMessage encoding a send ('!').

The ControlMessage is a tuple, where the first element indicates which distributed operation it encodes:

LINK

{1, FromPid, ToPid}

SEND

{2, Unused, ToPid}

Followed by Message.

Unused is kept for backward compatibility.

EXIT

{3, FromPid, ToPid, Reason}

This signal is sent when a link has been broken

UNLINK

{4, FromPid, ToPid}

NODE_LINK

{5}

REG_SEND

{6, FromPid, Unused, ToName}

Followed by Message.

Unused is kept for backward compatibility.

GROUP_LEADER

{7, FromPid, ToPid}

EXIT2

{8, FromPid, ToPid, Reason}

This signal is sent by a call to the erlang:exit/2 bif

14.4 New Ctrlmessages for distrvsn = 1 (Erlang/OTP R4)

SEND_TT

{12, Unused, ToPid, TraceToken}

Followed by Message.

Unused is kept for backward compatibility.

EXIT_TT

{13, FromPid, ToPid, TraceToken, Reason}

REG_SEND_TT

{16, FromPid, Unused, ToName, TraceToken}

Followed by Message.

Unused is kept for backward compatibility.

EXIT2_TT

{18, FromPid, ToPid, TraceToken, Reason}

14.5 New Ctrlmessages for distrvsn = 2

distrvsn 2 was never used.

14.6 New Ctrlmessages for distrvsn = 3 (Erlang/OTP R5C)

None, but the version number was increased anyway.

14.7 New Ctrlmessages for distrvsn = 4 (Erlang/OTP R6)

These are only recognized by Erlang nodes, not by hidden nodes.

MONITOR_P

{19, FromPid, ToProc, Ref}, where FromPid = monitoring process and ToProc = monitored process pid or name (atom)

DEMONITOR_P

{20, FromPid, ToProc, Ref}, where FromPid = monitoring process and ToProc = monitored process pid or name (atom)

We include FromPid just in case we want to trace this.

MONITOR_P_EXIT

{21, FromProc, ToPid, Ref, Reason}, where FromProc = monitored process pid or name (atom), ToPid = monitoring process, and Reason = exit reason for the monitored process

14.8 New Ctrlmessages for Erlang/OTP 21

SEND_SENDER

{22, FromPid, ToPid}

Followed by Message.

This control message replaces the SEND control message and will be sent when the distribution flag DFLAG_SEND_SENDER has been negotiated in the connection setup handshake.

Note

Messages encoded before the connection has been set up may still use the SEND control message. However, once a SEND_SENDER or SEND_SENDER_TT control message has been sent, no more SEND control messages will be sent in the same direction on the connection.

SEND_SENDER_TT

{23, FromPid, ToPid, TraceToken}

Followed by Message.

This control message replaces the SEND_TT control message and will be sent when the distribution flag DFLAG_SEND_SENDER has been negotiated in the connection setup handshake.

Note

Messages encoded before the connection has been set up may still use the SEND_TT control message. However, once a SEND_SENDER or SEND_SENDER_TT control message has been sent, no more SEND_TT control messages will be sent in the same direction on the connection.

14.9 New Ctrlmessages for Erlang/OTP 22

PAYLOAD_EXIT

{24, FromPid, ToPid}

Followed by Reason.

This control message replaces the EXIT control message and will be sent when the distribution flag DFLAG_EXIT_PAYLOAD has been negotiated in the connection setup handshake.

PAYLOAD_EXIT_TT

{25, FromPid, ToPid}

Followed by Reason.

This control message replaces the EXIT_TT control message and will be sent when the distribution flag DFLAG_EXIT_PAYLOAD has been negotiated in the connection setup handshake.

PAYLOAD_EXIT2

{26, FromPid, ToPid}

Followed by Reason.

This control message replaces the EXIT2 control message and will be sent when the distribution flag DFLAG_EXIT_PAYLOAD has been negotiated in the connection setup handshake.

PAYLOAD_EXIT2_TT

{27, FromPid, ToPid}

Followed by Reason.

This control message replaces the EXIT2_TT control message and will be sent when the distribution flag DFLAG_EXIT_PAYLOAD has been negotiated in the connection setup handshake.

PAYLOAD_MONITOR_P_EXIT

{28, FromPid, ToPid, Ref}

Followed by Reason.

This control message replaces the MONITOR_P_EXIT control message and will be sent when the distribution flag DFLAG_EXIT_PAYLOAD has been negotiated in the connection setup handshake.