Servers
NCAT : Concatenate and redirect sockets
NCAT is a feature-packed networking utility which reads and writes data across
networks from the command line.
It supports TCP/IP and UDP/IP as well as UNIX sockets.
It is one of all NETCAT incarnations. Both tools have existed for many years.
NCAT is used as network server in combination with Wiimmfi's PHP scripts.
For each connection (TCP/IP) or packet (UDP/IP),
a new sub-process is created to manage the single connection or packet.
So we have an own independent instance for each network client.
It was a solution to have a stable and well tested server without the need
to develop and test such a management server.
And when an instance crashes,
it had no impact to the NCAT server or the other instances.
A disadvantage is that a new sub-process is created for each network client,
using more processing power.
At the beginning, NCAT was used as server for
GPCM, GPSP, MASTER
and MS.
Nowadays it is used for GPCM, GPSP and COMPETITION.
NCAT can also used for non-interactive connections to the
backends of servers.
It allows a script to send commands to the servers and to collect the output.
NAS : Login server
NAS is the
Nintendo
Authentication
Server, using HTTP/POST.
Original Wii games use HTTPS, as do most DS, DSiWare and WiiWare games.
Because Wiimmfi can not serve certificates signed by Nintendo,
Wii games must be patched to login to Wiimmfi with HTTP and without SSL,
by either a RAM patch (i.e. MrBean35000vr's patcher),
or a permanent file patch (via ISO patcher).
The NAS server is written as a PHP script, run by an Apache web server.
The devices send their request for authentication as POST data,
and NAS replies in the usual way.
On login NAS checks parameters, maintenance mode
and login restrictions like disabled games or bans.
It also manages new consoles and new profiles.
Login data needed by the other servers is stored into the database.
GPCM : Client server
After a succesful
NAS login,
the game tries to connect the GPCM server using TCP/IP, authenticating
with a token from NAS. The connection will be closed with logout.
When the server closes the connection, the client is kicked from Wiimmfi immediately.
At Wiimmfi, GPCM is a combination of NCAT and PHP scripts.
NCAT is the server and creates an own PHP instance for each connecting client.
So each GPCM process manages exactly one client.
GPCM uses UNIX sockets for communication.
The communication is needed to deliver messages between the clients
(client → own GPCM → friend GPCM → friend client) and from and to other servers.
The MASTER server is informed about
logins and logouts for an optimized management.
Plans
Wiimm has already started to implement GPCM in C end 2015, but
paused the development.
The following points are already implemented but are hardly tested yet:
A main server (supervisor) manages all external connections
and creates a sub-process for new clients.
Each sub-process can handle any number of clients;
the limit is changeable at runtime.
If the limit set to NULL, the supervisor manages all clients by itself.
So a single process and a multi process server is available.
The single process is faster, but the multi process is more efficient
and stable if one process terminates unexpectedly.
So the multiprocess method was preferred at the beginning.
Each sub-process communicates with the supervisor using an unnamed socket
to exchange status messages and to deliver file descriptors of clients.
The supervisor redirects messages for other clients to the sub-processes.
This is very fast.
After sending a PING from the supervisor's backend
to a sub-process the answer arrives in 50–100 microseconds.
In this time the message is queued and send, the sub-process is woken up
and the message received and analysed, a reply is queued and send,
the supervisor is woken up and the message received and analysed
and last not least a summary is printed at the backend.
The supervisor collects status data of all clients.
So the supervisor knows all relevant data for statistics.
It would be possible to follow all connections and disconnections of
guests and hosts to create better status pages.
Data transfer with the servers MASTER and
JOB is done by SEQPACKET connections.
SEQPACKET is only available for UNIX sockets,
and is a network stream (like TCP) with record support (like UDP).
Once connected, the connection is only closed if one server is halted.
At restart, the connections are established again.
GPCM allows push messages to MASTER and JOB for status changes of profiles.
A test version of the JOB server already uses this to deliver some status
values to mkw-ana.
The new GPCM supports a backend like the other C servers.
It is available for the supervisor via such backend,
and can be used to send commands to each sub-process.
OPENHOST is already implemented, but has some
small issues. It needs more tests.
We tested the new GPCM at the Wiimmfi test server in March and April.
Therefore, MKW Intermezzos were patched
to use the test server.
Overall, the new GPCM must be tested more intensively.
At the moment, the new GPCM runs as server SV (SUPERVISOR).
SV : Supervisor
SUPERVISOR (short
SV) is a new analytic server.
In real, it is a special working mode of the coming
GPCM written in C,
so it can run in the background if the new GPCM is activated.
SV is the central server and has permanent connections to
MASTER+MS, NATNEG,
JOB, and all GPCM instances.
It analyses the data of all other instances to build internal models.
Another point is the gpcm-to-gpcm communication.
Before SV, each GPCM opens a UNIX stream socket of other GPCM to
deliver status messages.
With the new SV, GPCM use only one UNIX stream socket and use SV
as delivery agent. Also MASTER use SV as delivery agent to e.g. kick a player.
SV has the same watchdog, restart
and backend features like the other servers.
Status quo
September 2016
-
SV runs as central server with permanent connections to
MASTER+MS, NATNEG,
JOB, and all GPCM instances.
-
GPCM instances use SV as delivery agent.
The old UNIX sockets (one for each instance) can be used as an alternative
communication way until all services are switched to the new model.
-
GPCM instances inform SV about status changes of the clients.
GPCM90 messages (connection details) are additional send to SV.
-
MS informs SV about GPCM90 messages too.
-
MASTER informs SV about changes of MASTER-ID and NATNEG-ID.
It is used to identify for example NATNEG connections.
-
NATNEG informs SV about successful connections.
-
SV uses all data to create a room model especially for Mario Kart Wii.
So it knows exactly the memebers of the rooms and the NATNEG status.
But it is not clear, if special events like a room split is handled in a correct way.
-
The mkw status page gets data (JSON) from SV
to print all game statistics. This minimizes the database access.
Plans
-
SV integrates special \wiimmfi\ messages into the GPCM stream to give
tools like mkw-ana additional information.
The games itself ignore the unknown message type.
-
Analyse the data more deeply.
One point are split rooms. At the moment it is not clear,
if SV handles this event in a correct way.
-
Inform MASTER about rooms to support a new optimized matchmaking (MKW only).
GPSP : Retrieving friends
GPSP is used to retrieve information about friends.
The server is usually connected once after
GPCM login using TCP/IP.
At Wiimmfi, GPSP is implemented as a combination of NCAT and PHP scripts.
NCAT is the server and creates an own instance for each connection.
Usually, a connection is closed within a second.
MASTER : Master matchmaking server
While online, each client (game) sends a UDP/IP packet
with the current online status to the MASTER approximately every 30–60sec.
This data is stored in a game specific database table for matchmaking,
which can be retrieved by the
MS server.
In the beginning, MASTER was inplemented as a PHP script.
A NCAT server creates a client for each TCP/UDP packet,
which results in a very low performance.
The MASTER was the most database-querying server of Wiimmfi.
Very early Wiimm decided that the MASTER is the first server to be rewritten.
In June 2014 he started with the development of the C server.
At July 5, 2014
it replaced the old multiprocess NCAT+PHP version.
The next step was the integration of the MS
server into the MASTER code.
At the end of 2014 a first version was implemented and running at the test server.
It took a long time until all issues are solved.
So it was activated at Wiimmfi in January 2016.
The combination of both servers into one process reduces the database
lookups and queries even further.
In December 2015 the new backend and the
restart feature were implemented.
Since January 2016, watchdog will restart MASTER
after a crash. Only one crash happend until July 2016.
The MASTER+MS is now fully implemented.
Only special game specific features will be added if needed.
Some facts (based on an average of 150 online clients)
-
The MASTER part of the combined server receives about 1.3 millions packets (230 MB)
per day. 600 000 packets are matchmaking data,
and for only 20% the database is updated.
-
The MS part is connected about 130 000 times per day.
MS receives about 530 000 packets (34 MB)
and sends one packet for each connection.
-
The server makes about 280 000 database queries (100 MB) per day
and waits less than 5 minutes (<0.34%) for replies.
-
Analysis and execution need only about 0.24% of one CPU.
Most of the time (99.44%) MASTER+MS is waiting for packets.
-
The server needs about 4 to 4.5 MB of RAM.
The more games and MKW regions, the more memory is needed.
MS : Matchmaking retrieving server
MS is used by the clients to search for hosts for matchmaking.
Answers of MS are encrypted by a game-specific secret key.
If successful, MS is also used as relay for try-to-connect messages.
At Wiimmfi, MS was implemented as an NCAT and PHP script combination.
NCAT is the server and creates an own instance for each connection.
Usually, a connection is closed within 5 seconds.
In August 2014, Wiimm had rewritten the MS to solve different problems.
In January 2016, the PHP version was replaced
by a combined MASTER+MS server written in C.
The combination of both servers into one process reduces the database
lookups and queries again and allows to implement special game specific features.
NATNEG : Enable client-to-client connections behind firewalls
The name
NATNEG is a combination of
NAT
(
Network
Address
Translation)
and
NEG (
NEGotiation).
When playing with other players, the clients (games) must create
a client-to-client connection to exchange game data.
Because of firewalls and network address translations (NAT)
a server must be used to open the gates. And this server is NATNEG.
At the beginning of the development, Wiimmfi used the NATNEG server of
Nagato written in python, using an SQLite database.
Two weeks before official Wiimmfi start, Wiimm finished his own NATNEG server
written in C. It is a standalone NATNEG server and doesn't need a database,
because all protocol data is stored in an internal memory array.
It also supports 2 UNIX sockets: One for the delivering of faked UDP packets
and one for maintenance. The maintenance port accepts commands and sends back
the output.
NCAT can be used to establish the bidirectional connection.
Since June 2015,
there are 3 instances of NATNEG at 3 different servers.
In December 2015 the new backend and the
restart feature were implemented.
Since January 2016, watchdog will restart NATNEG
after a crash; but a crash never happened.
The 3 NATNER servers have been able to communicate with each other
since April 2020.
This is used to support other NATNEG types.
Log messages can also be forwarded to make debugging easier.
Some facts (based on an average of 150 online clients)
-
The primary NATNEG server receives about 200 000 packets (7.6 MB) per day.
Analysis and executions need only about 0.016% of one CPU in the server.
Most of the time (99.98%) NATNEG is waiting for packets.
-
Each of the secondary NATNEG servers receives about 50 000 packets
(1.7 MB) per day.
Analysis and executions need only about 0.002% of one CPU.
Most of the time (99.998%) the secondary NATNEGs are waiting for packets.
-
All 3 NATNEGs now run without any interruption.
Only manual restarts (e.g. after system reboot) were done.
For example, NATNEG #2 ran now for 3 years (since July 2017)
and get 34 backups (smart restart) in between.
-
The NATNEG server supports up to 1000 open connection requests at the same time.
The limit is hard coded, but could easily be increased if needed.
The maximum usage was 80.
-
To support a delayed answer (20ms; direct answers are too fast for the clients),
a send queue with 500 slots is implemented.
On overflow, the answers are sent earlier.
The limit is hard coded too, but could easily be increased if needed.
The maximum usage was 35.
-
Each NATNEG server needs about 4 MB of RAM.
SAKE : Server to exchange user-created content
The SAKE server is used by games to store user-created content.
For Mario Kart Wii, this server stores Miis and uploaded ghost data.
Mid-2015, Leseratte used the AltWFC SAKE server to understand how basic data
storage worked, and implemented bugfixes and additional features needed by
Mario Kart Wii.
In March 2016, Wiimm and Leseratte reimplemented the SAKE server with all its
improvements in PHP in order to use it on Wiimmfi, run by an Apache web server.
RACE : Mario-Kart-specific server for rankings
The RACE server is a Mario-Kart-specific server. The game uses this server
to up- and download data to / from the in-game rankings (both for regular tracks
and competitions).
The first test implementation in Python has been written by Leseratte in
mid-2015, and in March 2016, at the same time as the SAKE server,
it has been reimplemented in PHP by Wiimm and Leseratte, run by an Apache web server.
COMPETITION : Competition management server for Mario Kart Wii
The competition service was integrated into the RACE service on the original
servers, but on Wiimmfi it has been implemented as an own service for easier
development and debugging.
All the details and protocols needed for competitions were found out
by Leseratte with some help by PokeAcer.
Leseratte has then developed the server software
needed to deliver the competitions to the Wiis,
and has also improved and updated amibus
first test patcher so that it now uploads old competition data to Wiimmfi
and permanently patches the Wii competition download to the Wiimmfi servers.
The competition service has been developed from January 2016 to June 2016,
with the first competition starting on May 10th. The server then moved from
Leserattes development server to the actual Wiimmfi server in July.
The PHP scripts of COMPETITION are run by an Apache web server.
JOB : Server to execute mkw-ana jobs
The JOB server is available since
July 2015.
It is a new server and was not available at GameSpy/Nintendo WFC,
as it was written differently.
mkw-ana connects to the JOB server to get
information about the current room and its players.
It also sends kick and ban jobs.
Special rights at the Wiimmfi server are needed for such kick and ban jobs.
In December 2015 the new backend and the
restart feature were implemented.
Since January 2016, watchdog will restart NATNEG
after a crash; but a crash never happened.
Plans
It is planned that the JOB server sends much more information to mkw-ana
after the new
GPCM server is finished.
More servers ...
There are some more servers in Wiimmfi that aren't for main game play:
- DLS1 (game file download server).
- GAMESTATS (game statistics, leaderboards and data).
More information will be added later.
