We have included technical information for third party developers
to use our facemotion3d app with their pipeline.  See below

- To receive data via UDP

If you send the string "FACEMOTION3D_OtherStreaming" from the PC side to port 49993 on the iOS side, Facemotion3d will return a reply to port 49993 with UDP at 60FPS.


-Python Example
from socket import *
DstIP = "192.168.1.7"
DstPort = 49993
DstAddr = (DstIP,DstPort)
udpClntSock = socket(AF_INET, SOCK_DGRAM)
data = "FACEMOTION3D_OtherStreaming"
data = data.encode('utf-8')
udpClntSock.sendto(data,DstAddr)
server = socket(AF_INET, SOCK_DGRAM)
server.bind(("", 49983))
server.settimeout(0.05)

while True:
try:
messages, address = server.recvfrom(8192)
print(messages.decode('utf-8'))
except:
pass

-The character string sent per frame is as follows.
eyeLookOutLeft&3|mouthPressRight&14|jawForward&8|browInnerUp&19|jawLeft&4|mouthShrugUpper&28|eyeSquintLeft&4|eyeWideLeft&0|mouthPucker&5|mouthLowerDownLeft&6|eyeBlinkLeft&29|mouthShrugLower&20|mouthSmileLeft&0|cheekSquintRight&9|eyeWideRight&0|mouthFunnel&4|mouthStretchRight&13|browDownRight&2|tongueOut&0|cheekSquintLeft&8|mouthStretchLeft&17|mouthUpperUpRight&10|noseSneerLeft&28|browOuterUpLeft&0|eyeLookInRight&15|mouthDimpleRight&7|mouthRollUpper&5|mouthLowerDownRight&6|mouthClose&6|mouthFrownLeft&3|eyeBlinkRight&29|mouthFrownRight&2|mouthUpperUpLeft&11|browDownLeft&1|eyeLookOutRight&0|cheekPuff&4|eyeLookInLeft&0|mouthDimpleLeft&8|eyeLookDownLeft&36|eyeSquintRight&4|mouthRollLower&7|eyeLookUpLeft&0|eyeLookUpRight&0|mouthSmileRight&0|jawRight&0|noseSneerRight&25|mouthPressLeft&13|jawOpen&4|browOuterUpRight&0|eyeLookDownRight&36|mouthLeft&1|mouthRight&0|FM_SAD&0|FM_ANGRY&0|FM_VF&0|FM_TH&0|FM_browUpRight&0|FM_browUpLeft&0|=head#-22.167973,0.45402408,-4.8798294,-0.039975993,-0.048437987,-0.45562273|rightEye#12.759762,5.527712,1.2244859|leftEye#12.862795,1.264116,0.28145725|

-The character strings sent per frame are as follows.
BlendShape Name&Parameters (0 ~ 100) | BlendShape Names-Parameters (0 ~ 100) | .... | = head # Euler angles X (degree), Euler angles Y, Euler angles Z, Position values X, Position values Y, Position values Z | rightEye #Euler angles X, Euler angles Y, Euler angles Z | leftEye #Euler angles X, Euler angles Y, Euler angles Z |
The parameter may contain negative values or the value may exceed 100.

Angle-related data is sent in degrees, not radians.

- To receive data via TCP / IP
If it is UDP, frames may be dropped, so there is also a method of receiving with TCP / IP.
If you send the character string "FACEMOTION3D_OtherStreaming|protocol=tcp" to port 49993 on the iOS side by UDP, the data will be sent to port 49986 on the PC side.

 
The character string to be sent is just the character string "___FACEMOTION3D" added to the character string sent at the time of UDP.

-sample
BlendShape Name&Parameter (0 ~ 100) | BlendShape Name-Parameter (0 ~ 100) | .... | = head # Euler angles X, Euler angles Y, Euler angles Z, moving values X, moving values Y, moving values Z | rightEye #Euler angles X, Euler angles Y, Euler angles Z | leftEye #Euler angles X, Euler angles Y, Euler angles Z | ___ Facemotion3d
When receiving from iOS to PC with TCP / IP, a character string with broken frames will be sent, so use "___FACEMOTION3D" at the end as a delimiter.
To stop sending TCP / IP, send the string "StopStreaming_FACEMOTION3D" to port 49993.


-Sample Python usage example received over TCP / IP

from socket import socket, AF_INET, SOCK_DGRAM, SOCK_STREAM
#UDP send
HOST = ''
PORT = 49993
ADDRESS = "192.168.1.7" #iOS address
s = socket(AF_INET, SOCK_DGRAM)
msg = "FACEMOTION3D_OtherStreaming|protocol=tcp"
s.sendto(msg.encode(), (ADDRESS, PORT))
s.close()

#create tcp server
with socket(AF_INET, SOCK_STREAM) as tcp_server:
tcp_server.bind(('192.168.1.5', 49986)) #192.168.1.5 is pc address
tcp_server.listen(1)

while True:
conn, addr = tcp_server.accept()
with conn:
while True:
data = conn.recv(10000)
print(data.decode('utf-8'))

-To receive recorded data in iOS
If you send the character string "OtherBakeRecordedAnimation_FACEMOTION3D" to port 49993 on the iOS side by UDP, the recorded data will be sent to port 49987 on the PC side by TCP / IP.
The string that is sent is in the format [".....", ".....", "....."]___FACEMOTION3D.
The [".....", ".....", "....."] part is in JSON format and stores the same character strings used in UDP and TCP/IP.
Data recorded at 60 FPS is always stored. (Even if the processing drops to 30FPS in iOS, it will be recorded at 60FPS)

-To receive recorded data and audio in iOS
If you send the character string "OtherBakeRecordedAnimationAndAudio_FACEMOTION3D" to port 49993 on the iOS side by UDP, the recorded data will be sent to port 49987 on the PC side by TCP / IP.
The string that is sent is in the format [".....", ".....", "....."]===(Binary converted to hexadecimal string)___FACEMOTION3D.
The [".....", ".....", "....."] part is in JSON format and stores the same character strings used in UDP and TCP/IP.

-Show User that you are currently baking
To display "Currently baking" in the iOS app, send the "To hide "processing the data", send "BakeFinishOther_FACEMOTION3D" to port 49993." to port 49993.

-To start recording within the iOS app
To start recording within the iOS app, send "FACEMOTION3D_StartRecording" to port 49993.

-To stop recording within the iOS app
To stop recording within the iOS app, send "FACEMOTION3D_StopRecording" to port 49993.

-To stop streaming within the iOS app
To stop streaming within the iOS app, send "StopStreaming_FACEMOTION3D" to port 49993.

-Run the Reset Button
Send the string "FACEMOTION3D_resetButton" to iOS port 49993.


-Adding Information
-Facemotion3d will not accept external commands for 0.02 seconds once it accepts commands.
-Currently, the only server that receives commands is the UDP server. With UDP, data may sometimes be missed, so sending data about three times will increase the probability of accurate communication.
3rd party developers do not necessarily have to use the Other license.
3rd party developers can use their favorite licenses such as Unity license and Blender license to choose the type of data to send.
Please understand that the price of each license may change in the future depending on the situation.

How can I retarget the fbx output to my character?
-you should have arkit compatible 51 blendshapes that has apple arkit blendshape naming convention.   Not sure where to begin to create those awesome blendshapes yourself?  Contact us for more information on how you too can make awesome animations that you see in our gallery!

How do I create characters that will work with the Facemotion3d app myself?
-You have to know how to sculpt or at least be familiar with maya or blender (or other dcc) so that you can make your own 52 arkit compatible blendshapes for your character's face.   You can get an idea of what those blendshapes are supposed to be like by downloading a sample fbx recording of the Facemotion3d app and importing to maya or blender.  The 52 blendshapes are there for you to see (as well as all the names for the blendshapes)
-we have a pipeline solution that can help you create your own characters with blendshapes that are super optimized with out facemotion3d app.

What do I need to make Virtual characters for Vtubing?
-You need a realtime render engine like Unity or Unreal Engine.  And also characters that will stream live to those platforms and a solution to make those things happen.  It's a quite involved process but we make things easy for you, but be ready to learn some new things.. :)

I can't connect to my host pc application

• make sure that you are using the right IP address of your iphone FM app in your host program's plugin iphone IP field.  Your iphone FM IP can be found on the settings page (scroll down)
• also check your local IP.  Local IP changes often randomly and the plugins automatic local IP setting function sometimes can't keep up with random IP changes that can occur every few days.  You can find your local IP by doing command promptand entering "ipconfig" (explained HERE) or you can simply input "0.0.0.0" in your maya/blender plugin's locap IP field. 

It connects to my iphone FM app (wifi icon turns to square), but the character  on PC side is not moving
-It's possible that you need inbound firewall settings to allow connection.   Go to Control panel, firewall, advanced security, inbound rules, then find maya, blender or unity and set the inbound rule to allow incoming.

Face is moving but it's too slow and choppy
-for DAZ make sure your character is at subD 0 level, has no fiber mesh hair or too dense clothing, and the fps in DAZ plugin is set to 15fps (sometime in slower systems it may need to go down to 7.5 fps)
-for maya, set subD to 0 and also set fps in the app to 20 or 15 if needed
-for blender set app's fps to 20 or 15.  stay in shaded mode and not in EEVEE.  Also turn off turbo smooth or subD.