From https://forum.minetest.net/viewtopic.php?p=182737#p182737
If it helps any, here is the in_fov code from mobs redo mod that works (self is mob, pos is player position and self.fov is set to 90, self.rotate is usually 0 for front facing mobs):
CODE: SELECT ALL
```
in_fov = function(self,pos)
-- checks if POS is in self's FOV
local yaw = self.object:getyaw() + self.rotate
local vx = math.sin(yaw)
local vz = math.cos(yaw)
local ds = math.sqrt(vx^2 + vz^2)
local ps = math.sqrt(pos.x^2 + pos.z^2)
local d = { x = vx / ds, z = vz / ds }
local p = { x = pos.x / ps, z = pos.z / ps }
local an = ( d.x * p.x ) + ( d.z * p.z )
a = math.deg( math.acos( an ) )
```
Thank you everyone, especially TeTpaAka and Nore. All problems were at last solved, and the code now reports the angle difference between entities like it should! You get 0* when standing right in front of the mob, 90* when standing parallel to either side, 180* when standing behind... whereas flying up or down now accounts pitch correctly on top of that.
Here is the final and fully functional version. I hope it will be helpful to more people than just me, if anyone ever needs a simple and efficient FOV scanner for Minetest.
```
local function in_fov (pos1, pos2, yaw, pitch, fov)
local function yaw2vec (yaw, pitch)
-- we must invert the yaw for x to keep the result from inverting when facing opposite directions (0* becoming 180*)
return {x = math.sin(-yaw) * math.cos(pitch), y = math.sin(pitch), z = math.cos(yaw) * math.cos(pitch)}