There is a problem deleting entries from the hosts file (the DNS server in pfSense). When you add an entry, it is read immediately. When you delete an entry, you also have to remember to restart anmuscle.sh before the old entry disappears from any DNS lookup.
This is a longstanding issue as I remember it from ClearOS.
This was indeed an issue in the past. It is resolved with version 4.14.2.
You can now delete an IP from the dashboard and there’s no muscle restart required.
If you have this issue appear to persist where the IP returns, it means it is still visible in the ARP table, so it will just re-enrol after deletion.
For a specific support issue, you can also feel free to reach out to support@adamnet.works.
Hmm. This happened 2 days ago after upgrading to pfSense to 2.8.0 and installing whatever was the latest adamONE version then. It should have been 4.14.2-1.
I deleted 3 CNAMEs from one A record in pfSense and added them to another as I was setting up a reverse proxy. The faulty lookups cleared immediately I restarted adamONE.
Ok, so I may be wrong. It seems to be working. Only Windows is struggling with it now, so not the fault of adamONE.
I have deleted and re-added hosts, changing the IP, and also just changed the IP, and both seem to be working now.
So my PC was off overnight. I started it this morning and an nslookup returned both the old and new IP for a host that I’d changed yesterday. A restart of anmuscle.sh fixed it.
… but I cannot get a Ubuntu host to stop picking up both entries even after:
resolvectl flush-caches
systemctl restart systemd-resolved
root@hostd:~# nslookup pi2
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
Name: pi2.howitts.co.uk
Address: 172.17.1.30
Name: pi2.howitts.co.uk
Address: 172.17.3.132
172.17.1.30 should not be there.
After clearing the cache on hostd and doing another nslookup, this is what I see in tcpdump in pfsense:
08:19:35.768406 IP 172.17.2.35.60740 > 172.17.2.254.53: 27925+ [1au] A? api.siacentral.com. (47)
08:19:35.768557 IP 172.17.2.35.59752 > 172.17.2.254.53: 16030+ [1au] AAAA? api.siacentral.com. (47)
08:19:35.768892 IP 172.17.2.254.53 > 172.17.2.35.60740: 27925 2/0/1 A 172.67.199.239, A 104.21.76.196 (79)
08:19:35.769052 IP 172.17.2.254.53 > 172.17.2.35.59752: 16030 2/0/1 AAAA 2606:4700:3031::6815:4cc4, AAAA 2606:4700:3033::ac43:c7ef (103)
08:19:39.982007 IP 172.17.2.35.39239 > 172.17.2.254.53: 42883+ [1au] A? pi2.howitts.co.uk. (46)
08:19:39.982424 IP 172.17.2.254.53 > 172.17.2.35.39239: 42883 2/0/1 A 172.17.1.30, A 172.17.3.132 (78)
08:19:39.984217 IP 172.17.2.35.46192 > 172.17.2.254.53: 28748+ [1au] AAAA? pi2.howitts.co.uk. (46)
08:19:39.984361 IP 172.17.2.254.53 > 172.17.2.35.46192: 28748 0/1/1 (88)
So pfSense is still returning the bad record.
This is a horrible mess and I think I have worked out what is causing it. If you see the attached screenshot from the devices list:
root@hostd:~# nslookup laptop-nick
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
Name: laptop-nick.howitts.co.uk
Address: 172.17.2.116
Name: laptop-nick.howitts.co.uk
Address: 172.17.2.123
Name: laptop-nick.howitts.co.uk
Address: 172.17.2.100
Name: laptop-nick.howitts.co.uk
Address: 2a00:23c6:df9d:f101:d3e7:ff5d:77ca:bf01
Name: laptop-nick.howitts.co.uk
Address: 2a00:23c6:df9d:f101:c106:9038:a247:9390It begins to make sense. adamONE is returning DNS results from any old address it has remembered for the device except for the 169.254 address. The hosts file only contains the one record, 172.17.2.116.
I cleared my earlier problem by deleting the devices from the Device List where the hosts file had changed and suddenly I got correct DNS.
Now, as it happens, all the hosts I was changing were remote OpenVPN roadwarriors or servers configured as roadwarriors but these all appear in the devices list. I was moving them from one OpenVPN server using the 172.17.1.0/24 subnet to another OpenVPN server using 172.17.3.0/24. The 172.17.1.x IPs were presented first when doing a DNS lookup, so it took ages to connect to these devices by SSH as it had to wait until the app got fed up with no response from 172.17.1.x and switched to the next IP, so the 172.17.3.x IP. This also depends on how well the app was written and how quickly it would switch IPs (ping never would, for example).
Now, having deleted the devices from the Device List so they reset, I can connect to them much quicker as the apps immediately try the working IP.
Somehow this issue needs to be fixed as it is causing terrible network performance.
@NickH the automated enrolment process of adam:ONE makes use of the ARP table. Unfortunately sometimes the OpenVPN IPs are bound to the FreeBSD interface and therefore are enrolled as part of that interface.
There is an override mechanism you can use by following these steps:
service anmuscle.sh stopservice anmuscle.sh startEach IP address (MAC address optional in this case) will now ignore any IP:MAC re-assignment automation.
The original OpenVPN IP’s were handled by ClearOS and I routed traffic to the OpenVPN subnet via ClearOS in pfSense. They were only known in pfSense from the hosts file. I doubt if there is any ARP involved at this point as ARP doesn’t pass through OpenVPN tunnels.
Now I have switched to OpenVPN in pfSense. In the OpenVPN setup there, I have used Client Specific Overrides to prefer certain OpenVPN IP’s. Note these are not static IP’s, but just preferred IP’s and OpenVPN is free to not give them out if it doesn’t want to. Again, none of the devices appear in ARP table, so adamONE must be using some other mechanism to enrol them.
Ultimately it means every time we maintain the hosts file for OpenVPN IPs, we have to remember to maintain adamONE alongside it. Yuck!
If you are able to reproduce the problem with a debug version and log level 6 enabled, we would take another closer look. Just initiate a support request via email and ask for it to be assigned to me.
By the way, I was able to reproduce the problem you described @NickH insomuch as my OpenVPN IP was bound to the LAN interface of the appliance.
Upon investigating the debug log, I saw the FreeBSD kernel craft the packet to the LAN interface MAC address. Our automated mapping of IP-to-MAC is by the sent packet, which our packet monitor observes:
I 15/7 11:08:23.478957 11226 NS hwaddr a0:b3:39:1b:85:7e ipaddr 10.0.8.2
I 15/7 11:08:23.482219 11226 NS hwaddr 00:00:5e:00:01:06 ipaddr 10.0.8.2
Unfortunately this confirms what I stated above, in that it is a function completely outside of adam:ONE, and as long as we want the IP-to-MAC tracking to be automated, it is an all-or-nothing option.