Hi! Recently, probably after the update, I can no longer access mytools.management.
When I ping it, it resolves correctly to our gateway.
Any idea what could be wrong?
hello @edanpedragosa possible reasons can include:
- you have another binding that is using port 80 on the gateway - in the case of pfSense® make sure you check with
sockstat |grep \:80and make sure onlyanmuscleowns port 80 (if another process does, that is likely your conflict - most often this is when System → Advanced → WebGUI redirect isn’t disabled - you have a missing or out-of-order firewall rule - the TCP port 80 from LAN to itself should be high on your order of rules, and make sure no blocking rule is ahead of it
- you have an HSTS-cached entry in your browser that is trying to redirect http to https - in this case empty your browser cache completely (this cached entry can happen as a result of #1 above)
- you’re running endpoint software that thinks public FQDNs resolving locally should be blocked – there are several “anti-virus” packages that now have this as a false positive
Hope this helps!
We’ve expanded on this a bit with a knowledge base article here:
It happened after the update, I tried restarting the adam service but it didn’t help.
I restarted the pfsense box and everything is back to normal.
1 Like
Make sure you confirm there’s no port 80 conflict - on startup, sometimes anmuscle can beat the WebGUI redirector binding but if you stop/restart anmuscle it’ll fail to restart until you reboot again.
I believe there’s no port conflict on 80.
Also, from time to time now, we are losing connection to mytools.management.
I adjusted the log level to 3 from 6 now and will monitor if there’s a difference.
At the moment it is down, it’s important to find out the root cause. I would suggest running a tcpdump on the gateway to see if traffic is arriving or not. If not, the issue may be browser-side. For example, if DoH is running on a workstation to a public resolver that is reachable, it would override the local LAN resolution of mytools.management.
I will try tcpdump when it happens again. There is still internet though, it just cannot go to mytools.management nor show the blocked page message when you visit a blocked site.
It didn’t happen yet after changing the log level to 3 from 6 and lowered the log file size limit.
Hopefully everything will be well again for a longer time, I’ll keep you posted!
TY!
1 Like
And so it happened again today.
tcpdump does not spew anything
I tried restarting the webConfigurator from ssh and mytools.management becomes available again.
My webConfigurator settings is as in the image below:
On system logs I can see:
pid 87249 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)
Is there a way for anmuscle to automatically restart the webconfigurator when something is off?
Also, I’m getting a lot of these in PFSense System Logs:
|Jan 26 10:39:08 |kernel ||pid 77796 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)|
|—|—|—|—|
|Jan 26 11:09:52 |kernel ||pid 40858 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)|
|Jan 26 11:25:33 |kernel ||pid 52899 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)|
|Jan 26 12:30:00 |php |48702 |rc.update_urltables: /etc/rc.update_urltables: Starting up.|
|Jan 26 12:30:00 |php |48702 |rc.update_urltables: /etc/rc.update_urltables: Sleeping for 60 seconds.|
|Jan 26 12:31:00 |php |48702 |rc.update_urltables: /etc/rc.update_urltables: Starting URL table alias updates|
|Jan 26 12:31:01 |php |48702 |rc.update_urltables: /etc/rc.update_urltables: Updated ADAMnetworksSupport content from https://adamnetworks.dev/pub/fwaliases/-/raw/master/ips/adamsupport.txt: no changes.|
|Jan 26 12:31:01 |php |48702 |rc.update_urltables: /etc/rc.update_urltables: Updated ExplicitlyAllowedNetworks content from http://172.27.7.71/infusions/portal_access/util/allowedips.txt: no changes.|
|Jan 26 12:31:01 |php |48702 |rc.update_urltables: /etc/rc.update_urltables: Updated proxytor content from http://172.27.7.71/infusions/portal_access/util/proxytor.txt: no changes.|
|Jan 26 12:31:01 |php |48702 |rc.update_urltables: /etc/rc.update_urltables: Updated psiphonnetwork content from http://172.27.7.71/infusions/portal_access/util/psiphonips.txt: no changes.|
|Jan 26 13:02:12 |kernel ||pid 59478 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)|
|Jan 26 14:07:51 |kernel ||pid 21190 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)|
|Jan 26 14:18:51 |kernel ||pid 88111 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)|
|Jan 26 14:26:51 |kernel ||pid 88902 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)|
|Jan 26 14:29:27 |kernel ||pid 14187 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)|
|Jan 26 14:32:06 |kernel ||pid 97213 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)|
|Jan 26 14:34:27 |kernel ||pid 16925 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped) |
@edanpedragosa that looks like unexpected core dumps - there were some edge cases that were addressed in the latest rapid release - would you please try it as per adamnet.io/rr
I’m on the updated release, that is when the problem started happening on my install.
In your HA environment, make sure both instances are running this one:
[2.5.2-RELEASE][admin@pfsense.local.ad]/root: anmuscle -v
AdamnetMuscle version 4.8.0-214 | Copyright Adam Networks Inc. All rights reserved.
Okay, so both box should be on the same version every time right?
I’ve upgraded the other box now, hoping it won’t happen again now.
TY again for your response!
I updated both now and it is still happening:
Jan 28 07:58:04 kernel pid 83494 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)
Jan 28 08:04:02 php-fpm 342 /index.php: Session timed out for user 'admin' from: 172.27.0.12 (Local Database Fallback)
Jan 28 08:04:04 php-fpm 342 /index.php: Successful login for user 'admin' from: 172.27.0.12 (Local Database Fallback)
Jan 28 09:23:47 kernel pid 55276 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)
Jan 28 10:00:20 kernel pid 84741 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)
Jan 28 10:08:04 kernel pid 87215 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)
Jan 28 11:02:19 kernel pid 69411 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)
Jan 28 11:24:29 kernel pid 1066 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)
Jan 28 11:52:56 kernel pid 89698 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)
Jan 28 12:30:00 php 23900 rc.update_urltables: /etc/rc.update_urltables: Starting up.
Jan 28 12:30:00 php 23900 rc.update_urltables: /etc/rc.update_urltables: Sleeping for 55 seconds.
Jan 28 12:30:55 php 23900 rc.update_urltables: /etc/rc.update_urltables: Starting URL table alias updates
Jan 28 12:30:56 php 23900 rc.update_urltables: /etc/rc.update_urltables: Updated ADAMnetworksSupport content from https://adamnetworks.dev/pub/fwaliases/-/raw/master/ips/adamsupport.txt: no changes.
Jan 28 12:30:56 php 23900 rc.update_urltables: /etc/rc.update_urltables: Updated ExplicitlyAllowedNetworks content from http://172.27.7.71/infusions/portal_access/util/allowedips.txt: no changes.
Jan 28 12:30:56 php 23900 rc.update_urltables: /etc/rc.update_urltables: Updated proxytor content from http://172.27.7.71/infusions/portal_access/util/proxytor.txt: no changes.
Jan 28 12:30:56 php 23900 rc.update_urltables: /etc/rc.update_urltables: Updated psiphonnetwork content from http://172.27.7.71/infusions/portal_access/util/psiphonips.txt: no changes.
Jan 28 12:57:23 kernel pid 90721 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)
Jan 28 13:34:08 kernel pid 72585 (anmuscle), jid 0, uid 0: exited on signal 11 (core dumped)
Jan 28 13:41:21 sshd 56766 Accepted keyboard-interactive/pam for root from 172.27.0.12 port 7271 ssh2
Jan 28 13:41:33 php 1026 rc.restart_webgui: Creating rrd update script
Jan 28 13:42:03 php 47860 rc.restart_webgui: Creating rrd update script
Jan 28 13:42:12 php-fpm 88824 /index.php: Session timed out for user 'admin' from: 172.27.0.12 (Local Database Fallback)
Jan 28 13:42:14 php-fpm 88824 /index.php: Successful login for user 'admin' from: 172.27.0.12 (Local Database Fallback)
Jan 28 13:42:23 check_reload_status 380 Syncing firewall
Jan 28 13:42:23 check_reload_status 380 Reloading filter
Jan 28 13:42:23 php-fpm 18121 /system_advanced_admin.php: webConfigurator configuration has changed. Restarting webConfigurator.
Jan 28 13:42:23 check_reload_status 380 webConfigurator restart in progress
Jan 28 13:42:26 php-fpm 341 /rc.restart_webgui: Creating rrd update script
@edanpedragosa we found another situation where this occurred, was able to diagnose it being caused by a malformed dns query… expect a new rapid release tomorrow.
hi again @edanpedragosa try the latest rapid release now at adamnet.io/rr
Okay, I’ve installed the latest rapid release. I hope everything’s good.
Do we have to restart the PFSense box? I just restarted anmuscle but mytools.management is still not accessible.
Thank you again!
Update: After quite sometime now, mytools.management is still accessible.
I hope it will continue working from now on.
Thanks!
1 Like
