Time Warp: Crash 1 minute after boot from bwmon_gargoyle

Report issues relating to bandwith monitoring, bandwidth quotas or QoS in this forum.

Moderator: Moderators

smiller
Posts: 15
Joined: Tue Apr 05, 2016 1:16 am

Time Warp: Crash 1 minute after boot from bwmon_gargoyle

Postby smiller » Mon Mar 27, 2017 10:31 pm

Hello folks.

Yesterday I started getting yelled at by my loving family because their internet was going up and down. (mostly down) The webui showed no uptime more than one minute before it went away and I had to log in again.

Using ssh, I got into the box and looked around. Logs up to about the 43 second mark, then it worked for a while, then a silent reboot.

I ran top with a one-second delay for a few reboot cycles. In every case, when it died the process on top was bwmon_gargoyle. After some googling I tried /etc/init.d/bwmon_gargoyle stop (and disable).

Stability returned.

Archer C7 v2 running 1.9.1. QoS enabled for download and upload - although upload may have enabled itself as I don't remember doing it.

The router gets a daily 4am reboot. Nothing had changed in my config in weeks. The only thing I can think of is that the qos/bandwidth logs had grown to some critical level that instantly killed the box when bwmon_gargoyle was started.

I took the opportunity to update to 1.9.2 with a fresh config. However, I did a backup of my config, so I have a copy of everything in /usr/data/bwmon in the crash scenario.

Has anyone seen this before? Any data I can share to help track this down?

And thanks for everything you do!
Last edited by smiller on Wed Mar 29, 2017 12:39 am, edited 1 time in total.

Lantis
Moderator
Posts: 4850
Joined: Mon Jan 05, 2015 5:33 am
Location: Australia

Re: Crash 1 minute after boot from bwmon_gargoyle

Postby Lantis » Tue Mar 28, 2017 3:49 am

You fell victim to the "time warp bug" (that's what i'm going to call it from now on so people can search for it).

BW Usage data got out of sync with real time, and when the module attempted to reload it, it caused a kernel panic.

The cause is unknown and hard to track down.

Purging /usr/data/bwmon causes the reboot loop to stop.

Solution:
Lantis wrote:Disconnect the WAN. Allow it to boot successfully. Purge /usr/data/bwmon/*. Reconnect WAN and reboot.

If disconnecting the WAN doesn't stop the boot loop, drop it into failsafe and do the same thing.
Routers: Various ar71xx/mvebu/x86-64
http://lantisproject.com/downloads/gargoyle_ispyisail.php for the latest releases

smiller
Posts: 15
Joined: Tue Apr 05, 2016 1:16 am

Re: Time Warp: Crash 1 minute after boot from bwmon_gargoyle

Postby smiller » Wed Mar 29, 2017 12:40 am

Lantis,

Thanks.

I updated the thread title to put "Time Warp" in it.

Now I'm humming to myself "Let's do the time warp again!"

Lantis
Moderator
Posts: 4850
Joined: Mon Jan 05, 2015 5:33 am
Location: Australia

Re: Time Warp: Crash 1 minute after boot from bwmon_gargoyle

Postby Lantis » Wed Mar 29, 2017 5:01 am

Routers: Various ar71xx/mvebu/x86-64
http://lantisproject.com/downloads/gargoyle_ispyisail.php for the latest releases

michaelmantis
Posts: 4
Joined: Sun Jul 22, 2018 3:16 am

Re: Time Warp: Crash 1 minute after boot from bwmon_gargoyle

Postby michaelmantis » Sun Mar 10, 2019 11:12 am

Thank goodness for this forum!

Woke up today (day of daylight savings) to my router rebooting every minute.

Was able to get into the web interface (basically copied the "bandwidth.sh" link into my browser and hit refreshed non stop until I was able to get it right when the router rebooted) and hit "Delete Data" under the "Bandwidth Usage Table".

Been 15 minutes an no reboots yet. Did I pretty much solve the issue of needing to clean the data from /usr/data/bwmon (without having to SSH in?)

User avatar
RomanHK
Posts: 265
Joined: Sat May 04, 2013 4:18 pm
Location: Czech Republik

Re: Time Warp: Crash 1 minute after boot from bwmon_gargoyle

Postby RomanHK » Sun Mar 10, 2019 1:39 pm

michaelmantis wrote:Did I pretty much solve the issue of needing to clean the data from /usr/data/bwmon (without having to SSH in?)

If you do this, click Status - Bandwidth Usage - Delete Data in the GUI to solve the problem.
Turris Omnia with Gargoyle 1.11.x
Linksys WRT3200ACM with Gargoyle 1.11.x
TL-WR1043ND v2 with Gargoyle 1.10.0

michaelmantis
Posts: 4
Joined: Sun Jul 22, 2018 3:16 am

Re: Time Warp: Crash 1 minute after boot from bwmon_gargoyle

Postby michaelmantis » Sun Mar 10, 2019 9:10 pm

Quickly hitting "Status - Bandwidth Usage - Delete Data in the GUI" (within a minute of boot up) solved the problem.

Been up and running for 10 hours with no issue.


Return to “Monitoring / Quota / QoS Issues”

Who is online

Users browsing this forum: No registered users and 2 guests