ath10k_pci: crash after ~10d uptime
Username: Stijn Tintel
Origin: https://bugs.openwrt.org/index.php?do=details&task_id=333
After ~10 days of uptime, ath10k crashed and I could no longer associate.
Device: D-Link DAP-2695 rev A1
Running: LEDE r2370
Wireless config:
config wifi-device radio0 option type mac80211 option channel 100 option country BE option txpower 20 option hwmode 11a option path 'pci0000:00/0000:00:00.0' option htmode VHT80 option disabled 0
config wifi-iface option device 'radio0' option ifname 'wlan0' option encryption 'psk2' option ieee80211d '1' option ieee80211w '1' option peerkey '1' option rsn_preauth '1' option network 'lan' option mode 'ap' option ssid 'xxxxxxxx' option key 'XXXXXXXX'
config wifi-iface option device 'radio0' option ifname 'wlan2' option encryption 'psk2' option ieee80211d '1' option ieee80211w '1' option peerkey '1' option rsn_preauth '0' option network 'guest' option mode 'ap' option ssid 'Guests' option key 'YYYYYYYY' #option disassoc_low_ack '0'
config wifi-device radio1 option type mac80211 option channel 1 option country BE option txpower 20 option hwmode 11g option path 'platform/qca955x_wmac' option htmode HT40 #option noscan 1
config wifi-iface option device 'radio1' option ifname 'wlan1' option encryption 'psk2' option ieee80211d '1' option ieee80211w '1' option peerkey '1' option rsn_preauth '1' option mode 'ap' option network 'lan' option ssid 'xxxxxxxx' option key 'XXXXXXXX'
config wifi-iface option device 'radio1' option ifname 'wlan3' option encryption 'psk2' option ieee80211d '1' option ieee80211w '1' option peerkey '1' option rsn_preauth '0' option network 'guest' option mode 'ap' option ssid 'Guests' option key 'YYYYYYYY' #option disassoc_low_ack '0'
Kernel log:
[ 54.439387] ath10k_pci 0000:00:00.0: pci irq legacy oper_irq_mode 1 irq_mode 0 reset_mode 0 [ 54.652312] ath10k_pci 0000:00:00.0: Direct firmware load for ath10k/pre-cal-pci-0000:00:00.0.bin failed with error -2 [ 54.663197] ath10k_pci 0000:00:00.0: Falling back to user helper [ 54.742675] firmware ath10k!pre-cal-pci-0000:00:00.0.bin: firmware_loading_store: map pages failed [ 54.752039] ath10k_pci 0000:00:00.0: Direct firmware load for ath10k/cal-pci-0000:00:00.0.bin failed with error -2 [ 54.762568] ath10k_pci 0000:00:00.0: Falling back to user helper [ 61.648688] ath10k_pci 0000:00:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043202ff sub 0000:0000 [ 61.658088] ath10k_pci 0000:00:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 1 [ 61.671139] ath10k_pci 0000:00:00.0: firmware ver 10.2.4.70.54 api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9 [ 61.681571] ath10k_pci 0000:00:00.0: Direct firmware load for ath10k/QCA988X/hw2.0/board-2.bin failed with error -2 [ 61.692178] ath10k_pci 0000:00:00.0: Falling back to user helper [ 61.771624] firmware ath10k!QCA988X!hw2.0!board-2.bin: firmware_loading_store: map pages failed [ 61.786987] ath10k_pci 0000:00:00.0: board_file api 1 bmi_id N/A crc32 bebc7c08 [ 62.893736] ath10k_pci 0000:00:00.0: htt-ver 2.1 wmi-op 5 htt-op 2 cal file max-sta 128 raw 0 hwcrypto 1 [ 63.001904] ath: EEPROM regdomain: 0x0 [ 63.001920] ath: EEPROM indicates default country code should be used [ 63.001928] ath: doing EEPROM country->regdmn map search [ 63.001945] ath: country maps to regdmn code: 0x3a [ 63.001955] ath: Country alpha2 being used: US [ 63.001963] ath: Regpair used: 0x3a [ 63.284566] ath: EEPROM regdomain: 0x0 [ 63.284583] ath: EEPROM indicates default country code should be used [ 63.284591] ath: doing EEPROM country->regdmn map search [ 63.284608] ath: country maps to regdmn code: 0x3a [ 63.284617] ath: Country alpha2 being used: US [ 63.284625] ath: Regpair used: 0x3a [ 72.829974] ath: EEPROM regdomain: 0x8038 [ 72.834045] ath: EEPROM indicates we should expect a country code [ 72.840265] ath: doing EEPROM country->regdmn map search [ 72.845651] ath: country maps to regdmn code: 0x37 [ 72.850522] ath: Country alpha2 being used: BE [ 72.855022] ath: Regpair used: 0x37 [ 72.858554] ath: regdomain 0x8038 dynamically updated by user [ 72.864452] ath: EEPROM regdomain: 0x8038 [ 72.868513] ath: EEPROM indicates we should expect a country code [ 72.874707] ath: doing EEPROM country->regdmn map search [ 72.880092] ath: country maps to regdmn code: 0x37 [ 72.884943] ath: Country alpha2 being used: BE [ 72.889458] ath: Regpair used: 0x37 [ 72.892996] ath: regdomain 0x8038 dynamically updated by user [920549.560152] ath10k_pci 0000:00:00.0: failed to delete peer c0:ee:fb:4a:18:b7 for vdev 0: -145 [920549.568951] ath10k_pci 0000:00:00.0: found sta peer c0:ee:fb:4a:18:b7 (ptr 8d43f600 id 95) entry on vdev 0 after it was supposedly removed [920549.581722] ------------[ cut here ]------------ [920549.586618] WARNING: CPU: 0 PID: 1431 at compat-wireless-2016-10-08/net/mac80211/sta_info.c:964 sta_set_sinfo+0x92c/0x9e0 mac80211 [920549.598968] Modules linked in: ath9k ath9k_common crc_ccitt ath9k_hw act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_hfsc sch_ingress ath10k_pci ath10k_core ath mac80211 cfg80211 compat gpio_button_hotplug [920549.620582] CPU: 0 PID: 1431 Comm: hostapd Not tainted 4.4.35 #0 [920549.626772] Stack : 803c2d64 00000000 00000001 80420000 8fe26580 8040acc3 803a4424 00000597 [920549.626772] 8047379c 8ff8d000 00000000 77415e94 00000000 800a7198 803a9a90 80400000 [920549.626772] 00000003 8ff8d000 803a7e9c 8e5f3b2c 00000000 800a5114 80400000 00000000 [920549.626772] 00000001 801f3100 00000000 00000000 00000000 00000000 00000000 00000000 [920549.626772] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [920549.626772] ... [920549.663219] Call Trace: [920549.665810] [<80071ae0>] show_stack+0x50/0x84 [920549.670323] [<80081860>] warn_slowpath_common+0xa0/0xd0 [920549.675724] [<80081918>] warn_slowpath_null+0x18/0x24 [920549.680994] [<8f5870a4>] sta_set_sinfo+0x92c/0x9e0 [mac80211] [920549.686990] [<8f587188>] __sta_info_destroy+0x30/0x48 [mac80211] [920549.693222] [<8f587238>] sta_info_destroy_addr_bss+0x38/0x60 [mac80211] [920549.700118] [<8f54c144>] cfg80211_check_station_change+0xed8/0x1390 [cfg80211] [920549.707564] [920549.709166] ---[ end trace e15a227055dd36c2 ]--- [920866.693883] ath10k_pci 0000:00:00.0: failed to install key for vdev 0 peer b0:df:3a:12:79:80: -145 [920866.703088] wlan0: failed to remove key (0, b0:df:3a:12:79:80) from hardware (-145) [920866.719519] ath10k_pci 0000:00:00.0: cipher 0 is not supported [920866.725602] ath10k_pci 0000:00:00.0: failed to remove peer wep key 0: -122 [920866.732670] ath10k_pci 0000:00:00.0: failed to clear all peer wep keys for vdev 0: -122 [920866.740901] ath10k_pci 0000:00:00.0: failed to disassociate station: b0:df:3a:12:79:80 vdev 0: -122 [920866.750169] ------------[ cut here ]------------ [920866.755066] WARNING: CPU: 0 PID: 1431 at compat-wireless-2016-10-08/net/mac80211/sta_info.c:956 sta_set_sinfo+0x8d8/0x9e0 mac80211 [920866.767421] Modules linked in: ath9k ath9k_common crc_ccitt ath9k_hw act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_hfsc sch_ingress ath10k_pci ath10k_core ath mac80211 cfg80211 compat gpio_button_hotplug [920866.789044] CPU: 0 PID: 1431 Comm: hostapd Tainted: G W 4.4.35 #0 [920866.796465] Stack : 803c2d64 00000000 00000001 80420000 8fe26580 8040acc3 803a4424 00000597 [920866.796465] 8047379c 8ff8d000 00000000 77415e94 00000000 800a7198 803a9a90 80400000 [920866.796465] 00000003 8ff8d000 803a7e9c 8e5f3b2c 00000000 800a5114 00000000 00000000 [920866.796465] 80409dd0 801f3100 00000000 00000000 00000000 00000000 00000000 00000000 [920866.796465] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [920866.796465] ... [920866.832912] Call Trace: [920866.835502] [<80071ae0>] show_stack+0x50/0x84 [920866.840016] [<80081860>] warn_slowpath_common+0xa0/0xd0 [920866.845417] [<80081918>] warn_slowpath_null+0x18/0x24 [920866.850691] [<8f587050>] sta_set_sinfo+0x8d8/0x9e0 [mac80211] [920866.856685] [<8f587188>] __sta_info_destroy+0x30/0x48 [mac80211] [920866.862914] [<8f587238>] sta_info_destroy_addr_bss+0x38/0x60 [mac80211] [920866.869817] [<8f54c144>] cfg80211_check_station_change+0xed8/0x1390 [cfg80211] [920866.877266] [920866.878867] ---[ end trace e15a227055dd36c3 ]--- [920869.873901] ath10k_pci 0000:00:00.0: failed to delete peer b0:df:3a:12:79:80 for vdev 0: -11 [920869.882572] ath10k_pci 0000:00:00.0: found sta peer b0:df:3a:12:79:80 (ptr 8f3f3000 id 243) entry on vdev 0 after it was supposedly removed [920872.893898] ath10k_pci 0000:00:00.0: failed to set beacon mode for vdev 0: -11 [920875.893871] ath10k_pci 0000:00:00.0: failed to set dtim period for vdev 0: -11 [920878.913873] ath10k_pci 0000:00:00.0: failed to recalculate rts/cts prot for vdev 0: -11 [920881.913852] ath10k_pci 0000:00:00.0: failed to set protection mode 0 on vdev 0: -11 [920884.913828] ath10k_pci 0000:00:00.0: failed to set preamble for vdev 0: -11 [920887.923826] ath10k_pci 0000:00:00.0: failed to set beacon mode for vdev 1: -11 [920890.923830] ath10k_pci 0000:00:00.0: failed to set dtim period for vdev 1: -11 [920893.933787] ath10k_pci 0000:00:00.0: failed to recalculate rts/cts prot for vdev 1: -11 [920896.933770] ath10k_pci 0000:00:00.0: failed to set protection mode 0 on vdev 1: -11 [920899.933771] ath10k_pci 0000:00:00.0: failed to set preamble for vdev 1: -11
Kernel log after running “wifi” in the crashed state:
[921144.042842] ath10k_pci 0000:00:00.0: failed to install key for vdev 0 peer 9c:d6:43:29:cc:18: -11 [921144.051948] wlan0: failed to remove key (2, ff:ff:ff:ff:ff:ff) from hardware (-11) [921147.052846] ath10k_pci 0000:00:00.0: failed to install key for vdev 0 peer 9c:d6:43:29:cc:18: -11 [921147.061984] wlan0: failed to set key (2, ff:ff:ff:ff:ff:ff) to hardware (-11) [921150.082845] ath10k_pci 0000:00:00.0: failed to install key for vdev 1 peer 9e:d6:43:29:cc:18: -11 [921150.092017] wlan2: failed to remove key (2, ff:ff:ff:ff:ff:ff) from hardware (-11) [921153.092796] ath10k_pci 0000:00:00.0: failed to install key for vdev 1 peer 9e:d6:43:29:cc:18: -11 [921153.101928] wlan2: failed to set key (2, ff:ff:ff:ff:ff:ff) to hardware (-11) [932082.286927] ath10k_pci 0000:00:00.0: failed to send pdev bss chan info request [932390.335807] ath10k_pci 0000:00:00.0: failed to send pdev bss chan info request [932607.674665] ath10k_pci 0000:00:00.0: failed to down vdev_id 1: -11 [932610.674753] ath10k_pci 0000:00:00.0: failed to stop WMI vdev 1: -11 [932610.682338] ath10k_pci 0000:00:00.0: failed to stop vdev 1: -11 [932613.684666] ath10k_pci 0000:00:00.0: failed to install key for vdev 1 peer 9e:d6:43:29:cc:18: -11 [932613.694955] wlan2: failed to remove key (1, ff:ff:ff:ff:ff:ff) from hardware (-11) [932616.694882] ath10k_pci 0000:00:00.0: failed to submit AP/IBSS self-peer removal on vdev 1: -11 [932619.694635] ath10k_pci 0000:00:00.0: failed to delete WMI vdev 1: -11 [932622.694758] ath10k_pci 0000:00:00.0: failed to remove AP self-peer on vdev 1: -145 [932622.704039] ath10k_pci 0000:00:00.0: found vif peer 9e:d6:43:29:cc:18 entry on vdev 1 after it was supposedly removed [932622.715667] ath10k_pci 0000:00:00.0: removing stale peer 9e:d6:43:29:cc:18 from vdev_id 1 [932622.724455] br-guest: port 3(wlan2) entered disabled state [932622.796620] device wlan2 left promiscuous mode [932622.802446] br-guest: port 3(wlan2) entered disabled state [932622.856835] br-guest: port 2(wlan3) entered disabled state [932622.885493] device wlan3 left promiscuous mode [932622.891306] br-guest: port 2(wlan3) entered disabled state [932622.935955] device wlan0 left promiscuous mode [932622.940833] br-lan: port 2(wlan0) entered disabled state [932622.976382] device wlan1 left promiscuous mode [932622.981149] br-lan: port 3(wlan1) entered disabled state [932626.004584] ath10k_pci 0000:00:00.0: failed to down vdev_id 0: -11 [932629.004572] ath10k_pci 0000:00:00.0: failed to stop WMI vdev 0: -11 [932629.011031] ath10k_pci 0000:00:00.0: failed to stop vdev 0: -11 [932634.014537] ath10k_pci 0000:00:00.0: failed to flush transmit queue (skip 0 ar-state 1): 0 [932637.034556] ath10k_pci 0000:00:00.0: failed to install key for vdev 0 peer 9c:d6:43:29:cc:18: -11 [932637.043707] wlan0: failed to remove key (1, ff:ff:ff:ff:ff:ff) from hardware (-11) [932640.044532] ath10k_pci 0000:00:00.0: failed to submit AP/IBSS self-peer removal on vdev 0: -11 [932643.044507] ath10k_pci 0000:00:00.0: failed to delete WMI vdev 0: -11 [932646.044501] ath10k_pci 0000:00:00.0: failed to remove AP self-peer on vdev 0: -145 [932646.052277] ath10k_pci 0000:00:00.0: found vif peer 9c:d6:43:29:cc:18 entry on vdev 0 after it was supposedly removed [932646.063205] ath10k_pci 0000:00:00.0: removing stale peer 9c:d6:43:29:cc:18 from vdev_id 0 [932652.064468] ath10k_pci 0000:00:00.0: could not suspend target (-11) [932653.410146] IPv6: ADDRCONF(NETDEV_UP): wlan1: link is not ready [932654.710106] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready [932654.716877] device wlan1 entered promiscuous mode [932654.755523] device wlan0 entered promiscuous mode [932655.397455] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready [932655.404141] br-lan: port 3(wlan0) entered forwarding state [932655.409904] br-lan: port 3(wlan0) entered forwarding state [932655.426138] device wlan2 entered promiscuous mode [932655.445231] IPv6: ADDRCONF(NETDEV_UP): wlan2: link is not ready [932655.451454] br-guest: port 2(wlan2) entered forwarding state [932655.457370] br-guest: port 2(wlan2) entered forwarding state [932655.714483] br-guest: port 2(wlan2) entered disabled state [932655.842761] IPv6: ADDRCONF(NETDEV_CHANGE): wlan2: link becomes ready [932655.849460] br-guest: port 2(wlan2) entered forwarding state [932655.855380] br-guest: port 2(wlan2) entered forwarding state [932656.428869] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready [932656.435563] br-lan: port 2(wlan1) entered forwarding state [932656.441274] br-lan: port 2(wlan1) entered forwarding state [932656.466101] device wlan3 entered promiscuous mode [932656.477745] IPv6: ADDRCONF(NETDEV_UP): wlan3: link is not ready [932656.483865] br-guest: port 3(wlan3) entered forwarding state [932656.489791] br-guest: port 3(wlan3) entered forwarding state [932656.714486] br-guest: port 3(wlan3) entered disabled state [932656.869320] IPv6: ADDRCONF(NETDEV_CHANGE): wlan3: link becomes ready [932656.876022] br-guest: port 3(wlan3) entered forwarding state [932656.881921] br-guest: port 3(wlan3) entered forwarding state [932657.404425] br-lan: port 3(wlan0) entered forwarding state [932657.854498] br-guest: port 2(wlan2) entered forwarding state [932658.434436] br-lan: port 2(wlan1) entered forwarding state [932658.874432] br-guest: port 3(wlan3) entered forwarding state [932680.425787] br-guest: port 3(wlan3) entered disabled state [932680.442596] device wlan3 left promiscuous mode [932680.447279] br-guest: port 3(wlan3) entered disabled state [932680.467863] br-guest: port 2(wlan2) entered disabled state [932680.485729] device wlan2 left promiscuous mode [932680.490436] br-guest: port 2(wlan2) entered disabled state [932680.505002] device wlan1 left promiscuous mode [932680.509700] br-lan: port 2(wlan1) entered disabled state [932680.523775] device wlan0 left promiscuous mode [932680.528530] br-lan: port 3(wlan0) entered disabled state [932683.130864] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready [932683.169412] IPv6: ADDRCONF(NETDEV_UP): wlan1: link is not ready [932683.183967] device wlan0 entered promiscuous mode [932683.197266] device wlan1 entered promiscuous mode [932683.813544] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready [932683.820249] br-lan: port 2(wlan0) entered forwarding state [932683.826004] br-lan: port 2(wlan0) entered forwarding state [932683.841229] device wlan2 entered promiscuous mode [932683.857367] IPv6: ADDRCONF(NETDEV_UP): wlan2: link is not ready [932683.863484] br-guest: port 2(wlan2) entered forwarding state [932683.869386] br-guest: port 2(wlan2) entered forwarding state [932684.174365] br-guest: port 2(wlan2) entered disabled state [932684.265577] IPv6: ADDRCONF(NETDEV_CHANGE): wlan2: link becomes ready [932684.272265] br-guest: port 2(wlan2) entered forwarding state [932684.278193] br-guest: port 2(wlan2) entered forwarding state [932684.868399] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready [932684.875079] br-lan: port 3(wlan1) entered forwarding state [932684.880820] br-lan: port 3(wlan1) entered forwarding state [932684.905988] device wlan3 entered promiscuous mode [932684.912111] IPv6: ADDRCONF(NETDEV_UP): wlan3: link is not ready [932684.918275] br-guest: port 3(wlan3) entered forwarding state [932684.924142] br-guest: port 3(wlan3) entered forwarding state [932685.174351] br-guest: port 3(wlan3) entered disabled state [932685.305978] IPv6: ADDRCONF(NETDEV_CHANGE): wlan3: link becomes ready [932685.312646] br-guest: port 3(wlan3) entered forwarding state [932685.318582] br-guest: port 3(wlan3) entered forwarding state [932685.824311] br-lan: port 2(wlan0) entered forwarding state [932686.274310] br-guest: port 2(wlan2) entered forwarding state [932686.874321] br-lan: port 3(wlan1) entered forwarding state [932687.314318] br-guest: port 3(wlan3) entered forwarding state
After running wifi, I am able to connect again.
MAC addresses found in logs:
9c:d6:43:29:cc:18: DAP-2695, wlan0
9e:d6:43:29:cc:18: DAP-2695, wlan2
c0:ee:fb:4a:18:b7: OnePlus 2, Android 6.0.1
b0:df:3a:12:79:80: Samsung SM-T520, Android 5.1.1