The?
project helps with some stability fyi?

k

¿ªÔÆÌåÓý

Used the rpi-update to get onto 6.1.y kernel branch, which is 6.1.74-v8+.
The system was up for about 2h 24m.? I connected to the gateway twice and exchanged mail.

First I got a kernel message about rmsgw, shortly after I connected and sent mail.
Feb 03 19:14:00 rpi3-dev rmsgw[1191]: sendrf(): [; KK6VLO de WA6BGS-10 SK
Feb 03 19:14:00 rpi3-dev rmsgw[1191]: sendrf(): wrote 25 of 25 characters
Feb 03 19:14:00 rpi3-dev rmsgw[1191]: Logout KK6VLO??? tx:63 rx:9902 173.0s 57.6 Bytes/s (0)
Feb 03 19:14:00 rpi3-dev rmsgw[1191]: CMS Disconnected
Feb 03 19:14:00 rpi3-dev kernel: ------------[ cut here ]------------
Feb 03 19:14:00 rpi3-dev kernel: refcount_t: underflow; use-after-free.
Feb 03 19:14:00 rpi3-dev kernel: WARNING: CPU: 2 PID: 1191 at lib/refcount.c:28 refcount_warn_saturate+0xf8/0x148
Feb 03 19:14:00 rpi3-dev kernel: Modules linked in: mkiss ax25 cmac algif_hash aes_arm64 aes_generic algif_skcipher af_alg bnep vc4 brcmfmac snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper drm_kms_helper brcmutil binfmt_misc snd_soc_core hci_uart btbcm bluetooth cfg80211 snd_compress snd_pcm_dmaengine fb_sys_fops syscopyarea cdc_acm sysfillrect raspberrypi_hwmon sysimgblt bcm2835_codec(C) bcm2835_isp(C) bcm2835_v4l2(C) v4l2_mem2mem bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common snd_bcm2835(C) videodev i2c_bcm2835 snd_pcm ecdh_generic ecc rfkill libaes snd_timer mc snd vc_sm_cma(C) raspberrypi_gpiomem uio_pdrv_genirq uio fuse drm dm_mod drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
Feb 03 19:14:00 rpi3-dev kernel: CPU: 2 PID: 1191 Comm: rmsgw Tainted: G???????? C???????? 6.1.74-v8+ #1725
Feb 03 19:14:00 rpi3-dev kernel: Hardware name: Raspberry Pi 3 Model B Rev 1.2 (DT)
Feb 03 19:14:00 rpi3-dev kernel: pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Feb 03 19:14:00 rpi3-dev kernel: pc : refcount_warn_saturate+0xf8/0x148
Feb 03 19:14:00 rpi3-dev kernel: lr : refcount_warn_saturate+0xf8/0x148



Then it crashed a few minutes later on beacon...
Feb 03 19:20:01 rpi3-dev kernel: ------------[ cut here ]------------
Feb 03 19:20:01 rpi3-dev kernel: refcount_t: saturated; leaking memory.
Feb 03 19:20:01 rpi3-dev kernel: WARNING: CPU: 3 PID: 873 at lib/refcount.c:22 refcount_warn_saturate+0x78/0x148
Feb 03 19:20:01 rpi3-dev kernel: Modules linked in: mkiss ax25 cmac algif_hash aes_arm64 aes_generic algif_skcipher af_alg bnep vc4 brcmfmac snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper drm_kms_helper brcmutil binfmt_misc snd_soc_core hci_uart btbcm bluetooth cfg80211 snd_compress snd_pcm_dmaengine fb_sys_fops syscopyarea cdc_acm sysfillrect raspberrypi_hwmon sysimgblt bcm2835_codec(C) bcm2835_isp(C) bcm2835_v4l2(C) v4l2_mem2mem bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common snd_bcm2835(C) videodev i2c_bcm2835 snd_pcm ecdh_generic ecc rfkill libaes snd_timer mc snd vc_sm_cma(C) raspberrypi_gpiomem uio_pdrv_genirq uio fuse drm dm_mod drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
Feb 03 19:20:01 rpi3-dev kernel: CPU: 3 PID: 873 Comm: beacon Tainted: G??????? WC???????? 6.1.74-v8+ #1725
Feb 03 19:20:01 rpi3-dev kernel: Hardware name: Raspberry Pi 3 Model B Rev 1.2 (DT)
Feb 03 19:20:01 rpi3-dev kernel: pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Feb 03 19:20:01 rpi3-dev kernel: pc : refcount_warn_saturate+0x78/0x148
Feb 03 19:20:01 rpi3-dev kernel: lr : refcount_warn_saturate+0x78/0x148


And strangely it still had enough to run one more cron entry.
Feb 03 19:42:01 rpi3-dev CRON[1246]: pam_unix(cron:session): session opened for user rmsgw(uid=999) by (uid=0)
Feb 03 19:42:01 rpi3-dev CRON[1247]: (rmsgw) CMD (/usr/local/bin/rmsgw_aci > /dev/null 2>&1)
Feb 03 19:42:01 rpi3-dev rmsgw_aci[1248]: WA6BGS-10 - Linux RMS Gateway ACI 2.5.1 Feb? 3 2024 (DM12ms)
Feb 03 19:42:01 rpi3-dev rmsgw_aci[1248]: shm_init(): shmkey = 159412 (0x26eb4)
Feb 03 19:42:01 rpi3-dev rmsgw_aci[1248]: shm_init(): shmid = 0 (0x0)
Feb 03 19:42:01 rpi3-dev rmsgw_aci[1248]: Channel update interval = 7140
Feb 03 19:42:01 rpi3-dev rmsgw_aci[1248]: Using channel file '/etc/rmsgw/channels.xml
Feb 03 19:42:01 rpi3-dev rmsgw_aci[1248]: Channel file '/etc/rmsgw/channels.xml' age? 8109
Feb 03 19:42:01 rpi3-dev rmsgw_aci[1248]: Channel: WA6BGS-10 on radio (145070000 Hz, mode 0)
Feb 03 19:42:01 rpi3-dev rmsgw_aci[1248]: /usr/local/bin/rmschanstat ax25 radio WA6BGS-10 reports unavailable
Feb 03 19:42:01 rpi3-dev rmsgw_aci[1248]: Channel: WA6BGS-10 on radio is DOWN -- not updated
Feb 03 19:42:01 rpi3-dev rmsgw_aci[1248]: Channel Stats: 1 read, 1 active, 1 down, 0 updated, 0 errors
Feb 03 19:42:01 rpi3-dev rmsgw_aci[1248]: Version update interval = 86400
Feb 03 19:42:01 rpi3-dev rmsgw_aci[1248]: Version file '/usr/local/etc/rmsgw/stat/.version.WA6BGS-10' age 7578
Feb 03 19:42:01 rpi3-dev CRON[1246]: pam_unix(cron:session): session closed for user rmsgw


On the serial console, this appears.
[ 8670.831800] Call trace:
[ 8670.834290]? ax25cmp+0x14/0x60 [ax25]
[ 8670.838066]? ax25_bind+0x244/0x278 [ax25]
[ 8670.842194]? __sys_bind+0xe0/0x118
[ 8670.845668]? __arm64_sys_bind+0x28/0x38
[ 8670.849581]? invoke_syscall+0x4c/0x110
[ 8670.853409]? el0_svc_common.constprop.3+0x9c/0x100
[ 8670.858295]? do_el0_svc+0x34/0xd0
[ 8670.861679]? el0_svc+0x2c/0x78
[ 8670.864796]? el0t_64_sync_handler+0x98/0xc0
[ 8670.869059]? el0t_64_sync+0x18c/0x190
[ 8670.872800] Code: d503201f d2800002 d503233f 38626803 (38626824)
[ 8670.879001] ---[ end trace 0000000000000000 ]---


It's getting worse.
Time to find a release of Raspbian that will run 5.15, and while I work on that...? I'm going to load an x86_64 system and see if its any better.

I think I see now.? It makes a backup of /boot/firmware and installs the updated kernel*.img files
I'm used to seeing multiple kernels available in the GRUB menu, the RPi doesn't do this.

I did not know this existed.? Tons of warnings, upgrade first, read release notes later.? Thanks for the info!

I did this on Pi-3 and it took way less time than custom compile.? Now I have this version.
Linux rpi3-dev 6.1.74-v8+ #1725 SMP PREEMPT Mon Jan 22 13:35:32 GMT 2024 aarch64 GNU/Linux
Will load up rmsgw and see if we get more than 17 hours uptime.

And I'm a bit puzzled as to what that program did.? The boot commandline doesn't match the kernel.

??

The stable branch looks like 5.10.103, which predates the patch. ?If you are compiling from --branch=stable, I'd be careful here.

? Cheers
? Mike

? This link is into the stable branch; I'll agree that one looks unpatched. ?But if you are compiling the stable branch, you may not end up on 6.1.
You should look at what Git actually downloaded. ?When I get clone the default or with --branch rpi-6.1.y, I received patched versions.

? Cheers
? Mike

downloaded whatever version this brings? (which looks like it is rpi-6.1.y)
git clone --depth=1
It does have the ax25 patch.?

Downloaded this
git clone --depth=1 --branch=stable
And it does NOT have the patches.
Based on the links given in the KM4ACK-PI group, this seems like a reasonable path to gain stability. ?
I'm looking for a reliable answer on where raspbian gets their kernel source and what branch it is.?

If this fails, I'll be getting the 5.15 kernel

If I'm looking at the right thing, the ax25 patches don't appear here, and this stable version is what I think Raspbian is running.? Let me know if you learn otherwise, and where you found it.


Go down to line 62 and it doesn't match what I believe we both understand is the patched version.

What release did you pull from raspberryPi git?
I ran this
git clone --depth=1 --branch rpi-6.1.y https://github.com/raspberrypi/linux

On Sat, Feb 3, 2024 at 11:29 AM, Jon Bousselot KK6VLO wrote:

I'm going down this route, starting on the Pi-1

? Hi Jon,

Can you share why you think the patches aren't in the Rasp Pi OS kernel?

I pulled the raspberrypi.com git repo and diff'ed against the kernel.org source; they came out the same. ?I see the patches as applied:

I'm going down this route, starting on the Pi-1


git the rpi-6.1.y kernel
Verify the patches referenced in this thread exist in the code tree (they are)
compile
eat lunch and then dinner
install the kernel and see if it crashes 17 hours later.

For my ubuntu x86_64 desktop, I also did the apt source for the current source tree, and
apt source linux-image-unsigned-6.5.0-15-generic
in the downloaded tree, I notice the ax25 patches appear to be applied, so I wonder if desktop distros have these updates in place?
Add that to my list of things to try.


If I'm seeing this correctly, the stable kernel tree for rpi does NOT have the ax25 patches applied.?
The ax25 patches do appear in 5.15.y and the 6.1.y.? The only two I checked.

Ahh, well, there's been work done since kernel 5.15, I think it was kernel 6.2 I was trying, and that was for a different reason than crashing..

The bug description sounds more like a disconnect bug David was describing a few weeks ago. ?This doesn't look related to our kernel panic. ?Still I was curious ...

I used?apt-get source linux-image-$(uname -r) to pull the source for my kernel, which resulted in a linux-6.1.69 source tree. ?Down under net/ax25, I manually verified that 3 of the patches had been applied and a 4th applied at include/net/ax25.h . ?Seems like this has been patched in the source version for my system.

The only thing I couldn't figure out is why apt loads 6.1.69 source, when dmesg shows the running kernel is 6.1.63 . ?I guess the distro maintainers have moved on from .63, but I haven't updated yet. ?I'm a bit reluctant to update since the bug is currently reproducible. ?Instead, I pulled the 6.1.63 source from kernel.org and directly verified the ax.25 patch, so I know these fixes are in my running kernel. ?The patch was made for 5.15 a kernel and there are minor code differences with the 6.1.x kernels. ?However, it's very apparent the patch was applied to the running kernel version.

? Cheers
? Mike


??

there was an issue with axip/axudp links going dead and I found that kernel 6.2 seemed to resolve this issue, but I'm not sure if any of the recent patches address all problems, or just a few hi...

On 2024-02-03 1:53 a.m., Jon Bousselot KK6VLO wrote:

JJ, how far back must we go to get a Raspbian release where this isn't broken?
First to get an install of ax25 that doesn't crash while I work on doing a custom kernel on Bookworm.

The kernel.org links seem to indicate this is a problem for ax25 on all linux platforms, not just the Pi.? Is that correct?
-Jon

Correct.

¿ªÔÆÌåÓý

Ahh, well, there's been work done since kernel 5.15, I think it was kernel 6.2 I was trying, and that was for a different reason than crashing..there was an issue with axip/axudp links going dead and I found that kernel 6.2 seemed to resolve this issue, but I'm not sure if any of the recent patches address all problems, or just a few hi...

JJ, how far back must we go to get a Raspbian release where this isn't broken?
First to get an install of ax25 that doesn't crash while I work on doing a custom kernel on Bookworm.

The kernel.org links seem to indicate this is a problem for ax25 on all linux platforms, not just the Pi.? Is that correct?
-Jon

Correct.

JJ, how far back must we go to get a Raspbian release where this isn't broken?
First to get an install of ax25 that doesn't crash while I work on doing a custom kernel on Bookworm.

The kernel.org links seem to indicate this is a problem for ax25 on all linux platforms, not just the Pi.? Is that correct?
-Jon

¿ªÔÆÌåÓý

Also, there was a grant for working on the bugs in ax25 linux, quite awhile ago:

"

Grant: Fixing the Linux kernel AX.25
Date: December 2021
Amount: €179,690
Changes to the Linux kernel over the years have improved and modernized the kernel, but have also made existing AX.25 implementations incompatible and turned preexisting issues into bugs. This can make systems unpredictable or even unusable. Linux kernel development is complex, requiring deep specialized knowledge, and bugs are hard to trace. This may be one of the reasons, why the Linux kernel AX.25 stack is currently in such a bad state.

This ARDC grant funds will allow the Deutscher Amateur Radio Club to hire software developers who can create a stable Linux AX.25 implementation and prevent Linux distributions from dropping pre-compiled AX.25 support. The fixed and functional Kernel-AX.25 stack will improve global amateur radio infrastructure. Professional kernel development can bring Linux AX.25 back to life."

Related?

/g/KM4ACK-Pi/topic/howto_patch_the_kernel_for/95904470?p=,,,20,0,0,0::recentpostdate/sticky,,,20,0,0,95904470,previd%3D0,nextid%3D1691495389076951633&previd=0&nextid=1691495389076951633


Cheers, de John ve1jot

PS: I have two ports going usually..HF on network105 at 300b, and vhf 1200b port ve1jot-7 going to the provinces node stack..uptime seems infinite until I updated to latest bullseye, it pooched my entire system when I had a mem card failure.., and now I have to rebuild ugh!

I'm following this recipe to build my system.? I must be doing something consistently wrong across all builds, or I'm experiencing the same problem with my config.
I've ruled out the power supply issues with a 5v supply that is extremely capable of doing the job.? The babysitting pi never suffers a hiccup since it's not running any ax25 code.

Bullseye/Bookworm/32/64 from raspberrypi download., using the Lite version, no desktop.
Image the SD card, setup a first user, enable SSH, boot, go through the basic steps, setup timezone and en_us locale.
Patch until there are no more patches to apply.? This next list varies by one package name on Bullseye.? python-pip-whl instead of python3-pip-whl.

apt install -y rsync build-essential autoconf dh-autoreconf \
automake libtool git libasound2-dev libncurses5 \
libncurses5-dev libncursesw5-dev libudev-dev \
bc mg jed whois chrony ax25-apps \
ax25-tools git libxml2 libxml2-dev xutils-dev build-essential \
libax25-dev libx11-dev zlib1g-dev libncurses5-dev autoconf \
autogen libtool cmake libgps-dev screen lm-sensors \
python3-pip python3-pip-whl python3-requests

# create user/group that will run the ACI and RMSGW process
groupadd rmsgw
useradd -r -d /nonexistent -s /usr/sbin/nologin -g rmsgw rmsgw
git clone https://github.com/nwdigitalradio/rmsgw
chown -R rmsgw:rmsgw rmsgw
cd rmsgw
./autogen.sh
./configure
make -j`nproc --all`
make install
ln -s /usr/local/etc/rmsgw /etc
chown -R rmsgw:rmsgw /usr/local/etc/rmsgw
chown -R rmsgw:rmsgw /usr/local/bin/rms*

Populate the XML and CONF files with my data,? banner, channels.xml, gateway.conf, sysop.xml
Populate the ax25 files, axports, ax25d.conf with the minimum entries (removed all the default stuff)

Start a screen session so I can detach, and run this as root
/usr/sbin/kissattach /dev/ttyACM0 radio???? <<-sometimes give it a 44 address or leave empty->>
/usr/sbin/ax25d
/usr/sbin/mheardd??? <<-sometimes I forget to launch this->>
/usr/sbin/beacon -c WA6BGS -d "beacon" -t 35 radio "RMS Gate = WA6BGS-10"


Add crontab for rmsgw
{
echo "# m h? dom mon dow?? command"
echo "14,42 * * * * /usr/local/bin/rmsgw_aci > /dev/null 2>&1"
} | crontab -u rmsgw -



Everything works really great till about a day passes.
How does this compare to your non-crashy config?

4.3MB at 1200 baud...? whoa.

Hi Jon,

The Pi-1 crashed on its own today without the beacon app, after 1d 3h 47m uptime.? Here is what it cried out when going down.
[100075.375841] CPU: 0 PID: 3001 Comm: netstat Tainted: G???????? C???????? 6.1.0-rpi7-rpi-v6 #1? Raspbian 1:6.1.63-1+rpt1

Thanks for sharing this; this is very interesting. ?It shows that the crash can move from one user space app to another. ?It also proves the theory that we can't work around this by disabling beacons or turning off Pat; the crash will just be triggered by something else.

I agree that netstat is going to walk the kernel AX.25 memory structures, even if it's not directly related to other ax25-tools. ?Regarding rmsgw_aci, if you know what time of day the Pi crashed, I bet it matches your rmsgw_aci schedule in cron.

We're collecting quite a few variables to test here.? This could take a while if crashes only happen every few days.
32 or 64 bit?
Bullseye or Bookworm?
How many ax25 packets until the system becomes unstable?
Beacon or no-beacon?

Here's my feed back. ?I'm all 64-bit and all Bookworm. ?I also moved beacon to the TNC (Direwolf), but, as I think this crash proves, beacon is just a red-herring.

So, on the packet question, I'm going to retract my earlier statement that a certain amount of traffic is required.

I started thinking about this last night and was disappointed that there wasn't a common way we could test out our builds. ?I did, however, remember something from the Direwolf user's guide. ?WB2OSZ used a set of test audio to tune the Direwolf algorithms, and that gave me an idea. ?I tracked down the ?and extracted an AIFF file. ?Then I built a loop to continuously play this test audio through the on-board Pi headphone jack. ?I connected the headphone back back to a USB audio adapter and configured Direwolf to decode the audio back into the AX.25 stack.

I've had this setup running for about 2 days now. ?No crash thus far, but I've looped about 70,000 AX.25 packets (4.3MByte), which is probably more data than my station would see in 6 months: