I had originally thought to make this a wiki entry, but this and
other ideas are probably worth discussing. It may be of use to the
NBEMS groups. Note these methods generally are not worth it if the
TX payload size is (say) <500 bytes/characters.
I have been experimenting with sending larger files over HF (like
150kbytes max) and thought I may share some of the pre-processing
ideas I have worked though. I have been more or less using
flamp/fldigi with PSK250RC7 (and slower) as well as HamDRM EasyDRF
E/B/A 4/16/256 and recording/playback via a remote SDR. These
tests ranged from 40-15m on 300-4000km paths. I never seem to be
able to arrange ARQ and BSR sessions with other amateurs!
There are two basic kinds of transfers, lossy and not. Lossy in a
sense also includes the picture modes of Thor, IFKP, FSQ and MFSK
etc. I could argue too that a jpg file is also lossy, but for this
exercise I am talking about whatever file data I send is
faithfully received at the RX end. It is worth pointing out that
DSSTV programs like EasyPAL and QSSTV by default actually
re-encode pictures to a lesser quality (as JP2's), so the result
is not as good as the original.
So onto some real world uses.
- JPG's are an obvious for a scene/event photo, but terrible for a
scanned pieces of paper. They generally tend to be a bit large for
HF so any method to reduce size without drastically reducing their
usefulness is worth a thought. I am sure the quality and
resolution side (and cropping) is well understood, so in addition;
?? ?- Dropping from 24bit colour to 8bit monochrome reduces by
about 15-25%
?? ?- Changing DCT sub-sampling from 4:4:4 to 4:2:0 reduces by
about 15%
?? ?- The EXIF tags and image thumbnail can add 20-25kbytes to the
file, so worthwhile removing unless needed.
- PNG as a drop-in for 24bit/colour JPG's (conversion) doesn't
help at all as it is lossless. As a 1 (b/w) or 16 colour though
they are worth trying. It is worthwhile ensuring that the image to
process contains solid areas of 1 colour though. Dithering makes
them huge. Think cartoons or limited colour logos.
- G4 TIF is about the best compressed bitmap B&W standard
there is. This was the original format used in the Windows
Faxviewer, so seems well supported.
- Using a cell phone to take a photo of a black/written/drawn page
of paper works well. There probably isn't any need to go larger
than the resolution of a "fine mode fax". A 1500x2000 JPG half
page of text that has been threshold detected and converted to PNG
is about 39kBytes, but as a G4 tiff that comes down to maybe
26kBytes. A simple map drawn on paper with a thick pen will be
much smaller.
- Vector graphic standards romp all over bitmap for filesize.
These include PDF, EPS and SVG. Be wary though about just
converting an existing bitmap. By default postscript and PDF will
save that as a bitmap area and will therefore be huge. As a guide,
extracting just the PA-B circuit diagram page from my Icom 9100
PDF manual is about 123kBytes.
- There are tools that convert raster to vector, Potrace comes to
mind, but there are others. You might photograph a hand drawn
(with ruler) map or diagram, convert it to B&W (BMP, PBM etc)
then use potrace to create SVG, PDF, EPS etc. The straighter the
lines on the paper are the smaller the filesize will be.
- Simplifying GPX tracks. You may like to walk around with your
GPS equipped phone and log your track. Simplifying is worthwhile
as it removes points in the middle of straight lines. It is also
possible to use the standard phone camera and snap images with
location enabled. These images can be used to create a GPS track
and is also possible to overlay the images on a map. Google Earth
allows this as does Viking. In an EMS situation, walking around
photographing geolocated damage might be helpful, but only
relevant to radio xfers if the cell phone system was down!
Software tools - Linux & Windows (EasyDRF runs under wine).
Note the code examples are in Linux format.
- Personally I use Gimp (GUI) for many bitmap functions, for one
offs.
- Viking (GUI) for displaying geotagged images and some GPX
processing.
- GPSbabel (GUI/CLI) for conversion/simplifying
tracks/routes/waypoints. It will even create POIs for Garmin GPS
units.
- Exiftool (CLI) for manipulating image (and movie) EFIX tags
- ImageMagick (CLI) for manipulating images, en-mass if need be.
Main program subnames are convert and mogrify.
- EasyDRF (GUI) HamDRM based DSSTV like file/text transfers. Has
better performance than QSSTV/EasyPAL. Runs under Wine.
Operationally similar to flamp, ie full/partial resends, broadcast
FEC mode, but doesn't handle a modem change mid transfer.
- Fldigi, Flamp, Flmsg, Flarq, Of course!
Code examples, CLI
Imagemagick - In most cases "mogrify" can be substituted for
"convert" to overwrite the input file with the same name. Mogrify
also works well on directories of files whereas convert will need
a scripting loop.
Convert/overwrite a JPG image sub-sampling
This is a code segment I used to convert a through windscreen
webcam image for queued/scheduled flamp transmission. (which is
another project entirely) A good example of Imagemagick
possibilities.
?convert input.jpg? \
?? ?-gravity Center -crop 960x360+0+0 +repage \
?? ?-resize 480x180 \
?? ?-auto-level -channel RGB -auto-gamma \
?? ?-strip -interlace Plane -sampling-factor 4:2:0 \
?? ?-define jpeg:dct-method=float \
?? ?-gravity NorthWest -fill red -font Times-Roman -pointsize 16
-annotate +1+1 "See EXIF" \
?? ?-quality 65 \
?? ?/home/user/.nbems/FLAMP/tx/output.jpg
Convert/overwrite a JPG image to 8 bit monochrome
?convert input.jpg -set colorspace Gray -separate -average
output.jpg
Convert a JPG image to G4 TIF, including 50% resize
?convert input.jpg -auto-level -auto-gamma -threshold 50% -resize
50% -compress Group4 output.tif
Play with the threshold manually if needed
Convert a JPG to B&W bitmap then create a SVG file.
?convert input.jpg -auto-level -auto-gamma -threshold 50%
interim.pbm
?potrace interim.pbm -b svg -o output.svg
Obviously possible to pipe the first to second command directly.
Remove EXIF tags and thumbnail/preview from JPG's (and any other
image type that would have them)
?exiftool -ThumbnailImage= input.jpg (Remove just the thunbnail)
?exiftool -Preview:All= input.jpg (Remove just the preview
thunbnail)
?exiftool -ifd1:all= -ext jpg input.jpg (remove thumbnail and
directly associated EXIF tags)
?exiftool -all= input.jpg (remove all EXIF tags)
exiftool can work on wildcard filenames and directories but by
default will save the original as input.jpg_original. This can be
changed by adding the -overwrite_original switch.
Simplify GPS tracks - for gpx and nmea input
?gpsbabel -i gpx -f input.gpx -x simplify,crosstrack,error=0.002k
-x position,distance=5m -o gpx -F output.gpx
?gpsbabel -i nmea -f input.nmea -x
simplify,crosstrack,error=0.002k -x position,distance=5m -o gpx -F
output.gpx
The output includes date/time position speed, but also some
satellite info and vdop/hdop. For on the ground re-usage it may be
better to reduce the filesize by converting to a route rather than
leave as a track. There are many options in gpsbabel that one can
play with! If you don't feel like checking, the simple filter;
?cat output.gpx |grep -vE 'dop|speed|sat' >output2.gpx
will remove the extraneous data approximately halving the file
size.
The output is also essentially text so it will compress quite
happily with gzip/zip etc, but it may be better to just use the on
the fly (text) compressing of flamp/flmsg/EasyDRF. As a general
guide I took a drive day 5 fix/sec gpx file of 50 miles over 70
minutes (7.3MBytes), applied the filter above down to 100kBytes,
then compressed down to 9.3kBytes
Create GPS GPX from directory of geoencoded images
Firstly create a format file as below
()
?exiftool -fileOrder gpsdatetime -p gpx.fmt <input-dir> >
output.gpx
#------------------------------------------------------------------------------
# File:???????? gpx.fmt
#
# Description:? Example ExifTool print format file to generate a
GPX track log
#
# Usage:??????? exiftool -p gpx.fmt -ee3 FILE [...] > out.gpx
#
# Requires:???? ExifTool version 10.49 or later
#
# Revisions:??? 2010/02/05 - P. Harvey created
#?????????????? 2018/01/04 - PH Added IF to be sure position
exists
#?????????????? 2018/01/06 - PH Use DateFmt function instead of -d
option
#?????????????? 2019/10/24 - PH Preserve sub-seconds in
GPSDateTime value
#
# Notes:???? 1) Input file(s) must contain GPSLatitude and
GPSLongitude.
#??????????? 2) The -ee3 option is to extract the full track from
video files.
#??????????? 3) The -fileOrder option may be used to control the
order of the
#?????????????? generated track points when processing multiple
files.
#------------------------------------------------------------------------------
#[HEAD]<?xml version="1.0" encoding="utf-8"?>
#[HEAD]<gpx version="1.0"
#[HEAD] creator="ExifTool $ExifToolVersion"
#[HEAD] xmlns:xsi=
#[HEAD] xmlns=
#[HEAD] xsi:schemaLocation=>
#[HEAD]<trk>
#[HEAD]<number>1</number>
#[HEAD]<trkseg>
#[IF]? $gpslatitude $gpslongitude
#[BODY]<trkpt lat="$gpslatitude#" lon="$gpslongitude#">
#[BODY]? <ele>$gpsaltitude#</ele>
#[BODY]? <time>${gpsdatetime#;my
($ss)=/\.\d+/g;DateFmt("%Y-%m-%dT%H:%M:%SZ");s/Z/${ss}Z/ if
$ss}</time>
#[BODY]</trkpt>
#[TAIL]</trkseg>
#[TAIL]</trk>
#[TAIL]</gpx>
And that's it! I'm impressed if you read right through!
Cheers Bob VK2YQA
