Unable to output script results with column/table formatting

Answered - previously titled 'Cron job for shell script not running'

I recently downloaded Speedtest onto my Raspberry Pi, and wrote a script to output the results in csv format to a CSV file.

I'm trying to do this regularly via a cron job, but for some reason, it won't execute the shell script as intended.

Here's the script below. I've commented/cut out a lot to try and find the issue

#!/bin/bash

# Commented out if statement detects presence of data file and creates one if it doesn't exist. Was going to adjust later to include variables/input options if I wanted to used script on alternate systems, but commented out while working on main issue.

file='/home/User/Documents/speedtestdata.csv' 
# have tried this with and without quotes, does not seem to make a difference either way
#HEADERS='/usr/bin/speedtest-cli --csv-header'

SPEEDTEST='/usr/bin/speedtest-cli --csv'
# Used absolute path for the executable

#LOG=/home/User/scripts/testreclog.txt
#DATE=$( date )
# Was using the above to log steps of script running successfully with timestamp, commented out 

#if [ ! -f $file ]
#then
#       echo "Creating results file">>$LOG
#       touch $file
#       $HEADERS > $file
#fi

#echo "Running speedtest">>$LOG

$SPEEDTEST >> $file

#echo "Formatting results">>$LOG

#column -s, -t < $file
# this step was used to format the log file neatly

#echo "Time completed ",$DATE>>$LOG

And here's how the crontab currently looks

# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command

*/5 * * * * /bin/bash /home/User/scripts/testandrec.sh

# 2> /home/User/scripts/testrecerror.txt
# Was attempting to log errors to this file, nothing seen so commented out on a newline.

#* * * * * /home/User/scripts/testscript.sh test to verify cron works (it does)

I've added my scripts folder to the end of my path, but for some reason this only shows up when I'm using the Pi directly, when I ssh in I'm missing the scripts folder on the end. However, given that I've used absolute path for everything I'm not sure why this would be an issue.

First I tested whether a simple Cron job would work, so I created testscript.sh, which simply returned 'Test' and a timestamp to a specific file and used the same shebang, and used the absolute paths, and functioned as intended.

I have checked systemctl for Cron, restarted Cron with sudo service cron restart and made sure a new line is in place in the crontab.

I have tried with and without /bin/bash in the cron tab entry, it seemingly hasn't made a difference.

I tried cd /home/User/scripts && ./testandrec.sh but no luck.

I changed the run time to every 5 then every 10 minutes, which has not worked.

I have noticed that when I ran the script manually with column -s, -t < $file left in, when cating the results file it is formatted as intended. However, the next instance of when the cron job should run reverts this to CSV with a , as a delimitter, so clearly something is running.

To confuse matters further, I think the script may be firing once after restarting cron, and then not working when it should be running subsequently. When I leave the column line in, this appears to just revert the formatting, but if I comment it out it appears to run a speed test and append the results, but only once. However, I may be wrong on this and reproducing it

If I instead try 0 * * * * /usr/bin/speedtest-cli --csv >> /home/User/Documents/speedtestdata.csv && column -s, -t < /home/User/Documents/speedtestdata.csv, it appeared to perform/append speedtest but does not action the column command.

I would much rather neatly tie up the process in a shell script, however, rather than have the above which isn't very DRY code.

I've looked extensively, but none of the solutions I've found on this site or others have fixed the issue.

Any troubleshooting suggestions/help would be greatly appreciated.