Synchronized script execution across multiple servers

Mr. Llama

I'm looking to run an automated task (a shell script) on multiple Ubuntu servers at the same time with as much timing accuracy as possible. Ideally, I'd like the time discrepancy between the start of script executions to be less than a second across all of the servers. The servers are all in different locations and most are in different time zones. However, the tasks I intend to run will be at regular intervals, so time zones shouldn't be an issue.

I'm currently considering using cron to schedule the recurring task on each server, but I'm worried that it may not be accurate enough on its own. To compensate, I was considering using network time protocol to ensure that all servers have clocks that stay in sync (plus or minus time zones).

So my question is two part:

  1. Will using cron and ntp be accurate enough to meet my needs?
  2. Is there a better way to go about this?
feitingen

It's difficult to answer accurately as there is different versions of cron with potentially varying accuracy.

NTP will definitively be a must for a solution like this, and use ntpd instead of ntpdate, as ntpd will constantly adjust the "length of a second" to the correct length of a second as opposed to ntpdate which just sets the time and allows it to drift again. Once a system has booted, it no longer uses the battery backed RTC clock, but will use CPU generated interrupts instead which are much more accurate, but might need to be adjusted slightly.

On to cron:

From the man page of cron:

cron then wakes up every minute, examining all stored crontabs, checking each command to see if it should be run in the current minute. When executing commands, any output is mailed to the owner of the crontab (or to the user named in the MAILTO environment variable in the crontab, if such exists). The children copies of cron run‐ ning these processes have their name coerced to uppercase, as will be seen in the syslog and ps output.

Unless cron wakes up every minute on the minute, you have no gurantee for secondly accuracy, but I'd recommend not to take this for granted and put it to the test. Synchronize the servers using NTP, schedule a cron job to touch files over a period of time and stat them.

After you've touched enough files you should be able to get an idea of what kind of accuracy your cron has.

If your application absolutely has to synchronously within the second, or if your cron is too inaccurate, it will be better to start the scripts a few seconds early and using your NTP synchronized clock, have the scripts wait for the right moment in time to proceed with your request.

Edit: A quick bash script to sleep until the specified minute:

#!/bin/bash
minute_to_start=$1
minute_now=`date +%M`
minutes_to_wait=`echo $minute_to_start - $minute_now -1 | bc -l`
now=`date +%S.%N`
to_sleep=`echo 60 - $now + \( $minutes_to_wait \* 60 \) | bc -l`
sleep $to_sleep
$2

You can run it in cron like:

29 7 * * * /path/to/above/script.sh 30 'the real script'

And the job will start at 07:30:00

In my testing it drifted within 10-20 milliseconds, and it doesn't work if you want to start at X:00:00.00 (on the hour), so not something I'd actually use in production.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

Shell script execution on multiple servers

MySQL database across multiple servers

Bash Script to Transfer File across Two Servers

Generate unique ID across multiple independent servers

How to manage websockets across multiple servers / workers

How to scale a trie across multiple servers

Java: synchronizing threads across multiple servers

Scaling Socket.IO across multiple servers

Synchronization xml file across multiple servers(machine)

Shared Varnish cache across multiple servers

Using sed command in for loop across multiple servers

Play and task execution with multiple groups and servers with ansible

Rsync backup script for multiple servers

How data can be synchronized among multiple linux servers

how to run my selenium script on multiple servers at the same time so that i would saved one by one execution time?

D3.js Synchronized Zoom Across Multiple Graphs

Spring batch restrict single instance of job across multiple servers

Laravel Task Scheduler across multiple servers without overlap

Submitting "ps" across multiple servers, output going to one text file

Golang file and folder replication / mirroring across multiple servers

Using Quartz to Schedule Single Job Across Multiple Stateless App Servers

Proper way distribute several scientific experiments across multiple servers?

Get-ADUser for multiple users across different servers

Distributing a Neo4j Graph Across Multiple Servers

Mass deployment of package to multiple Linux servers across different continents

script to check if I can access multiple servers

Pinging to multiple servers via windows batch script

Monitoring multiple Linux Systems or Servers Script

Synchronizing webapps across servers