Skip to content

Investigate if daylight savings affects session stats transfer to galftp

Issue

The stats for 20211107 did not correctly transfer from the gsp0* servers to galftp. The time change is the most likely suspect.

Investigation

Logger bug?

What should happen:

  • login, links, search stats each go in a separate (single!) file for each day, e.g.,
bmb@gsp00:/app/galileo_search/stats$ ls -la *.stats.D20221105
-rw-r--r-- 1 galileo galileo  76031 Nov  6 04:00 links.stats.D20221105
-rw-r--r-- 1 galileo galileo 172437 Nov  6 04:00 login.stats.D20221105
-rw-r--r-- 1 galileo galileo  59067 Nov  6 04:00 search.stats.D20221105

What happened on the 6th (note, starting 20221106 00:00:00 before the time change)

  • the files are given these names (which makes zero sense):
bmb@gsp00:/app/galileo_search/stats$ ls -la *D20221106.D20221105
-rw-r--r-- 1 galileo galileo 135420 Nov  6 22:59 links.stats.D20221106.D20221105
-rw-r--r-- 1 galileo galileo 288806 Nov  6 22:59 login.stats.D20221106.D20221105
-rw-r--r-- 1 galileo galileo 110407 Nov  6 22:59 search.stats.D20221106.D20221105

Then--odder still--during the final hour of the day (20221106 23:00:00 - 23:59:59)

  • the logger created a new file for every line written (so each file contains one line)
bmb@gsp00:/app/galileo_search/stats$ ls -lat search.stats.D20221106.D20221105.*|head -3
-rw-r--r-- 1 galileo galileo 130 Nov  6 23:57 search.stats.D20221106.D20221105.74
-rw-r--r-- 1 galileo galileo 131 Nov  6 23:57 search.stats.D20221106.D20221105.73
-rw-r--r-- 1 galileo galileo 125 Nov  6 23:54 search.stats.D20221106.D20221105.72
bmb@gsp00:/app/galileo_search/stats$ ls -lat search.stats.D20221106.D20221105.*|tail -3
-rw-r--r-- 1 galileo galileo 132 Nov  6 23:01 search.stats.D20221106.D20221105.3
-rw-r--r-- 1 galileo galileo 131 Nov  6 23:01 search.stats.D20221106.D20221105.2
-rw-r--r-- 1 galileo galileo 249 Nov  6 23:01 search.stats.D20221106.D20221105.1
bmb@gsp00:/app/galileo_search/stats$ ls -lat links.stats.D20221106.D20221105.*|head -3
-rw-r--r-- 1 galileo galileo 151 Nov  6 23:59 links.stats.D20221106.D20221105.86
-rw-r--r-- 1 galileo galileo 145 Nov  6 23:57 links.stats.D20221106.D20221105.85
-rw-r--r-- 1 galileo galileo 145 Nov  6 23:56 links.stats.D20221106.D20221105.84
bmb@gsp00:/app/galileo_search/stats$ ls -lat links.stats.D20221106.D20221105.*|tail -3
-rw-r--r-- 1 galileo galileo 140 Nov  6 23:00 links.stats.D20221106.D20221105.3
-rw-r--r-- 1 galileo galileo 149 Nov  6 23:00 links.stats.D20221106.D20221105.2
-rw-r--r-- 1 galileo galileo 140 Nov  6 23:00 links.stats.D20221106.D20221105.1
bmb@gsp00:/app/galileo_search/stats$ ls -lat login.stats.D20221106.D20221105.*|head -3
-rw-r--r-- 1 galileo galileo 163 Nov  6 23:59 login.stats.D20221106.D20221105.100
-rw-r--r-- 1 galileo galileo 159 Nov  6 23:41 login.stats.D20221106.D20221105.99
-rw-r--r-- 1 galileo galileo 159 Nov  6 23:39 login.stats.D20221106.D20221105.98
bmb@gsp00:/app/galileo_search/stats$ ls -lat login.stats.D20221106.D20221105.*|tail -3
-rw-r--r-- 1 galileo galileo 161 Nov  6 23:00 login.stats.D20221106.D20221105.3
-rw-r--r-- 1 galileo galileo 162 Nov  6 23:00 login.stats.D20221106.D20221105.2
-rw-r--r-- 1 galileo galileo 160 Nov  6 23:00 login.stats.D20221106.D20221105.1

Finally, when our rake task ran at 4am on the 7th

  • it wrote a single line to these files (and sent those files to galftp):
bmb@gsp00:/app/galileo_search/stats$ ls -la *.stats.D20221106
-rw-r--r-- 1 galileo galileo 147 Nov  7 04:00 links.stats.D20221106
-rw-r--r-- 1 galileo galileo 169 Nov  7 04:00 login.stats.D20221106
-rw-r--r-- 1 galileo galileo  87 Nov  7 04:00 search.stats.D20221106

Restore stats

No automation here. Just laborious manual manipulation. Hopefully the problem will be fixed before next year.

cat /app/galileo_search/stats/search.stats.D20221106.D20221105 > search
ls -tr /app/galileo_search/stats/search.stats.D20221106.D20221105.*|xargs cat >> search
cat /app/galileo_search/stats/search.stats.D20221106 >> search
grep 20221106 search > search.2
vi search search.2 
[copy column headings from first file to second]

cat /app/galileo_search/stats/links.stats.D20221106.D20221105 > links
ls -tr /app/galileo_search/stats/links.stats.D20221106.D20221105.*|xargs cat >> links
cat /app/galileo_search/stats/links.stats.D20221106 >> links
grep 20221106 links > links.2
vi links links.2 
[copy column headings from first file to second]

cat /app/galileo_search/stats/login.stats.D20221106.D20221105 > login
ls -tr /app/galileo_search/stats/login.stats.D20221106.D20221105.*|xargs cat >> login
cat /app/galileo_search/stats/login.stats.D20221106 >> login
grep 20221106 login > login.2
vi login login.2 
[copy column headings from first file to second]
  • login in as galileo, i.e., sudo su - galileo
  • compare the above scratch files to a sample stats file for a sanity check
  • if it looks good copy it to the correct file
diff -u search.2 /app/galileo_search/stats/search.stats.D20221106.D20221105|less -N
cp search.2 /app/galileo_search/stats/search.stats.D20221106
[note we're comparing to a different file than we're copying to]

diff -u links.2 /app/galileo_search/stats/links.stats.D20221106.D20221105|less -N
cp links.2 /app/galileo_search/stats/links.stats.D20221106
[note we're comparing to a different file than we're copying to]

diff -u login.2 /app/galileo_search/stats/login.stats.D20221106.D20221105|less -N
cp login.2 /app/galileo_search/stats/login.stats.D20221106
[note we're comparing to a different file than we're copying to]
  • list the files as a final check
> ls -la /app/galileo_search/stats/*20221106
-rw-r--r-- 1 galileo galileo 145907 Nov  8 14:58 /app/galileo_search/stats/links.stats.D20221106
-rw-r--r-- 1 galileo galileo 311504 Nov  8 14:58 /app/galileo_search/stats/login.stats.D20221106
-rw-r--r-- 1 galileo galileo 112235 Nov  8 14:58 /app/galileo_search/stats/search.stats.D20221106
  • run rake task to push to galftp
> cd /app/galileo_search/
galileo@gsp07:/app/galileo_search$ RAILS_ENV=production bundle exec rake galileo:push_stats\['20221106'\]
Edited by Brad Baxter