Towards Freedom

Information, Inspiration, Imagination
truly a site for soaring Is


Shell programming notes.

TAGS: WHERNTO: erudite  techniq 

image of Shell

Tools of various sorts in zsh shell.

concat csv

  ls -1 *.csv | while read filnam ; do cat "$filnam" >> output.csv.file; done
  ls -1 *.csv | sort | while read filnam ; do cat "$filnam" >> output.csv.file; done
  ls -1t *.csv | while read filnam ; do cat "$filnam" >> output.csv.file; done


replace softlinks with file

    for f in $(find -type l);do
        cp --remove-destination $(readlink $f) $f;

the idea is good, but remember it will find all softlinks and you may not want to do that (as we found replacing 00.jpg).

dir output to array

shopt -s nullglob
shopt -u nullglob # Turn off nullglob to make sure it doesn't interfere with anything later
echo "${array[@]}"  # Note double-quotes to avoid extra parsing of funny characters in filenames


split video files

ffmpeg -ss 00:03:04.560 -i input.mp4 -c copy -t 00:03:39.299 output.mp4

to split video files which sometimes even losscut cannot but ffmpeg can


copying files

find MMMM/local/.ocs.agio.organ/cur/ -maxdepth 1 -type f | xargs cp -t mail/cur/
find . -maxdepth 1 -type f |head -1000| xargs cp -t "$destdir"

list by size

find . -size +1M (greater than 1M)
find . -size -1M (less than 1M)
find . -size +1M -exec ls -lh {} +

modifying files

find . -type d -name "*UNUSED" |
while read f;
    mv $f "${f%/*}/unused";

using for with find is not a good idea.

rename renumber files

find . -type f |
    while read f;
        cp $f ../NEWDIR/${(l(5)(0))k}.pdf

can be done as per code below, but zmv is preferable

for z in $(find /home/pradocs -name; do 
echo `mv $z ${z%/*}/`; 


lines not beginning with #

grep "^[^#;]" smb.conf

In other words, it reports lines that start with any character other than and ;. It's not the same as reporting the lines that don't start with and ; (for which you'd use grep -v '^[#;]') in that it also excludes empty lines

heroic commands in linux

exit terminal but leave all processes running

disown -a && exit

intercept stdout and log to file

cat | tee -a log.txt | cat > /dev/null

keep command out of history

start any command with a leading space and it will not go into history

put something in the background** ramdisk for fast rw

mkdir /mnt/ram
mount -t tmpfs tmpfs /mnt/ram -o size=8192M
dd if=/dev/zero of=test.iso bs=1M count=8000

on an ssd the above worked at 2.2G/s, but with ramdisk it was 3.9G/s!

recursive directory creation

mkdir -p folder/{sub1,sub2}/{sub1,sub2,sub3}
mkdir -p folder/{1..100}/{1..10}

redo last command via sudo

sudo !!

particularly helpful if you don't want to retype a long command.


tunneling to connect to cloud through a port of choice:
ssh -L localport:host:port-relative-to-machine login-credentials

we did this at one point to connect to lentil via wifi somehow.

use an editor to alter a long messed up command

push enter on the messed up command and then type fc to work with the command in default editor

execute the command then use bg

largest files and folders identification

du -h | sort -hr | head -n 10
du -ah | sort -hr | head -n 10

pad with zero

seq -w 1 11

or for more control

  for i in {2..11};
      echo $u1${(l:2::0:)i}$u2$i

and can get something like this

redo softlinks

find ~/bin -type l -ls | tr -s " " "\t" | cut -f11,13

couldn't do tr with sed for some reason
though the ideas using realpath will work, it doesn't do what we are trying to accomplish which is to replace 'ba' with 'sh' in the softlink

perl onliners

add a line after match

perl -pi -e '/paper.ily/ and $_.="\\include \"../../zztmpls/functions.scm\"\n"'



rsync -av --delete --exclude={'dir1','dir2','file1.txt'} ~/ocs ~/mnts/sd64/


  rename -v pradagio pradesigner */*/* #like grep -r


backref peculiarities

schnell% echo $v
[Integrating Values and Ethics into Wildlife Policy and Management Lessons from North America](

schnell% echo $v | sed -r -n 's/\[(.+)\]/\1/p'
Integrating Values and Ethics into Wildlife Policy and Management Lessons from North America(

schnell% echo $v | sed -rn 's/\[(.+)\]\((.*)\)/\2\1/p' Values and Ethics into Wildlife Policy and Management Lessons from North America

why does the entire $v get printed in the second item?

getting pics from html file to proper directories

from html source:

<p class="hdr">Iwrters</p>
<p class="dsc">Icquaint yourself with those who contributed to this site over the years!</p>
<div style="background-color:#ccccff; margin:1em 1em 5em 1em" >
<img class="floatrig" src="y-283.jpg" width="75" alt="Alexandra"/>
<p class="secitmtle">Alexandra</p>
<p class="secitmdsc">I love animals more than anything else!</p>
<p class="secitmurl"><a href="">see</a></p>
<p class="whern"></p>

<div style="background-color:#ddddff; margin:1em 1em 5em 1em" >
<img class="floatrig" src="y-288.jpg" width="75" alt="Angelkate"/>
<p class="secitmtle">Angelkate</p>
<p class="secitmdsc">Here we find divinity, for we are it.</p>
<p class="secitmurl"><a href="288.html">see</a></p>
<p class="whern"></p>

need to pull out the y-*.jpg and put it into the alt="NAME" directory.

the problem couldn't be solved with sed because non-greedy finds aren't possible, but cut worked nicely:

  sed -n "/^<img class/p" $tf/2.html | cut -d \" -f 4,8 | sed 's/"/:/' > TMP
  # need to change the " to : so it can be used as a delimiter for splitting.

  while read li; do
      cp $a[1] ~/hugos/towardsfreedom/content/iwrters/$a[2]/00.jpg;
  done < ~/hugos/towardsfreedom/TMP

md to org links conversion** multiple in-place replace:

  sed -i
      -e 's/\\skip1//g'
      -e 's/\"//g' 
      -e 's/  / /g' 
      -e 's/\\dynamicUp//g' 
      -e 's/\\set ignoreMelismata = ##t//g' *.ly

replace string with contents of file

sed -e "/Acss/r qq" -e "/Acss/d" index.php

puts contents of file qq right after Acss line and deletes the Acss line people have wondered how to just substitute the string and not the line

swap lines

this sort of does it but not too well there seems to be no way to delete line only the first occurance of a match

sed -e '/subsubtitle/ {h; n;p; x}' | 
sed -e '6d' | 

various extraction methods


reads from the standard input and writes to both standard output and one or more files at the same time (it helped to solve the issue we were having)

it can be used to write to multiple files as well as append:

command | tee file1.out file2.out file3.out
command | tee -a file.out

very useful when combined with sudo:

echo "newline" | sudo tee -a /etc/file.conf

or remotely:

for e in $@; do
    echo "$e REJECT" | ssh REMOTE sudo tee -a /etc/postfix/access


We really need to examine the docs to sort things out:


for n in Artmisia Cytheri Vaishob Chastity Retha Eliphas; do
    sed -i "s/$n/${(U)n}/g" AAlib.fountain

for itms out of a file

for l in "${(@f)"$(<filename)"}"
             fnumber=$(youtube-dl -F $l | tail -n 1 | cut -f 1 -d ' ')
             youtube-dl --format $fnumber $l


add these to .zshrc:

autoload -Uz promptinit
prompt bart (or whatever)

5 prompts:

$RPS1 rightside
$PS2 waiting for more input
$PS3 making choices
$PS4 debugging scripts

remove empty directories recursively

rmdir --verbose **/*(/^F)
rmdir --verbose **/*(/od)

[setting] up and using an array in for loop

this is a bit weird, but it works

#setup L1 directories
L1dirs=(Iwrter Issays Ithngs Ibrary Ictvty Inotes)
for d in "${L1dirs[@]}"; do
    echo $d

though just this will work too:

for d in Iwrter Issays Ithngs Ibrary Ictvty Inotes; do
    echo $d

shell options

zsh -l #
zsh -v #
zsh -x #xtrace—which can prove invaluable when debugging your scripts
zsh -f #start a clean instance of zsh using the default settings can set after shell started

text file into array


# urls should not be separated by blank lines or
# have quotes (it seems or we get missing scheme error)
# getting the urls into an array needs to be looked at

for l in "${(@f)"$(<./urls)"}"
             #echo "wget -O $fn $l"
             wget -O "$fn" "$l"

echo "all done"



lowercase the directories:

zmv -vw '**/(*)/' '${f:l}'

we had to run it 3 times though to get everything TODO

get rid of single quotes:

zmv -nQ '*.html' "\${f//'/}"

with Q we need to escape certain things like $

rename .php to .html:

zmv -Q '*.php' "\${f//\.php/.html}

config files

  • .zshenv - Should only contain user’s environment variables.
  • .zprofile - Can be used to execute commands just after logging in.
  • .zshrc - Should be used for the shell configuration and for executing commands.
  • .zlogin - Same purpose than .zprofile, but read just after .zshrc.
  • .zlogout - Can be used to execute commands when a shell exit.