Towards Freedom

Information, Inspiration, Imagination
truly a site for soaring Is


Shell programming notes.

TAGS: WHERNTO: erudite  techniq 

image of Shell

We use the z shell and program in zsh. All items in .zsh.d with .zshenv and .zshrc symlinked. This setup is easily migrated to other machines. The items below illustrate various shell commands some which are specific to zsh.

adding ~/bin to PATH

### PATH ### typeset -U path #converts to uppercase (to avoid accidental duplication as per bing) path+=~/bin

alternate expressions

  ls *glob*.(txt|h) #note use of ()
  # *glob*.txt and *glob*.h

block commenting

Though it is unlikely we will actually use this technique, it is handy to know.

  echo "this is a block comment" 
  : '
      what is this


All cap specific words in a file using perl|sed. Note that single quotes may work with sed, but not produce desired result with perl.

  # perl
  perl -pi -e "s/(Eliphas|Retha|Chastity)/\U$1/g" <filename>

  for n in  Chastity Retha Eliphas; do # or using loop
      perl -i -pe "s/$n/\U$n/g" <filename>

  # sed
  for n in Cytheri Vaishob Chastity Retha Eliphas; do
          sed -i "s/$n/${(U)n}/g" <filename>


What does zstyle :compinstall filename do?

It's for when you run compinstall again. It lets compinstall find where it has written out zstyle statements for you last time. This way, you can run compinstall again to update them. Read in source code.

concat csv

Here are some ways to concat text files.

  ls -1 *.csv | while read filnam ; do cat "$filnam" >> output.csv.file; done
  ls -1 *.csv | sort | while read filnam ; do cat "$filnam" >> output.csv.file; done
  ls -1t *.csv | while read filnam ; do cat "$filnam" >> output.csv.file; done

config files

  • .zshrc - Should be used for the shell configuration and for executing commands.
  • .zshenv - Should only contain user’s environment variables.
  • .zprofile - Can be used to execute commands just after logging in.
  • .zlogin - Can be used as .zprofile, but read just after .zshrc.
  • .zlogout - Can be used to execute commands when a shell exit.


replace softlinks with actual file

  cp --remove-destination $(readlink <filename>) <filename>

  for f in $(find -type l);do # or do all softlinks
      cp --remove-destination $(readlink $f) $f;

direct ls output to array


  # parsing ls is not a good idea usually so

  # if there are subdirectories
  shopt -s nullglob
  shopt -u nullglob # Turn off nullglob to make sure it doesn't interfere with anything later
  echo "${array[@]}"  # Note double-quotes to avoid extra parsing of funny characters in filenames

See How do I assign ls to array discussion.

docs for zmv, zcp, zln:

docs packit book scripting

this book contains a lot of useful stuff

zle (ch3)

Emacs users will find themselves at home with the Esc + X sequence. we can use where-is to find an alternative shortcut man zshzle has lots!

globbing (ch4)

the negations ^,~ and alternates (|),[] need work

tilde echo b*~*.o match starting with b, don't match anything with o extension echo a*~*9.* match starting with a, don't match anything with 9 before the . echo a*~*9?.* match starting with a, don't match anything with 9 one item before . echo */.sh~tmp/* exclude anything within tmp/

qualifiers cheatsheet *(…) TAB *(@) symlinks *(On) reverse sort by name *(.) files only *(@,/) symlinks or directories

TODO things to do

zcalc zargs man zshexpn

dot z files

From posting:

  • .zshenv is always sourced. It often contains exported variables that should be available to other programs. For example, $PATH, $EDITOR, and $PAGER are often set in .zshenv. Also, you can set $ZDOTDIR in .zshenv to specify an alternative location for the rest of your zsh configuration.
  • .zprofile is for login shells. It is basically the same as .zlogin except that it's sourced before .zshrc whereas .zlogin is sourced after .zshrc. According to the zsh documentation, ".zprofile is meant as an alternative to .zlogin for ksh fans; the two are not intended to be used together, although this could certainly be done if desired."
  • .zshrc is for interactive shells. You set options for the interactive shell there with the setopt and unsetopt commands. You can also load shell modules, set your history options, change your prompt, set up zle and completion, et cetera. You also set any variables that are only used in the interactive shell (e.g. $LS_COLORS).
  • .zlogin is for login shells. It is sourced on the start of a login shell but after .zshrc, if the shell is also interactive. This file is often used to start X using startx. Some systems start X on boot, so this file is not always very useful.
  • .zlogout is sometimes used to clear and reset the terminal. It is called when exiting, not when opening.

Also, the configuration files of random Github users gives possibilities.

expanding and sorting

the o,O sort doesn't work for ls unless we disable ls sort with -U

Expanding Depending on a Predicate these things may be quite powerful if we learn what to do with them e: *<cmd>


Expansions Depending On the Permissions might be useful ls *(f755) would list only directories (usually) though ls -d */ simpler however, there are many other features for this

Expansions Depending on Date or Size ls **/*(L0) list all empty files ls **/*(LG+1) list all files >1G ls **/*(mh-1) list all files modified within last hour

find fd

copying files

find MMMM/local/.ocs.agio.organ/cur/ -maxdepth 1 -type f | xargs cp -t mail/cur/
find . -maxdepth 1 -type f |head -1000| xargs cp -t "$destdir"

list by size

find . -size +1M (greater than 1M)
find . -size -1M (less than 1M)
find . -size +1M -exec ls -lh {} +

find only .git dirs

fd –exclude=".cache" –type d –hidden '.git$' find ocs/ -type d -name ".git"

ignore dir

fd –exclude dir find . \( -name ".cache" -prune \) -o -name ".git" -type d

modifying files

find . -type d -name "*UNUSED" |
while read f;
    mv $f "${f%/*}/unused";

using for with find is not a good idea for some reason

renumber files

best done with zmv, but using find is still interesting

find . -type f |
    while read f;
        cp $f ../NEWDIR/${(l(5)(0))k}.pdf

# can be done as per code below, but zmv is preferable

for z in $(find /home/pradocs -name; do 
echo `mv $z ${z%/*}/`; 

find exec

this is a strange syntax but worth learning. the following finds all .org in the fireworks/ and replaces localhost:1313 with find . -name "*.org" -exec perl -pi -e 's/localhost:1313/' {} \;

find . -name ".pdf" -exec wc {} \; find . -name ".pdf" -exec wc {} \+ find . -name "*.pdf" -print0 | xargs -0 wc (same as above) print0 and xargs -0 handles spaces in filename which is bad idea anyway

exec Really Means exec! "It uses Linux’s built-in exec to run the command, replacing the current process—your shell—with the command. So the command that is launched isn’t running in a shell at all." which is why exec ls wipes out the shell!

chaining (instead of piping?) example find . -name "*.page" -type f -exec bash -c "basename "{}" && words-only "{}"" \;


split video files

ffmpeg -ss 00:03:04.560 -i input.mp4 -c copy -t 00:03:39.299 output.mp4

to split video files which sometimes even losscut cannot but ffmpeg can

file into variable


for itms out of a file

for l in "${(@f)"$(<filename)"}"
             fnumber=$(youtube-dl -F $l | tail -n 1 | cut -f 1 -d ' ')
             youtube-dl --format $fnumber $l


for i in {1..9}; do echo $i; done for ((i=1; i<10; i++)); echo $i


lines not beginning with #

grep "^[^#;]" smb.conf

In other words, it reports lines that start with any character other than and ;. It's not the same as reporting the lines that don't start with and ; (for which you'd use grep -v '^[#;]') in that it also excludes empty lines

heroic commands in linux

exit terminal but leave all processes running

disown -a && exit

intercept stdout and log to file

cat | tee -a log.txt | cat > /dev/null

keep command out of history

start any command with a leading space and it will not go into history

put something in the background** ramdisk for fast rw

mkdir /mnt/ram
mount -t tmpfs tmpfs /mnt/ram -o size=8192M
dd if=/dev/zero of=test.iso bs=1M count=8000

on an ssd the above worked at 2.2G/s, but with ramdisk it was 3.9G/s!

recursive directory creation

mkdir -p folder/{sub1,sub2}/{sub1,sub2,sub3}
mkdir -p folder/{1..100}/{1..10}

redo last command via sudo

sudo !!

particularly helpful if you don't want to retype a long command.


tunneling to connect to cloud through a port of choice:
ssh -L localport:host:port-relative-to-machine login-credentials

we did this at one point to connect to lentil via wifi somehow.

use an editor to alter a long messed up command

push enter on the messed up command and then type fc to work with the command in default editor

execute the command then use bg

largest files and folders identification

du -h | sort -hr | head -n 10
du -ah | sort -hr | head -n 10



the inverse is giving by ^

  ls *M.mp4
  ls ^*M.mp4

list only directories

List only directories (post) ls -d – *() ls -ld – *()

new zsh script

cp ~/sh/zztmpls/ ~/

enter title that is descriptive
set VDATe: v1 - M-x org-time-stamp write PURPose
provide desc

block comment above

fill out help section "use" and "how"

add variables and coding

numeric sequence generation

touch a{01..99}.txt works touch a[01-99].txt doesn't work

x a{2..4}* works x a[2-4]* works

pad with zero

seq -w 1 11

or for more control

  for i in {2..11};
      echo $u1${(l:2::0:)i}$u2$i

and can get something like this

perl onliners

add a line after match

perl -pi -e '/paper.ily/ and $_.="\\include \"../../zztmpls/functions.scm\"\n"'


add these to .zshrc:

autoload -Uz promptinit
prompt bart (or whatever)

5 prompts:

$RPS1 rightside
$PS2 waiting for more input
$PS3 making choices
$PS4 debugging scripts

redo softlinks

find ~/bin -type l -ls | tr -s " " "\t" | cut -f11,13

couldn't do tr with sed for some reason
though the ideas using realpath will work, it doesn't do what we are trying to accomplish which is to replace 'ba' with 'sh' in the softlink

remove empty directories

rmdir --verbose **/*(/^F)
rmdir --verbose **/*(/od)

remove empty directories recursively

rmdir --verbose **/*(/^F)
rmdir --verbose **/*(/od)


  rename -v pradagio pradesigner */*/* #like grep -r



rsync -av --delete --exclude={'dir1','dir2','file1.txt'} ~/ocs ~/mnts/sd64/


Use M-h on a command

Setup properly in .zshrc unalias run-help 2>/dev/null autoload -U run-help


backref peculiarities

schnell% echo $v
[Integrating Values and Ethics into Wildlife Policy and Management Lessons from North America](

schnell% echo $v | sed -r -n 's/\[(.+)\]/\1/p'
Integrating Values and Ethics into Wildlife Policy and Management Lessons from North America(

schnell% echo $v | sed -rn 's/\[(.+)\]\((.*)\)/\2\1/p' Values and Ethics into Wildlife Policy and Management Lessons from North America

why does the entire $v get printed in the second item?

getting pics from html file to proper directories

from html source:

<p class="hdr">Iwrters</p>
<p class="dsc">Icquaint yourself with those who contributed to this site over the years!</p>
<div style="background-color:#ccccff; margin:1em 1em 5em 1em" >
<img class="floatrig" src="y-283.jpg" width="75" alt="Alexandra"/>
<p class="secitmtle">Alexandra</p>
<p class="secitmdsc">I love animals more than anything else!</p>
<p class="secitmurl"><a href="">see</a></p>
<p class="whern"></p>

<div style="background-color:#ddddff; margin:1em 1em 5em 1em" >
<img class="floatrig" src="y-288.jpg" width="75" alt="Angelkate"/>
<p class="secitmtle">Angelkate</p>
<p class="secitmdsc">Here we find divinity, for we are it.</p>
<p class="secitmurl"><a href="288.html">see</a></p>
<p class="whern"></p>

need to pull out the y-*.jpg and put it into the alt="NAME" directory.

the problem couldn't be solved with sed because non-greedy finds aren't possible, but cut worked nicely:

  sed -n "/^<img class/p" $tf/2.html | cut -d \" -f 4,8 | sed 's/"/:/' > TMP
  # need to change the " to : so it can be used as a delimiter for splitting.

  while read li; do
      cp $a[1] ~/hugos/towardsfreedom/content/iwrters/$a[2]/00.jpg;
  done < ~/hugos/towardsfreedom/TMP

md to org links conversion** multiple in-place replace:

  sed -i
      -e 's/\\skip1//g'
      -e 's/\"//g' 
      -e 's/  / /g' 
      -e 's/\\dynamicUp//g' 
      -e 's/\\set ignoreMelismata = ##t//g' *.ly

replace string with contents of file

sed -e "/Acss/r qq" -e "/Acss/d" index.php

puts contents of file qq right after Acss line and deletes the Acss line people have wondered how to just substitute the string and not the line

swap lines

this sort of does it but not too well there seems to be no way to delete line only the first occurance of a match

sed -e '/subsubtitle/ {h; n;p; x}' | 
sed -e '6d' | 

various extraction methods

setting and using an array in for loop

this is a bit weird, but it works

#setup L1 directories
L1dirs=(Iwrter Issays Ithngs Ibrary Ictvty Inotes)
for d in "${L1dirs[@]}"; do
    echo $d

though just this will work too:

for d in Iwrter Issays Ithngs Ibrary Ictvty Inotes; do
    echo $d

shell options

zsh -l #
zsh -v #
zsh -x #xtrace—which can prove invaluable when debugging your scripts
zsh -f #start a clean instance of zsh using the default settings can set after shell started

shortcut patterns

A-Za-z <==> :alpha: ascii,lower,upper,digit,alnum,print,blank(space,tab),space,punct

<-> numeric x a<70-90>.txt

sh_word_split option (bing)

The zsh sh_word_split option is a compatibility feature that makes zsh behave like other shells when it comes to word splitting. Word splitting is the process of breaking a string into words based on some delimiter characters, such as spaces or colons. This is often done when passing arguments to a command or looping over a list of values.

By default, zsh does not perform word splitting on unquoted parameter expansions. This means that if you have a variable that contains multiple words separated by spaces, such as `str="foo bar baz"`, and you use it as an argument to a command, such as `echo $str`, zsh will treat it as a single word and pass it as one argument to echo. This is different from other shells, such as bash or ksh, which would split the variable into three words and pass them as three arguments to echo.

The sh_word_split option makes zsh emulate the behavior of other shells and perform word splitting on unquoted parameter expansions. This means that if you set this option, `echo $str` will print `foo bar baz` on three separate lines, as if you had written `echo "$ { (@s: :)str}"`. The option also affects other forms of parameter expansion, such as command substitution and arithmetic expansion.

The sh_word_split option is useful when you want to write scripts that are compatible with other shells, or when you want to use variables that contain lists of values separated by some delimiter. However, it can also introduce unexpected behavior and errors if you are not careful with quoting. For example, if you have a variable that contains spaces in its values, such as `str="foo 'bar baz' qux"`, and you use it in a for loop, such as `for x in $str; do echo $x; done`, zsh will split the variable into four words and print them on four lines, even though you might have intended to treat `bar baz` as a single word.

A better way to handle lists of values in zsh is to use arrays, which are variables that can store multiple values without relying on delimiters. You can create an array by enclosing a list of words in parentheses, such as `arr=(foo 'bar baz' qux)`, and access its elements by using the subscript notation, such as `echo $arr[2]`. You can also use the `typeset -T` builtin command to tie an array variable to a scalar variable, so that any changes to one are reflected in the other. For example, `typeset -T str str_array` will create an array variable `str_array` that is tied to the scalar variable `str`. If you assign a value to `str`, such as `str="foo:bar:baz"`, the array variable will be updated accordingly, and vice versa.

For more information about word splitting and arrays in zsh, you can refer to the following sources:

  • [What is word splitting? Why is it important in shell programming?](^1^)
  • [How to split a string by ':' character in bash/zsh?](^2^)
  • [Z-Shell Frequently-Asked Questions](^3^)
  • [Re: Word splitting with zsh fix](^4^)

Source: Conversation with Bing, 9/12/2023 (1) zsh - What is word splitting? Why is it important in shell programming …. (2) How to split a string by ':' character in bash/zsh?. (3) Z-Shell Frequently-Asked Questions - SourceForge. (4) Re: Word splitting with zsh fix.

specific computers

  # broot setup for specific computers
  if [[$HOST = 'schnell' || $HOST = 'adagio']];


paru -S sshfs sshfs <computer>: /home/mnts/<computer> umount /home/mnts/<computer>


reads from the standard input and writes to both standard output and one or more files at the same time (it helped to solve the issue we were having)

it can be used to write to multiple files as well as append:

command | tee file1.out file2.out file3.out
command | tee -a file.out

very useful when combined with sudo:

echo "newline" | sudo tee -a /etc/file.conf

or remotely:

for e in $@; do
    echo "$e REJECT" | ssh REMOTE sudo tee -a /etc/postfix/access

text file into array


# urls should not be separated by blank lines or
# have quotes (it seems or we get missing scheme error)
# getting the urls into an array needs to be looked at

for l in "${(@f)"$(<./urls)"}"
             #echo "wget -O $fn $l"
             wget -O "$fn" "$l"

echo "all done"


variables as commands

setopt sh_word_split otherwise $s="sleep 2" is non-functional or $e="emacsclient -c" is not found


We should use this since it is zsh perl-rename may be better in some situations rename is rather limited except for very straightforward stuff

Very powerful commands though the syntax may be difficult to remember unless we get used to it through regular practice which is unlikely since we don't do this too often. So looking it up is better.

This is a multiple move based on zsh pattern matching. To get the full power of it, you need a postgraduate degree in zsh. However, simple tasks work OK, so if that's all you need, here are some basic examples:

foo.txt to foo.lis, etc

The items below shows move, copy, and link using zmv.

  zmv '(*).txt' '$1.lis'

The parenthesis is the thing that gets replaced by the $1 (not the `*', as happens in mmv, and note the `$', not `=', so that you need to quote both words).

  zmv '(**/)(*).txt '$1$2.lis'

The same, but scanning through subdirectories. The $1 becomes the full path. Note that you need to write it like this; you can't get away with '(*/).txt'.

  noglob zmv -W **/*.txt **/*.lis
  zmv -w '**/*.txt' '$1$2.lis'

These are the lazy version of the one above; with -w, zsh inserts the parentheses for you in the search pattern, and with -W it also inserts the numbered variables for you in the replacement pattern. The catch in the first version is that you don't need the / in the replacement pattern. (It's not really a catch, since $1 can be empty.) Note that -W actually inserts ${1}, ${2}, etc., so it works even if you put a number after a wildcard (such as zmv -W '*1.txt' '*2.txt'). [We alias zmv -[wW] to zm[wW]]

  zmv -C '**/(*).txt' ~/save/'$1'.lis

Copy, instead of move, all .txt files in subdirectories to .lis files in the single directory `~/save'. Note that the ~ was not quoted. You can test things safely by using the `-n' (no, not now) option. Clashes, where multiple files are renamed or copied to the same one, are picked up.

Here's a more detailed description.

Use zsh pattern matching to move, copy or link files, depending on the last two characters of the function name. The general syntax is

zmv '<inpat>' '<outstring>'

<inpat> is a globbing pattern, so it should be quoted to prevent it from immediate expansion, while <outstring> is a string that will be re-evaluated and hence may contain parameter substitutions, which should also be quoted. Each set of parentheses in <inpat> (apart from those around glob qualifiers, if you use the -Q option, and globbing flags) may be referred to by a positional parameter in <outstring>, i.e. the first (…) matched is given by $1, and so on. For example,

  zmv '([a-z])(*).txt' '${(C)1}$2.txt'

renames algernon.txt to Algernon.txt, boris.txt to Boris.txt and so on. The original file matched can be referred to as $f in the second argument; accidental or deliberate use of other parameters is at owner's risk and is not covered by the (non-existent) guarantee.

As usual in zsh, 's don't work inside parentheses. There is a special case for (**) and (***/): these have the expected effect that the entire relevant path will be substituted by the appropriate positional parameter.

There is a shortcut avoiding the use of parenthesis with the option -w (with wildcards), which picks out any expressions `*', `?', `<range>' (<->, <1-10>, etc.), `[…]', possibly followed by `#'s, `**/', `***/', and automatically parenthesises them. (You should quote any ['s or ]'s which appear inside […] and which do not come from ranges of the form `[:alpha:]'.) So for example, in

  zmv -w '[[:upper:]]*' '${(L)1}$2'

the $1 refers to the expression `[[:upper:]]' and the $2 refers to `*'. Thus this finds any file with an upper case first character and renames it to one with a lowercase first character. Note that any existing parentheses are active, too, so you must count accordingly. Furthermore, an expression like '(?)' will be rewritten as '((?))' — in other words, parenthesising of wildcards is independent of any existing parentheses.

Any file whose name is not changed by the substitution is simply ignored. Any error — a substitution resulted in an empty string, two substitutions gave the same result, the destination was an existing regular file and -f was not given — causes the entire function to abort without doing anything.


  1. Parenthesised expressions can be confused with glob qualifiers, for example a trailing '(*)' would be treated as a glob qualifier in ordinary globbing. This has proved so annoying that glob qualifiers are now turned off by default. To force the use of glob qualifiers, give the flag -Q.
  1. The pattern is always treated as an extendedglob pattern. This can also be interpreted as a feature.


You don't need braces around the 1 in expressions like '$1t' as non-positional parameters may not start with a number, although paranoiacs like the author will probably put them there anyway.

capitalize all vowels

  zmv '(*)' '${1//(#m)[aeiou]/${(U)MATCH}}'

get rid of single quotes

zmv -nQ '*.html' "\${f//'/}"

with Q we need to escape certain things like $

lowercase and space-to-hyphen

zmv -n '*' '${f:l:s/ /-}'

lowercase the directories

zmv -vw '**/(*)/' '${f:l}'

we had to run it 3 times though to get everything TODO

remove 1st four char

  zmv '*' '$f[5,-1]' (where f is a zmv generated variable)

renumber files

  i=0;zmv '(FILENAME-COMMON-PART)(*)' '$1$((i++)).mp4' #[] works too but may cause problems

  i=1; zmv -n '(img*.jpg)' '${(l:2::0:)$((i++))}$1' #[] results in weird stuff

  i=0;zmv -n '(bind.*)' '$1${(l:2::0:)}$((i++))' on bind.c, bind.h, bind.o
  mv -- bind.c bind.c000
  mv -- bind.h bind.h001
  mv -- bind.o bind.o002

running other programs

  zmv -p cp -o'-r' 'img/(*)' 'tmp/${1:u}'
  zmv -C -o '-r' 'img/(*)' 'tmp/${1:u}'
  either recursively copies contents of img/ to tmp/ uppercasing

zmW examples

Using zmv -w or zmW is very effective through globbing.

  #  upcase the filename but not the extension
  zmW '*.jpg' '${*:u}.jpg'

  # pad left with zero
  zmW 'img*.jpg' 'img${(l:2::0:)*}.jpg'

  # replace spaces with hyphens
  zmW '*' '${*// /-}'
  better than
  zmv '*-*' '${f// /-}'
  zmv '*-*' '${f:gs/ /-}'


-f  force overwriting of destination files.  Not currently passed
    down to the mv/cp/ln command due to vagaries of implementations
    (but you can use -o-f to do that).
-i  interactive: show each line to be executed and ask the user whether
    to execute it.  Y or y will execute it, anything else will skip it.
    Note that you just need to type one character.
-n  no execution: print what would happen, but don't do it.
-q  Turn bare glob qualifiers off:  now assumed by default, so this
    has no effect.
-Q  Force bare glob qualifiers on.  Don't turn this on unless you are
    actually using glob qualifiers in a pattern (see below).
-s  symbolic, passed down to ln; only works with zln or z?? -L.
-v  verbose: print line as it's being executed.
-o <optstring>
    <optstring> will be split into words and passed down verbatim
    to the cp, ln or mv called to perform the work.  It will probably
    begin with a `-'.
-p <program>
    Call <program> instead of cp, ln or mv.  Whatever it does, it should
    at least understand the form '<program> -- <oldname> <newname>',
    where <oldname> and <newname> are filenames generated. <program>
    will be split into words.
-P <program>
    As -p, but the program doesn't understand the "--" convention.
    In this case the file names must already be sane.
-w  Pick out wildcard parts of the pattern, as described above, and
    implicitly add parentheses for referring to them.
-W  Just like -w, with the addition of turning wildcards in the
    replacement pattern into sequential ${1} .. ${N} references.
-M  Force cp, ln or mv, respectively, regardless of the name of the


  • specific computers
  # broot setup for specific computers
  if [[$HOST = 'schnell' || $HOST = 'adagio']];
  • bd (using tarrasch function for zsh because bd install doesn't seem to work)
  • fzf a good addition too and works rather like the fuzzy find on broot
  • globbing seems to be inconsistent over the different versions of zsh
  • operations on multiple extensions ls *.{jpg,png}
  • list only directories (post) ls -d – *() ls -ld – *()