Towards Freedom


Information, Inspiration, Imagination
truly a site for soaring Is

Shell

Shell programming notes.

TAGS: WHERNTO: erudite  techniq 

image of Shell

We use the z shell and program in zsh.

block commenting trick

[ -z $BASH ] || shopt -s expand_aliases
alias BEGINCOMMENT="if [ ]; then"
alias ENDCOMMENT="fi"

BEGINCOMMENT
  echo "This line appears in a commented block"
  echo "And this one too!"
ENDCOMMENT

echo "This is outside the commented block"

Executing the above code would result in:

This is outside the commented block https://stackoverflow.com/questions/947897/block-comments-in-a-shell-script

concat csv

  ls -1 *.csv | while read filnam ; do cat "$filnam" >> output.csv.file; done
  ls -1 *.csv | sort | while read filnam ; do cat "$filnam" >> output.csv.file; done
  ls -1t *.csv | while read filnam ; do cat "$filnam" >> output.csv.file; done

cp

replace softlinks with file

    for f in $(find -type l);do
        cp --remove-destination $(readlink $f) $f;
    done

the idea is good, but remember it will find all softlinks and you may not want to do that (as we found replacing 00.jpg).

dir output to array

shopt -s nullglob
array=(*/)
shopt -u nullglob # Turn off nullglob to make sure it doesn't interfere with anything later
echo "${array[@]}"  # Note double-quotes to avoid extra parsing of funny characters in filenames

https://stackoverflow.com/questions/18884992/how-do-i-assign-ls-to-an-array-in-linux-bash

ffmpeg

split video files

ffmpeg -ss 00:03:04.560 -i input.mp4 -c copy -t 00:03:39.299 output.mp4

to split video files which sometimes even losscut cannot but ffmpeg can

(fd-)find

copying files

find MMMM/local/.ocs.agio.organ/cur/ -maxdepth 1 -type f | xargs cp -t mail/cur/
find . -maxdepth 1 -type f |head -1000| xargs cp -t "$destdir"

list by size

find . -size +1M (greater than 1M)
find . -size -1M (less than 1M)
find . -size +1M -exec ls -lh {} +

find only .git dirs

fd –exclude=".cache" –type d –hidden '.git$' find ocs/ -type d -name ".git"

ignore dir

fd –exclude dir find . \( -name ".cache" -prune \) -o -name ".git" -type d

modifying files

find . -type d -name "*UNUSED" |
while read f;
  do
    mv $f "${f%/*}/unused";
  done

using for with find is not a good idea for some reason

renumber files

best done with zmv, but using find is still interesting

# filenumberer.sh
k=0
find . -type f |
    while read f;
    do
        k=$(($k+1))
        cp $f ../NEWDIR/${(l(5)(0))k}.pdf
    done
exit

# can be done as per code below, but zmv is preferable

for z in $(find /home/pradocs -name zotes.org); do 
echo `mv $z ${z%/*}/zzotes.org`; 
done

grep

lines not beginning with #

https://unix.stackexchange.com/questions/60994/how-to-grep-lines-which-does-not-begin-with-or

grep "^[^#;]" smb.conf

In other words, it reports lines that start with any character other than and ;. It's not the same as reporting the lines that don't start with and ; (for which you'd use grep -v '^[#;]') in that it also excludes empty lines

heroic commands in linux

https://www.youtube.com/watch?v=Zuwa8zlfXSY

exit terminal but leave all processes running

disown -a && exit

intercept stdout and log to file

cat contacts.org | tee -a log.txt | cat > /dev/null

keep command out of history

start any command with a leading space and it will not go into history

put something in the background** ramdisk for fast rw

mkdir /mnt/ram
mount -t tmpfs tmpfs /mnt/ram -o size=8192M
dd if=/dev/zero of=test.iso bs=1M count=8000

on an ssd the above worked at 2.2G/s, but with ramdisk it was 3.9G/s!

recursive directory creation

mkdir -p folder/{sub1,sub2}/{sub1,sub2,sub3}
mkdir -p folder/{1..100}/{1..10}

redo last command via sudo

sudo !!

particularly helpful if you don't want to retype a long command.

tunneling

tunneling to connect to cloud through a port of choice:
ssh -L localport:host:port-relative-to-machine login-credentials

we did this at one point to connect to lentil via wifi somehow.

use an editor to alter a long messed up command

push enter on the messed up command and then type fc to work with the command in default editor

execute the command then use bg

ls

inverse

the inverse is giving by ^

  ls *M.mp4
  ls ^*M.mp4

largest files and folders identification

du -h | sort -hr | head -n 10
du -ah | sort -hr | head -n 10

pad with zero

seq -w 1 11

or for more control

  for i in {2..11};
  do
      echo $u1${(l:2::0:)i}$u2$i
  done

and can get something like this

https://tubitv.com/tv-shows/524508/s01-e02-episode-2
https://tubitv.com/tv-shows/524508/s01-e03-episode-3
https://tubitv.com/tv-shows/524508/s01-e04-episode-4

redo softlinks

find ~/bin -type l -ls | tr -s " " "\t" | cut -f11,13

couldn't do tr with sed for some reason

https://unix.stackexchange.com/questions/188057/update-symlinks-by-scripting
though the ideas using realpath will work, it doesn't do what we are trying to accomplish which is to replace 'ba' with 'sh' in the softlink

perl onliners

add a line after match

perl -pi -e '/paper.ily/ and $_.="\\include \"../../zztmpls/functions.scm\"\n"' score.ly

rsync

exclude

rsync -av --delete --exclude={'dir1','dir2','file1.txt'} ~/ocs ~/mnts/sd64/

remove empty directories

rmdir --verbose **/*(/^F)
rmdir --verbose **/*(/od)

https://www.zsh.org/mla/users/2005/msg01375.html

rename

  rename -v pradagio pradesigner */*/* #like grep -r

sed

backref peculiarities

schnell% echo $v
[Integrating Values and Ethics into Wildlife Policy and Management Lessons from North America](http://www.mdpi.com/2076-2615/1/1/126)

schnell% echo $v | sed -r -n 's/\[(.+)\]/\1/p'
Integrating Values and Ethics into Wildlife Policy and Management Lessons from North America(http://www.mdpi.com/2076-2615/1/1/126)

schnell% echo $v | sed -rn 's/\[(.+)\]\((.*)\)/\2\1/p'
http://www.mdpi.com/2076-2615/1/1/126Integrating Values and Ethics into Wildlife Policy and Management Lessons from North America

why does the entire $v get printed in the second item?

getting pics from html file to proper directories

from html source:

<p class="hdr">Iwrters</p>
<p class="dsc">Icquaint yourself with those who contributed to this site over the years!</p>
<br/>
<div style="background-color:#ccccff; margin:1em 1em 5em 1em" >
<img class="floatrig" src="y-283.jpg" width="75" alt="Alexandra"/>
<p class="secitmtle">Alexandra</p>
<p class="secitmdsc">I love animals more than anything else!</p>
<p class="secitmurl"><a href="http://www.30bananasaday.com/profile/AlexandraR?xg_source=profiles_memberList">see</a></p>
<p class="whern"></p>
</div>

<div style="background-color:#ddddff; margin:1em 1em 5em 1em" >
<img class="floatrig" src="y-288.jpg" width="75" alt="Angelkate"/>
<p class="secitmtle">Angelkate</p>
<p class="secitmdsc">Here we find divinity, for we are it.</p>
<p class="secitmurl"><a href="288.html">see</a></p>
<p class="whern"></p>
</div>

need to pull out the y-*.jpg and put it into the alt="NAME" directory.

the problem couldn't be solved with sed because non-greedy finds aren't possible, but cut worked nicely:

  sed -n "/^<img class/p" $tf/2.html | cut -d \" -f 4,8 | sed 's/"/:/' > TMP
  # need to change the " to : so it can be used as a delimiter for splitting.

  while read li; do
      a=("${(@s/:/)li}");
      cp $a[1] ~/hugos/towardsfreedom/content/iwrters/$a[2]/00.jpg;
  done < ~/hugos/towardsfreedom/TMP

md to org links conversion** multiple in-place replace:

  sed -i
      -e 's/\\skip1//g'
      -e 's/\"//g' 
      -e 's/  / /g' 
      -e 's/\\dynamicUp//g' 
      -e 's/\\set ignoreMelismata = ##t//g' *.ly

replace string with contents of file

sed -e "/Acss/r qq" -e "/Acss/d" index.php

puts contents of file qq right after Acss line and deletes the Acss line people have wondered how to just substitute the string and not the line

swap lines

this sort of does it but not too well there seems to be no way to delete line only the first occurance of a match

sed -e '/subsubtitle/ {h; n;p; x}'  Ta-Ta-Ta.ly | 
sed -e '6d' | 
head

various extraction methods

tee

reads from the standard input and writes to both standard output and one or more files at the same time (it helped to solve the rejects.sh issue we were having)

it can be used to write to multiple files as well as append:

command | tee file1.out file2.out file3.out
command | tee -a file.out

very useful when combined with sudo:

echo "newline" | sudo tee -a /etc/file.conf

or remotely:

for e in $@; do
    echo "$e REJECT" | ssh REMOTE sudo tee -a /etc/postfix/access
done

zmv

lowercase the directories

zmv -vw '**/(*)/' '${f:l}'

we had to run it 3 times though to get everything TODO

get rid of single quotes

zmv -nQ '*.html' "\${f//'/}"

with Q we need to escape certain things like $

rename .php to .html

zmv -Q '*.php' "\${f//\.php/.html}

renumber files

  i=0;zmv '(FILENAME-COMMON-PART)(*)' '$1$((i++)).mp4' #[] works too but may cause problems
  i=1; zmv -n '(img*.jpg)' '${(l:2::0:)$((i++))}$1' #[] results in weird stuff

zsh

We really need to examine the docs to sort things out:
https://zsh.sourceforge.io/Doc/Release/zsh_toc.html

capitalizing

for n in Artmisia Cytheri Vaishob Chastity Retha Eliphas; do
    sed -i "s/$n/${(U)n}/g" AAlib.fountain
done

file into variable

var=$(<stuff.txt)

for itms out of a file

for l in "${(@f)"$(<filename)"}"
         {
             fnumber=$(youtube-dl -F $l | tail -n 1 | cut -f 1 -d ' ')
             youtube-dl --format $fnumber $l
         }

prompts

add these to .zshrc:

autoload -Uz promptinit
promptinit
prompt bart (or whatever)

5 prompts:

$PS1
$RPS1 rightside
$PS2 waiting for more input
$PS3 making choices
$PS4 debugging scripts

remove empty directories recursively

rmdir --verbose **/*(/^F)
rmdir --verbose **/*(/od)

https://www.zsh.org/mla/users/2005/msg01375.html

[setting] up and using an array in for loop

this is a bit weird, but it works

#setup L1 directories
L1dirs=(Iwrter Issays Ithngs Ibrary Ictvty Inotes)
for d in "${L1dirs[@]}"; do
    echo $d
done

though just this will work too:

for d in Iwrter Issays Ithngs Ibrary Ictvty Inotes; do
    echo $d
done

shell options

zsh -l #
zsh -v #
zsh -x #xtrace—which can prove invaluable when debugging your scripts
zsh -f #start a clean instance of zsh using the default settings can set after shell started

text file into array

#!/bin/zsh

# urls should not be separated by blank lines or
# have quotes (it seems or we get missing scheme error)
# getting the urls into an array needs to be looked at

ln=1
for l in "${(@f)"$(<./urls)"}"
         {
             fn=$ln.mp4
             #echo "wget -O $fn $l"
             wget -O "$fn" "$l"
             ((ln++))
         }

echo "all done"

exit

config files

  • .zshenv - Should only contain user’s environment variables.
  • .zprofile - Can be used to execute commands just after logging in.
  • .zshrc - Should be used for the shell configuration and for executing commands.
  • .zlogin - Same purpose than .zprofile, but read just after .zshrc.
  • .zlogout - Can be used to execute commands when a shell exit.