TAGS: WHERNTO: erudite techniq

We use the z shell and program in zsh.
block commenting trick
[ -z $BASH ] || shopt -s expand_aliases
alias BEGINCOMMENT="if [ ]; then"
alias ENDCOMMENT="fi"
BEGINCOMMENT
echo "This line appears in a commented block"
echo "And this one too!"
ENDCOMMENT
echo "This is outside the commented block"
Executing the above code would result in:
This is outside the commented block https://stackoverflow.com/questions/947897/block-comments-in-a-shell-script
concat csv
ls -1 *.csv | while read filnam ; do cat "$filnam" >> output.csv.file; done
ls -1 *.csv | sort | while read filnam ; do cat "$filnam" >> output.csv.file; done
ls -1t *.csv | while read filnam ; do cat "$filnam" >> output.csv.file; done
cp
replace softlinks with file
for f in $(find -type l);do
cp --remove-destination $(readlink $f) $f;
done
the idea is good, but remember it will find all softlinks and you may not want to do that (as we found replacing 00.jpg).
dir output to array
shopt -s nullglob
array=(*/)
shopt -u nullglob # Turn off nullglob to make sure it doesn't interfere with anything later
echo "${array[@]}" # Note double-quotes to avoid extra parsing of funny characters in filenames
https://stackoverflow.com/questions/18884992/how-do-i-assign-ls-to-an-array-in-linux-bash
ffmpeg
split video files
ffmpeg -ss 00:03:04.560 -i input.mp4 -c copy -t 00:03:39.299 output.mp4
to split video files which sometimes even losscut cannot but ffmpeg can
(fd-)find
copying files
find MMMM/local/.ocs.agio.organ/cur/ -maxdepth 1 -type f | xargs cp -t mail/cur/
find . -maxdepth 1 -type f |head -1000| xargs cp -t "$destdir"
list by size
find . -size +1M (greater than 1M)
find . -size -1M (less than 1M)
find . -size +1M -exec ls -lh {} +
find only .git dirs
fd –exclude=".cache" –type d –hidden '.git$' find ocs/ -type d -name ".git"
ignore dir
fd –exclude dir find . \( -name ".cache" -prune \) -o -name ".git" -type d
modifying files
find . -type d -name "*UNUSED" |
while read f;
do
mv $f "${f%/*}/unused";
done
using for with find is not a good idea for some reason
renumber files
best done with zmv, but using find is still interesting
# filenumberer.sh
k=0
find . -type f |
while read f;
do
k=$(($k+1))
cp $f ../NEWDIR/${(l(5)(0))k}.pdf
done
exit
# can be done as per code below, but zmv is preferable
for z in $(find /home/pradocs -name zotes.org); do
echo `mv $z ${z%/*}/zzotes.org`;
done
grep
lines not beginning with #
https://unix.stackexchange.com/questions/60994/how-to-grep-lines-which-does-not-begin-with-or
grep "^[^#;]" smb.conf
In other words, it reports lines that start with any character other than and ;. It's not the same as reporting the lines that don't start with and ; (for which you'd use grep -v '^[#;]') in that it also excludes empty lines
heroic commands in linux
https://www.youtube.com/watch?v=Zuwa8zlfXSY
exit terminal but leave all processes running
disown -a && exit
intercept stdout and log to file
cat contacts.org | tee -a log.txt | cat > /dev/null
keep command out of history
start any command with a leading space and it will not go into history
put something in the background** ramdisk for fast rw
mkdir /mnt/ram
mount -t tmpfs tmpfs /mnt/ram -o size=8192M
dd if=/dev/zero of=test.iso bs=1M count=8000
on an ssd the above worked at 2.2G/s, but with ramdisk it was 3.9G/s!
recursive directory creation
mkdir -p folder/{sub1,sub2}/{sub1,sub2,sub3}
mkdir -p folder/{1..100}/{1..10}
redo last command via sudo
sudo !!
particularly helpful if you don't want to retype a long command.
tunneling
tunneling to connect to cloud through a port of choice:
ssh -L localport:host:port-relative-to-machine login-credentials
we did this at one point to connect to lentil via wifi somehow.
use an editor to alter a long messed up command
push enter on the messed up command and then type fc to work with the command in default editor
execute the command then use bg
ls
inverse
the inverse is giving by ^
ls *M.mp4
ls ^*M.mp4
largest files and folders identification
du -h | sort -hr | head -n 10
du -ah | sort -hr | head -n 10
pad with zero
seq -w 1 11
or for more control
for i in {2..11};
do
echo $u1${(l:2::0:)i}$u2$i
done
and can get something like this
https://tubitv.com/tv-shows/524508/s01-e02-episode-2
https://tubitv.com/tv-shows/524508/s01-e03-episode-3
https://tubitv.com/tv-shows/524508/s01-e04-episode-4
redo softlinks
find ~/bin -type l -ls | tr -s " " "\t" | cut -f11,13
couldn't do tr with sed for some reason
https://unix.stackexchange.com/questions/188057/update-symlinks-by-scripting
though the ideas using realpath will work, it doesn't do what we are trying to accomplish which is to replace 'ba' with 'sh' in the softlink
perl onliners
add a line after match
perl -pi -e '/paper.ily/ and $_.="\\include \"../../zztmpls/functions.scm\"\n"' score.ly
rsync
exclude
rsync -av --delete --exclude={'dir1','dir2','file1.txt'} ~/ocs ~/mnts/sd64/
remove empty directories
rmdir --verbose **/*(/^F)
rmdir --verbose **/*(/od)
rename
rename -v pradagio pradesigner */*/* #like grep -r
sed
backref peculiarities
schnell% echo $v
[Integrating Values and Ethics into Wildlife Policy and Management Lessons from North America](http://www.mdpi.com/2076-2615/1/1/126)
schnell% echo $v | sed -r -n 's/\[(.+)\]/\1/p'
Integrating Values and Ethics into Wildlife Policy and Management Lessons from North America(http://www.mdpi.com/2076-2615/1/1/126)
schnell% echo $v | sed -rn 's/\[(.+)\]\((.*)\)/\2\1/p'
http://www.mdpi.com/2076-2615/1/1/126Integrating Values and Ethics into Wildlife Policy and Management Lessons from North America
why does the entire $v get printed in the second item?
getting pics from html file to proper directories
from html source:
<p class="hdr">Iwrters</p>
<p class="dsc">Icquaint yourself with those who contributed to this site over the years!</p>
<br/>
<div style="background-color:#ccccff; margin:1em 1em 5em 1em" >
<img class="floatrig" src="y-283.jpg" width="75" alt="Alexandra"/>
<p class="secitmtle">Alexandra</p>
<p class="secitmdsc">I love animals more than anything else!</p>
<p class="secitmurl"><a href="http://www.30bananasaday.com/profile/AlexandraR?xg_source=profiles_memberList">see</a></p>
<p class="whern"></p>
</div>
<div style="background-color:#ddddff; margin:1em 1em 5em 1em" >
<img class="floatrig" src="y-288.jpg" width="75" alt="Angelkate"/>
<p class="secitmtle">Angelkate</p>
<p class="secitmdsc">Here we find divinity, for we are it.</p>
<p class="secitmurl"><a href="288.html">see</a></p>
<p class="whern"></p>
</div>
need to pull out the y-*.jpg and put it into the alt="NAME" directory.
the problem couldn't be solved with sed because non-greedy finds aren't possible, but cut worked nicely:
sed -n "/^<img class/p" $tf/2.html | cut -d \" -f 4,8 | sed 's/"/:/' > TMP
# need to change the " to : so it can be used as a delimiter for splitting.
while read li; do
a=("${(@s/:/)li}");
cp $a[1] ~/hugos/towardsfreedom/content/iwrters/$a[2]/00.jpg;
done < ~/hugos/towardsfreedom/TMP
md to org links conversion** multiple in-place replace:
sed -i
-e 's/\\skip1//g'
-e 's/\"//g'
-e 's/ / /g'
-e 's/\\dynamicUp//g'
-e 's/\\set ignoreMelismata = ##t//g' *.ly
replace string with contents of file
sed -e "/Acss/r qq" -e "/Acss/d" index.php
puts contents of file qq right after Acss line and deletes the Acss line people have wondered how to just substitute the string and not the line
swap lines
this sort of does it but not too well there seems to be no way to delete line only the first occurance of a match
sed -e '/subsubtitle/ {h; n;p; x}' Ta-Ta-Ta.ly |
sed -e '6d' |
head
various extraction methods
https://stackoverflow.com/questions/17511639/sed-print-only-matching-group
sed -rn 's/\[(.+)\]\((.*)\)/\1/p'
tee
reads from the standard input and writes to both standard output and one or more files at the same time (it helped to solve the rejects.sh issue we were having)
it can be used to write to multiple files as well as append:
command | tee file1.out file2.out file3.out
command | tee -a file.out
very useful when combined with sudo:
echo "newline" | sudo tee -a /etc/file.conf
or remotely:
for e in $@; do
echo "$e REJECT" | ssh REMOTE sudo tee -a /etc/postfix/access
done
zmv
lowercase the directories
zmv -vw '**/(*)/' '${f:l}'
we had to run it 3 times though to get everything TODO
get rid of single quotes
zmv -nQ '*.html' "\${f//'/}"
with Q we need to escape certain things like $
rename .php to .html
zmv -Q '*.php' "\${f//\.php/.html}
renumber files
i=0;zmv '(FILENAME-COMMON-PART)(*)' '$1$((i++)).mp4' #[] works too but may cause problems
i=1; zmv -n '(img*.jpg)' '${(l:2::0:)$((i++))}$1' #[] results in weird stuff
zsh
We really need to examine the docs to sort things out:
https://zsh.sourceforge.io/Doc/Release/zsh_toc.html
capitalizing
for n in Artmisia Cytheri Vaishob Chastity Retha Eliphas; do
sed -i "s/$n/${(U)n}/g" AAlib.fountain
done
file into variable
var=$(<stuff.txt)
for itms out of a file
for l in "${(@f)"$(<filename)"}"
{
fnumber=$(youtube-dl -F $l | tail -n 1 | cut -f 1 -d ' ')
youtube-dl --format $fnumber $l
}
prompts
add these to .zshrc:
autoload -Uz promptinit
promptinit
prompt bart (or whatever)
5 prompts:
$PS1
$RPS1 rightside
$PS2 waiting for more input
$PS3 making choices
$PS4 debugging scripts
remove empty directories recursively
rmdir --verbose **/*(/^F)
rmdir --verbose **/*(/od)
[setting] up and using an array in for loop
this is a bit weird, but it works
#setup L1 directories
L1dirs=(Iwrter Issays Ithngs Ibrary Ictvty Inotes)
for d in "${L1dirs[@]}"; do
echo $d
done
though just this will work too:
for d in Iwrter Issays Ithngs Ibrary Ictvty Inotes; do
echo $d
done
shell options
zsh -l #
zsh -v #
zsh -x #xtrace—which can prove invaluable when debugging your scripts
zsh -f #start a clean instance of zsh using the default settings can set after shell started
text file into array
#!/bin/zsh
# urls should not be separated by blank lines or
# have quotes (it seems or we get missing scheme error)
# getting the urls into an array needs to be looked at
ln=1
for l in "${(@f)"$(<./urls)"}"
{
fn=$ln.mp4
#echo "wget -O $fn $l"
wget -O "$fn" "$l"
((ln++))
}
echo "all done"
exit
config files
- .zshenv - Should only contain user’s environment variables.
- .zprofile - Can be used to execute commands just after logging in.
- .zshrc - Should be used for the shell configuration and for executing commands.
- .zlogin - Same purpose than .zprofile, but read just after .zshrc.
- .zlogout - Can be used to execute commands when a shell exit.