Towards Freedom

howto combine strings and regex https://stackoverflow.com/questions/21416917/multiple-regular-expressions-in-clojure

  (defn union-regex [& patterns]
    \\\"joins regex together\" with |\"
    (re-pattern
     (apply str
            (interpose \""|"
                       (map #(str "(" % ")")
                            patterns)))))
  (union-regex #"\w+ " (re-pattern w) " " #"\w+") ;; w set to "how"

=> #"(\w+ )|(how)|( )|(\w+)"

or this solution from webscraper project

  (defn join-regex [& patterns]
    "joins regex together"
    (re-pattern
     (apply str
            (map #(str %)
                 patterns))))
  (join-regex #"\w+ " (re-pattern w) " " #"\w+") ;; w set to "how"

=> #"\w+ how \w+"

https://stackoverflow.com/questions/26081820/regular-expression-to-extract-whole-sentences-with-matching-word [^.]* flung [^.]*\. [^.?!]*(?<=[.?\s!])flung(?=[\s.?!])[^.?!]*[.?!]

(defn mk-tohlc "creates the dataset with date, open, high, low, close from filename" [fpath] (let [time (or "t" "column-0") open (or "o" "column-1") high (or "h" "column-2") low (or "l" "column-3") close (or "c" "column-4")] (rename-columns (->dataset fpath {:header-row? false :parser-fn {time :string open :float32 high :float32 low :float32 close :float32}}) {time :time open :open high :high low :low close :close})))

https://stackoverflow.com/questions/8566531/listing-files-in-a-directory-in-clojure (seq (.list (clojure.java.io/file "."))) (map #(.getPath %) (file-seq (clojure.java.io/file ".")))

file https://clojuredocs.org/clojure.java.io/file

fs - File system utilities for Clojure https://github.com/Raynes/fs

;using regex (def files (filter #(re-find #"1D|4H" %) (map #(.getPath %) (file-seq (clojure.java.io/file "/zata/truefx"))))) ;using string (def files (filter #(clojure.string/includes? % "1D") (map #(.getPath %) (file-seq (clojure.java.io/file "/zata/truefx"))))) ;going through a vector (def files (map (fn [x] (filter #(clojure.string/includes? % x) (map #(.getPath %) (file-seq (clojure.java.io/file "/zata/truefx"))))) ["1D" "4H" "1D"]))

(def filepaths "produces a list of truefx path strings" (flatten (map (fn [x] (sort (filter #(str/includes? % x) (map #(.getPath %) (file-seq (clojure.java.io/file "/zata/truefx")))))) ["1D" "4H" "1H"])))

(def filenames "extracts file names from file paths" (map #(last %) (map #(str/split % #"/") filepaths)))

(map #(apply max %) (partition 3 r)) since partition gives list, we need to apply max to each list we map the apply max over the partitions

(filter #(not(nil? %)) [1 nil 2 3 nil 5]) (1 2 3 5) (remove nil? [1 nil 2 3 nil 5]) (1 2 3 5)

(defn nusr "identifies sr-levels for instrument using nu" [fp] (let [[o h l c fn pf] (inst-info fp) part-size 5 step-size 1 midpt (quot part-size 2) hi (partition part-size step-size h) lo (partition part-size step-size l) n (remove nil? (map #(nu % max midpt) hi)) u (remove nil? (map #(nu % min midpt) lo))] [:ns n :us u]))

(defn nu "finds a nu given seq, maxmin, midpt of seq" [sq maxmin midpt] (let [midval (nth sq midpt)] (when (= (apply maxmin sq) midval) midval)))

(nusr (first truefx-filepaths))

problem came up trying to work with bounsr and penesr where we needed to take 2 vectors and generate a vector of vectors matching items in sequence:

[4 4 6] [7 8 9] => 4 7] [4 8] [6 9

(zipmap [4 4 6] [7 8 9]) => {4 8, 6 9} due to repetition of the 4 because it is a map

this thread Is there an equivalent for the Zip function in Clojure had 2 solutions:

(map vector [4 4 6] [7 8 9]) => ([4 7] [4 8] [6 9]) (partition 2 (interleave [1 2 3] [4 5 6])) => ((1 4) (2 5) (3 6))

(interleave [1 2 3] [4 5 6] [7 8 9]) (apply interleave 1 2 3] [4 5 6] [7 8 9)

or as a function

(defn zip [& colls] (partition (count colls) (apply interleave colls)))

(apply F [1 2 3 4 5]) => (F 1 2 3 4 5) (map F [1 2 3 4 5]) => [(F 1) (F 2) (F 3) (F 4) (F 5)] https://stackoverflow.com/questions/2311528/clojure-apply-vs-map

(require '[clojure.set :refer [map-invert]])

(defn balance [form] (let [brackets {\[ \] \( \) \{ \}} scan (fn [q x] (cond (brackets x) (conj q x) ((map-invert brackets) x) (if (= (brackets (peek q)) x) (pop q) (throw (ex-info (str "Unmatched delimiter " x) {}))) :else q))] (reduce scan [] form)))

(balance "(let [a (inc 1]) (+ a 2))") ExceptionInfo Unmatched delimiter ]

(balance "(let [a (inc 1)] (+ a 2))") []

(type (incanter/to-matrix ds)) mikera.arrayz.impl.JoinedArray or vectorz/matrix (type (m/array ds)) clojure.lang.PersistentVector

1. using $ producing lazyseq

his ($ :h dohlc) clojure.lang.LazySeq or (sel dohlc :cols :h) los ($ :l dohlc) clojure.lang.LazySeq or (sel dohlc :cols :l)

1. using to-map producing persistent vectors

ds (d/to-map dohlc) clojure.lang.PersistentArrayMap his (:h ds) clojure.lang.PersistentVector los (:l ds) clojure.lang.PersistentVector

1. using m/array producing vectorz

ds (d/to-map dohlc) clojure.lang.PersistentArrayMap his (m/array (:h ds)) mikera.vectorz.Vector los (m/array (:l ds)) mikera.vectorz.Vector can't (m/array ds) because clojure.lang.PersistentArrayMap cannot be cast to java.lang.Number though (type (m/array d)) clojure.lang.PersistentVector

it seems as though we can use $ for pretty well everything we use sel for (sel d :cols :h) <=> ($ :h d) (sel d :rows (range 2 11) :cols :d) <=> ($ (range 1 11) :d d) (sel d :rows 1 :cols :h) <=> (sel d :rows 1 :cols 2) <=> (sel d 1 2) <=> ($ 1 2 d) <=> ($ 1 :h d)

head is an incanter fn and cannot be used with lazy sequences

print index and values (map (fn [x] [(.indexOf hivals x) x]) hivals)

the ratios function doesn't produce same results as the individual codings we haven't figured out why or which is really correct (defn anulysisV2 " analyzes brs and prs price action on nu

dohlc dataset columns :d date :o open :h high :l low :c close identify nu using partition map bsr? psr? to hilo and determine how many of each " [filepath] (let [[dohlc filename pipfactor] (instrument-dohlc filepath) dohlc (d/select-rows dohlc (range 33)) dohlc-size (nrow dohlc) ;size of dataset less 2 because of header buffer (/ 12.0 pipfactor) ;buffer line calculation in pips for pair part-size 7 ;size of partition chosen [nseq useq] (nu-finder dohlc)

+brs? (fn [nu hilo cl >< -+] "n: (brs? n hi cl < -) u: (bsr? u lo cl > )" (and (btn? hilo (- nu buffer) ( nu buffer)) (>< cl (-+ nu buffer)))) prs? (fn [nu cl hilo >< -+] "n: (prs? n hi cl > ) u: (prs? u lo cl < -" (and (>< hilo (- nu buffer)) (>< cl nu)))

ratios (fn [rs nu-seq] (loop [nu nu-seq brs 0 prs 0 i 0] (let [nu0 (first nu-seq) hi ($ i :h dohlc) lo ($ i :l dohlc) cl ($ i :c dohlc) [is-brs is-prs] (if (= rs "r") [(brs? nu0 hi cl < -) (prs? nu0 hi cl > +)] [(brs? nu0 lo cl > +) (prs? nu0 lo cl < -)]) ] (if (> i (- dohlc-size part-size)) [brs prs] (cond is-brs (recur nu (inc brs) prs (inc i)) is-prs (recur (drop 1 nu) brs (inc prs) (inc i)) :else (recur nu brs prs (inc i))) ))))

n-ratio (loop [nn nseq brs 0 prs 0 i 0] (let [firstnn (first nn) hi ($ i :h dohlc) cl ($ i :c dohlc)] (if (> i (- dohlc-size part-size)) [brs prs] (cond (brs? firstnn hi cl < -) (recur nn (inc brs) prs (inc i)) (prs? firstnn hi cl > +) (recur (drop 1 nn) brs (inc prs) (inc i)) :else (recur nn brs prs (inc i))) ))) u-ratio (loop [uu useq brs 0 prs 0 i 0] (let [firstuu (first uu) lo ($ i :l dohlc) cl ($ i :c dohlc)] (if (> i (- dohlc-size part-size)) [brs prs] (cond (brs? firstuu lo cl > +) (recur uu (inc brs) prs (inc i)) (prs? firstuu lo cl < -) (recur (drop 1 uu) brs (inc prs) (inc i)) :else (recur uu brs prs (inc i))) ))) n-ratio (ratios "r" nseq) u-ratio (ratios "s" useq) ] [n-ratio u-ratio] ))

while the ideas here are ok we require 20s by not using partitions and 25ms when we do!! however, we need a way to figure out the index

(defn nu-finderV2 " finds nu give a dohlc dataset returns i nn] [i uu " [dohlc] (let [size (nrow dohlc) nu-breadth 7 ;nu width buf-bars (quot nu-breadth 2) ;number of bars on either side of nu ibeg buf-bars ;begining bar to look for nu iend (- (nrow dohlc) 1 buf-bars) ;ending bar to look for nu nu? (fn [i hilo maxmin] (= ($ i hilo dohlc) (apply maxmin ($ (range (- i buf-bars) (+ i buf-bars 1)) hilo dohlc)))) nus (fn [hilo maxmin] (loop [i ibeg iv []] (if (>= i iend) iv (if (nu? i hilo maxmin) (recur (+ i buf-bars 1) (conj iv [i ($ i hilo dohlc)])) (recur (inc i) iv)))))] [(nus :h max) (nus :l min)]))

(defn anulysisV1 " analyzes brs and prs price action on nu

dohlc dataset columns :d date :o open :h high :l low :c close identify nu using partition map bsr? psr? to hilo and determine how many of each " [filepath] (let [[dohlc filename pipfactor] (instrument-dohlc filepath) dohlc-size (nrow dohlc) ;size of dataset less 2 because of header buffer (/ 12.0 pipfactor) ;buffer line calculation in pips for pair part-size 7 ;size of partition chosen nn (nu-finder ($ :h dohlc) part-size max) uu (nu-finder ($ :l dohlc) part-size min) in-zone? (fn [hilo nu] (btn? hilo (- nu buffer) (+ nu buffer)))] (loop [i 4 brs 0] (if (> i (- dohlc-size 9)) brs (recur (inc i) (if (in-zone? ($ i :h dohlc) (first nn)) (inc brs) brs))) ) [(count nn) (count uu)]))

(defn mk-dohlc "creates the dataset with date, open, high, low, close from filename" [filename] (d/rename-columns (read-dataset filename) {:col0 :d :col1 :o :col2 :h :col3 :l :col4 :c}))

(defn nu-finderV1 " finds all nu in a sequence of hilo given a sequence, the partition size, max|min operator for his|los TODO have option to remove invalid nu " [aseq part-size maxmin] (let [partitions (partition part-size 1 aseq) midpt (quot part-size 2) nu (for [p partitions :when (= (apply maxmin p) (nth p midpt))] (nth p midpt))] nu))

(def truefx-filepaths "produces a list of truefx path strings" (vectorize (flatten (map (fn [x] (sort (filter #(str/includes? % x) (map #(.getPath %) (file-seq (clojure.java.io/file "/zata/truefx")))))) ["1D" "4H" "1H"]))))

(defn adjacent-confluence "determines oc confluence behavior of adjacent bars" [fp] (let [dohlc (mk-dohlc fp) fn (filename-from-filepath fp) pf (pip-factor fn) m (to-matrix (ds/remove-columns dohlc [:d])) o (sel m :cols 0) h (sel m :cols 1) l (sel m :cols 2) c (sel m :cols 3) co (map pips c o (repeat pf)) pn (map #(pos? (* %1 %2)) co (drop 1 co)) p% (Math/round (/ (count (filter true? pn)) (float (count pn)) 0.01))] (printf "%s %d %n" fn p%) p%))

interesting item to look at Group-by too slow (13 million-lines file) the solution is very instructive and covers generating a test file as well as timing.

here is a vector of maps (def asym-parts [{:name "head" :size 3} {:name "mouth" :size 1} {:name "nose" :size 1} {:name "neck" :size 2} {:name "left-eye" :size 1} {:name "left-ear" :size 1}to {:name "left-shoulder" :size 3}])

match the :name to be "nose" and print that item TODO

an interesting solution to something similar https://stackoverflow.com/questions/18176372/clojure-get-map-key-by-value

(->> {:a "bar" :b "foo" :c "bar" :d "baz"} ; initial map (group-by val) ; sorted into a new map based on value of each key (#(get % "bar")) ; extract the entries that had value "bar" (map key)) ; get the keys that had value bar or (map key (#(get % "bar") (group-by val {:a "bar" :b "foo" :c "bar" :d "baz"})))

<2020-07-24 Fri> figured out how to install own work to .m2 started rewrite of algolyze using these ideas <2020-07-23 Thu> went through clojure style guide got better understanding of ns installed and worked with hylang - though we should check out cljpython too at some point

discussion on tech.ml was very promising with both gigasquid and joinr praising it. so that's what we'll go with for now - convert existing items to it and look for a visualization mechanism. <2020-07-22 Wed> got nuen.clj v1 done and testing.

<2020-07-18 Sat> go thru clojure reader is unsafe post about times post about project mechanism

<2020-07-17 Fri> nuen mostly done thx to help on clojurians

techml read into techml dataset a csv file nearly 3 million lines drop one column form new column the result of subtracting one column from another printout head of adjusted dataset

timings: lein run 62.93s user 0.86s system 123% cpu 51.800 total emacs repl "Elapsed time: 40112.441757 msecs" java -jar target/uberjar/techml-0.1.0-SNAPSHOT-standalone.jar 29.95s user 0.77s system 162% cpu 18.916 total

<2020-07-16 Thu> start nuen

rewrite anulysis for tech.ml rewrite anulysis for arrays

<2020-07-15 Wed> finish mikera and incanter build a matrix, dataset, array toolkit (mikera incanter and anything else) plan for clj cheatsheet explorations -> make own cheatsheet from it and others plan for java function access

will conversion to separate arrays be more efficient for subsequent processing? using arrays instead of incanter datasets may be practical?

<2020-07-14 Tue> finishing mikera and incanter we should go through cheatsheet items and understand how the main functions in the categories work

<2020-07-13 Mon> last couple of days were spent with julia programming. though the docs are excellent, emacs doesn't work too well with it right now. emacs does work well with R. so may be …

attempt to get python libraries on clojure using libpython-clj - but carin's example doesn't work completely. anyway, we don't really need to do this now because what we were interested in seeing was how well anulysis worked and we can actually do this better in other ways - such as plotting hlc dots with buffers from nu

checked out graalvm and that does look very interesting! again, we don't need it now, but it is definitely something to keep in mind for the future considering one demo showed 20 performance improvement. it may be useful even without its polyglot capabilities.

for now, here's what we focus on:

• learn mikera and incanter
• learn what we can do with direct access to java
• optimize anulysis
• get nuen completed

<2020-07-10 Fri> got R and candlesticks working - it's pretty good! main difficulty was that manjaro doesn't autoload make and gcc-fortran

plotly.clj doesn't seem to have candlesticks plan is to go through incanter plotting and compare to plotly.clj plotting

<2020-07-09 Thu> anulysis finish hopefully

plan for manual trading

1. examine situation around 12am
2. put in suitable entry limit or stop trades with sltp
3. try to exit trade around 12pm next day
4. close orders and start at 1 again

<2020-07-08 Wed> cider site debugging cider debugging tools debugging in clojure several items not quite understood but we now have a useable debugging mechanism

more learnt about loop-recur in that the main if statement can have various forms (eg let) in true or false. this knowledge enabled the best version to-date: (fn [rs nu-seq] (loop [nu nu-seq brs 0 prs 0 i 0] (if (or (> i (- dohlc-size part-size)) (empty? nu)) [brs prs] (let [nu0 (first nu-seq) hi ($ i :h dohlc) lo ($ i :l dohlc) cl ($ i :c dohlc) [is-brs is-prs] (if (= rs "r") [(brs? nu0 hi cl < -) (prs? nu0 hi cl > +)] [(brs? nu0 lo cl > +) (prs? nu0 lo cl < -)]) ] (cond is-brs (recur nu (inc brs) prs (inc i)) is-prs (recur (drop 1 nu) brs (inc prs) (inc i)) :else (recur nu brs prs (inc i)))))))

<2020-07-07 Tue> getting nu the partition mechanism provides far faster results than iterating over groups of 7 bars - even if you skip bars regularly because after a nu the latter three can never be a nu. however, the partitions don't provide anyway of keeping track of the index. a problem can develop in that as we go through nu we could be counting brs even before the occurence of the nu!

so here are 2 ideas: figure out the index of the nu through a loop-recur setup. something like loop ds to find index conj index+accumulatedindices to nu drop index+1 ds

similar idea, but we don't need to actually put the index anywhere. find index of nu drop to index+1 start to check for brs once prs drop current nu do over again this way we only start checking for brs after we find where the nu is

<2020-07-06 Mon> figured out how to use loop-recur better - it is a likeable combo and seems preferable to me than recursion now. got a good version of nu-finder done which includes index amongst other things and can be used outside anulysis. <2020-07-05 Sun> setup nu-orders.clj for populating nu with orders <2020-07-04 Sat> finish anulysis <2020-07-03 Fri> go through google hits example

<2020-07-02 Thu> determine whether to use core.matrix or incanter for most things incanter is being incorporated within core.matrix and it is not really developed much. so we'll make the switch and start looking at some of the other data science libraries.

discovered core.matrix creator mike anderson or mikera who has good youtube videos on the topic

seems as though core.matrix (most operations) and incanter (rw graphics etc) can work well together.

<2020-07-01 Wed> issues with datasets and matrices using map and apply figured out a lot about for loop <2020-06-30 Tue> dilemmas with figuring out how to work hilo with nus <2020-06-29 Mon> figuring out a more iterative anulysis <2020-06-28 Sun> work on sicp - understanding cond and if <2020-06-27 Sat> started sicp and INDUCTIVESYNTHESISOFFUNCTIONALPROGRAMS

<2020-06-26 Fri> figure out mechanism to compare hilo to stacked nu problem is that i keep wanting to put things into boxes rather than thinking functionally.

<2020-06-25 Thu> work on anulysis using stacks and proper influence.

(def d (ds/select-rows ds (range 9))) (ds/add-column d :sums (mt/add (sel d :cols :o) (sel d :cols :h))) (ds/add-column d :sums (mt/gt (sel d :cols :c) 83)) (ds/add-column d :sums (mt/mul (sel d :cols :c) (mt/gt (sel d :cols :c) 83))) ;crude

<2020-06-24 Wed> creating bounsr and penesr functions within let got involved with creating pairs

though there are some nice ideas in V1 of anulysis the overall attempt to have nu checked over the entire matrix is meaningless and produced absurdly high numbers - eg almost 8000 bounsr for nrow = 2248. so we need to restrict the nu influence to forward items only and have them flip after first penesr and disappear after a second penesr.

(defn anulysisV1 "analyzes pa near nu" [fp] (let [[ops his los cls fnm pf] (instrument-info fp) buffer (/ 12 pf) bounsr? (fn [nus hilos cls >< -+] (count (remove nil? (for [nu nus [hilo cl] (zip hilos cls)] (let [nu-buffer (-+ nu buffer)] (if (and (btn? hilo [nu nu-buffer]) (>< nu-buffer cl)) true)))))) bpsr? (fn [nus hilos ops cls >< -+] (for [nu nus [hilo op cl] (zip hilos ops cls)] (let [b-buffer (-+ nu buffer) p-buffer (-+ nu (- buffer))] (cond (and (btn? hilo [nu b-buffer]) (>< b-buffer cl)) :bounsr (and (btn? nu [op cl]) ) :penesr)))) part-size 5 midpt (quot part-size 2) hi-partition (partition part-size 1 his) lo-partition (partition part-size 1 los) nn (remove nil? (map #(nu-finder % max midpt) hi-partition)) uu (remove nil? (map #(nu-finder % min midpt) lo-partition)) boun-s (bounsr? uu los cls < +) boun-r (bounsr? nn his cls > -)

isbounsr #(= :bounsr %) ispenesr #(= :penesr %) bpu (bpsr? uu los ops cls < +) bpn (bpsr? nn his ops cls > -) ] [fnm boun-s boun-r (count (filter isbounsr bpu)) (count (filter isbounsr bpn)) (count (filter ispenesr bpu)) (count (filter ispenesr bpn))]))

(map anulysis (take 1 truefx-filepaths)) => (["audjpy1D.csv" 3829 3607]) => (["audjpy1D.csv" 4167 4035])

<2020-06-23 Tue> coined term lalwal lose a little win a lot rewrote outliers in truefx files - all were with the lows. some involving CHF may have been legitimate. anulysis almost done after figuring out that :while is not to be used with for - not sure what it is

<2020-06-22 Mon> started anulysis

<2020-06-21 Sun> get barchar working first idea of adjacent-confluence may work in reverse for shorter tf

started snippets in this file

made neat ns discovry - whatever is in core.clj including def/defn becomes available to every other file provided the namespace isn't changed! this really means that we can load up core.clj with whatever is required rather than import everything for every related program we create!

<2020-06-20 Sat> working with dataset functions add-column, reorder-column, remove-column wrote sel-col function which is useful to work with single column in dataset

discovered that sel on a dataset is very slow compared to sel on a matrix it seems to be far, far faster to convert a dataset to a matrix to do things and then convert it back if you want to utilize the variety of types a dataset offers. (see xpts in datmat.clj)

<2020-06-19 Fri> investigating various things all week without entry into diary many of clojure.core.matrix is replacing the incanter functions: see updates to incanter 2.0 https://github.com/incanter/incanter/wiki/Incanter-2.0-change-log so we investigate these in datmat.clj

<2020-06-11 Thu> get-orders-for done rearrange the ztut which was getting awkward as far as ns goes as well as other things. set it all up on gitlab

changing to ns works with C-c M-n after we C-c C-k the buffer. for some reason, (in-ns namespace) doesn't always work. this page is useful as may be other things on that site.

<2020-06-10 Wed> incanter things work nicely so far, but we need to become far more knowledgeable working with datasets, functions (what they do) and plotting.

also, the how i start thing is a bit confusing as carin jumps to things that we're not familiar with so we'll get back to that later.

<2020-06-09 Tue> incanter:

• work through data sorcery

  • cheatsheet use regularly to recall syntax and concepts
  • make use of api documentation throughout
  • namespace switching solved: repl takes main ns, but can switch with (in-ns 'whatever) so long as it is seen
• started How I Start which should be useful for namespace understanding and testing

<2020-06-08 Mon> incanter:

• work through ch6 Working with Incanter Datasets from Clojure Data Analysis Cookbook which was helpful though the datasets used were huge and therefore difficult to examine in some situations

<2020-06-07 Sun> installed inlein started thinking bayes

<2020-06-06 Sat> transferred from github to gitlab which seems to be smoother and cleaner. cleaned up tutorials into ztut folder cleaned up clj bookmarks went through excellent birdseye view of clj A Brief Beginner's Guide To Clojure

<2020-06-05 Fri> more cider it seems that form order is important because loading the buffer when dependencies don't come first makes eval impossible. cider has a lot of functions, but we need to figure out which ones we'll actually use successfully connected to an existing repl at localhost after getting port using: connection info C-c M-d

downloaded cheatsheets: clojuredocs

paredit:

cider:

<2020-06-04 Thu> got all of oanda ide to date as far as we had python going order fn greatly simplified - we never needed to use if-let

investigated cider further watched alchy bozbat

<2020-06-03 Wed> stuck on json not being parsed but that was because clojure follows the curl more closely using :body instead of :json whereas python didn't

<2020-06-02 Tue> get api functions and sring theory to where we were with python all api done as far as python progress except for the json issue

hobbit problem using reduce is neatest

excellent video with edmund jackson data science and clojure shows the landscape in 2012 and what options there were then probably worth following up on this.

<2020-06-01 Mon> understood the hobbit setup mostly in brave it is messy using loop, but much messier other ways

<2020-05-31 Sun> set up tut structures in clojure directory set up problems to solve file in to-solve.clj file

Clojure Libs and Namespaces: require, use, import and ns (require 'clojure.string 'clojure.test) (require '[clojure.string :as string]) to avoid (require ['clojure.string :as 'string]) (require 'clojure.string '[clojure.test :as test]) (require '(clojure string test)) (require '(clojure [string :as string] test))

consider the incanter lib issue (use '(incanter core stats charts io)) works right away and we can call view, histogram etc (require '(incanter core stats charts io)) doesn't seem to work at all, but that's due to ns issues: (incanter.core/view (incanter.charts/histogram (incanter.stats/sample-normal 1000))) is just fine

also see Clojure Namespaces and Vars for better ways as well as this answer from stackoverflow. we can solve this with (ns tut-incanter.core (:require (incanter [core :refer :all] [stats :refer :all] [charts :refer :all]))) though (use '(incanter core stats charts)) seems much simpler

both api requests with clj-http and json readability accomplished next step is to get functions to work

<2020-05-30 Sat> figuring out git mostly getting through brave ch3 except last exercise

<2020-05-29 Fri> started clojure for brave updated several youtube resources

<2020-05-28 Thu> finished cds introduction

<2020-05-27 Wed> halfway through cds introduction why clojure video by vijay is useful

<2020-05-26 Tue> continued with cds set up resources and videos watched russ olsen functional programming in 40 min

<2020-05-25 Mon> set up emacs for clojure watched rick hickey video on creating it got cider and lein working started cds tutorial

origin https://github.com/clojurians-org/clojure-ebook-2.git (fetch) origin https://github.com/clojurians-org/clojure-ebook-2.git (push)

Clojure website Clojure Documentation clojure cheatsheet CIDER ClojureDocs ask.clojure forum Pedestal kixi stats

Physics in Clojure - Colin Smith Emacs and Clojure, a Lispy Love Affair Solving Problems the Clojure Way - Rafal Dittwald Building Machine Learning Models with Clojure and Cortex - Joyce Xu Why Clojure? by Vijay Kiran Deep Learning Needs Clojure - Carin Meier

cider-jack-in C-c M-j

do programming with awareness of each function's io

  clj -X:new-app :name pradagio/myapp
  clj -X:new-lib :name pradagio/mylib

after adding to ~/.clojure/deps.edn: ;; add these into your :aliases map:

  :new-app {:extra-deps {com.github.seancorfield/clj-new
                         {:mvn/version "1.1.309"}}
            :exec-fn clj-new/create
            :exec-args {:template "app"}}
  :new-lib {:extra-deps {com.github.seancorfield/clj-new
                         {:mvn/version "1.1.309"}}
            :exec-fn clj-new/create
            :exec-args {:template "lib"}}}

deps.edn can help you figure out how to do things because of aliases :run-x :uberjar etc contains howto instructions! clj -X:run-x clj -X:uberjar

https://stackoverflow.com/questions/8566531/listing-files-in-a-directory-in-clojure

  (seq (.list (clojure.java.io/file ".")))
  (map #(.getPath %)
       (file-seq (clojure.java.io/file ".")))

file https://clojuredocs.org/clojure.java.io/file

fs - File system utilities for clojure https://github.com/Raynes/fs

;using regex

  (def files
    (filter #(re-find #"1D|4H" %)
            (map #(.getPath %)
                 (file-seq (clojure.java.io/file "/zata/truefx")))))

;using string

(def files 
  (filter #(clojure.string/includes? % "1D")
          (map #(.getPath %)
               (file-seq (clojure.java.io/file "/zata/truefx")))))

;going through a vector

  (def files
    (map (fn [x]
           (filter #(clojure.string/includes? % x)
                   (map #(.getPath %)
                        (file-seq (clojure.java.io/file "/zata/truefx")))))
         ["1D" "4H" "1D"]))

  (def filepaths
    "produces a list of truefx path strings"
    (flatten (map (fn [x]
                    (sort (filter #(str/includes? % x)
                                  (map #(.getPath %)
                                       (file-seq (clojure.java.io/file "/zata/truefx"))))))
                  ["1D" "4H" "1H"])))

  (def filenames
    "extracts file names from file paths"
    (map #(last %)
         (map #(str/split % #"/")
                 filepaths)))

since partition gives list, we need to apply max to each list we map the apply max over the partitions

  (map #(apply max %) (partition 3 r))

problem came up trying to work with bounsr and penesr where we needed to take 2 vectors and generate a vector of vectors matching items in sequence:

[4 4 6] [7 8 9] => 4 7] [4 8] [6 9

(zipmap [4 4 6] [7 8 9]) => {4 8, 6 9} due to repetition of the 4 because it is a map

this thread Is there an equivalent for the Zip function in Clojure had 2 solutions:

(map vector [4 4 6] [7 8 9]) => ([4 7] [4 8] [6 9]) (partition 2 (interleave [1 2 3] [4 5 6])) => ((1 4) (2 5) (3 6))

(interleave [1 2 3] [4 5 6] [7 8 9]) (apply interleave 1 2 3] [4 5 6] [7 8 9)

or as a function

(defn zip [& colls] (partition (count colls) (apply interleave colls)))

(apply F [1 2 3 4 5]) => (F 1 2 3 4 5) (map F [1 2 3 4 5]) => [(F 1) (F 2) (F 3) (F 4) (F 5)] https://stackoverflow.com/questions/2311528/clojure-apply-vs-map

(type (incanter/to-matrix ds)) mikera.arrayz.impl.JoinedArray or vectorz/matrix (type (m/array ds)) clojure.lang.PersistentVector

clojure.core.matrix.impl.dataset.DataSet

1. using $ producing lazyseq

his ($ :h dohlc) clojure.lang.LazySeq or (sel dohlc :cols :h) los ($ :l dohlc) clojure.lang.LazySeq or (sel dohlc :cols :l)

1. using to-map producing persistent vectors

ds (d/to-map dohlc) clojure.lang.PersistentArrayMap his (:h ds) clojure.lang.PersistentVector los (:l ds) clojure.lang.PersistentVector

1. using m/array producing vectorz

ds (d/to-map dohlc) clojure.lang.PersistentArrayMap his (m/array (:h ds)) mikera.vectorz.Vector los (m/array (:l ds)) mikera.vectorz.Vector can't (m/array ds) because clojure.lang.PersistentArrayMap cannot be cast to java.lang.Number though (type (m/array d)) clojure.lang.PersistentVector

it seems as though we can use $ for pretty well everything we use sel for (sel d :cols :h) <=> ($ :h d) (sel d :rows (range 2 11) :cols :d) <=> ($ (range 1 11) :d d) (sel d :rows 1 :cols :h) <=> (sel d :rows 1 :cols 2) <=> (sel d 1 2) <=> ($ 1 2 d) <=> ($ 1 :h d)

the ratios function doesn't produce same results as the individual codings we haven't figured out why or which is really correct (defn anulysisV2 " analyzes brs and prs price action on nu

dohlc dataset columns :d date :o open :h high :l low :c close identify nu using partition map bsr? psr? to hilo and determine how many of each " [filepath] (let [[dohlc filename pipfactor] (instrument-dohlc filepath) dohlc (d/select-rows dohlc (range 33)) dohlc-size (nrow dohlc) ;size of dataset less 2 because of header buffer (/ 12.0 pipfactor) ;buffer line calculation in pips for pair part-size 7 ;size of partition chosen [nseq useq] (nu-finder dohlc)

+brs? (fn [nu hilo cl >< -+] "n: (brs? n hi cl < -) u: (bsr? u lo cl > )" (and (btn? hilo (- nu buffer) ( nu buffer)) (>< cl (-+ nu buffer)))) prs? (fn [nu cl hilo >< -+] "n: (prs? n hi cl > ) u: (prs? u lo cl < -" (and (>< hilo (- nu buffer)) (>< cl nu)))

ratios (fn [rs nu-seq] (loop [nu nu-seq brs 0 prs 0 i 0] (let [nu0 (first nu-seq) hi ($ i :h dohlc) lo ($ i :l dohlc) cl ($ i :c dohlc) [is-brs is-prs] (if (= rs "r") [(brs? nu0 hi cl < -) (prs? nu0 hi cl > +)] [(brs? nu0 lo cl > +) (prs? nu0 lo cl < -)]) ] (if (> i (- dohlc-size part-size)) [brs prs] (cond is-brs (recur nu (inc brs) prs (inc i)) is-prs (recur (drop 1 nu) brs (inc prs) (inc i)) :else (recur nu brs prs (inc i))) ))))

n-ratio (loop [nn nseq brs 0 prs 0 i 0] (let [firstnn (first nn) hi ($ i :h dohlc) cl ($ i :c dohlc)] (if (> i (- dohlc-size part-size)) [brs prs] (cond (brs? firstnn hi cl < -) (recur nn (inc brs) prs (inc i)) (prs? firstnn hi cl > +) (recur (drop 1 nn) brs (inc prs) (inc i)) :else (recur nn brs prs (inc i))) ))) u-ratio (loop [uu useq brs 0 prs 0 i 0] (let [firstuu (first uu) lo ($ i :l dohlc) cl ($ i :c dohlc)] (if (> i (- dohlc-size part-size)) [brs prs] (cond (brs? firstuu lo cl > +) (recur uu (inc brs) prs (inc i)) (prs? firstuu lo cl < -) (recur (drop 1 uu) brs (inc prs) (inc i)) :else (recur uu brs prs (inc i))) ))) n-ratio (ratios "r" nseq) u-ratio (ratios "s" useq) ] [n-ratio u-ratio] ))

while the ideas here are ok we require 20s by not using partitions and 25ms when we do!! however, we need a way to figure out the index

(defn nu-finderV2 " finds nu give a dohlc dataset returns i nn] [i uu " [dohlc] (let [size (nrow dohlc) nu-breadth 7 ;nu width buf-bars (quot nu-breadth 2) ;number of bars on either side of nu ibeg buf-bars ;begining bar to look for nu iend (- (nrow dohlc) 1 buf-bars) ;ending bar to look for nu nu? (fn [i hilo maxmin] (= ($ i hilo dohlc) (apply maxmin ($ (range (- i buf-bars) (+ i buf-bars 1)) hilo dohlc)))) nus (fn [hilo maxmin] (loop [i ibeg iv []] (if (>= i iend) iv (if (nu? i hilo maxmin) (recur (+ i buf-bars 1) (conj iv [i ($ i hilo dohlc)])) (recur (inc i) iv)))))] [(nus :h max) (nus :l min)]))

(defn anulysisV1 " analyzes brs and prs price action on nu

dohlc dataset columns :d date :o open :h high :l low :c close identify nu using partition map bsr? psr? to hilo and determine how many of each " [filepath] (let [[dohlc filename pipfactor] (instrument-dohlc filepath) dohlc-size (nrow dohlc) ;size of dataset less 2 because of header buffer (/ 12.0 pipfactor) ;buffer line calculation in pips for pair part-size 7 ;size of partition chosen nn (nu-finder ($ :h dohlc) part-size max) uu (nu-finder ($ :l dohlc) part-size min) in-zone? (fn [hilo nu] (btn? hilo (- nu buffer) (+ nu buffer)))] (loop [i 4 brs 0] (if (> i (- dohlc-size 9)) brs (recur (inc i) (if (in-zone? ($ i :h dohlc) (first nn)) (inc brs) brs))) ) [(count nn) (count uu)]))

(defn mk-dohlc "creates the dataset with date, open, high, low, close from filename" [filename] (d/rename-columns (read-dataset filename) {:col0 :d :col1 :o :col2 :h :col3 :l :col4 :c}))

(defn nu-finderV1 " finds all nu in a sequence of hilo given a sequence, the partition size, max|min operator for his|los TODO have option to remove invalid nu " [aseq part-size maxmin] (let [partitions (partition part-size 1 aseq) midpt (quot part-size 2) nu (for [p partitions :when (= (apply maxmin p) (nth p midpt))] (nth p midpt))] nu))

(def truefx-filepaths "produces a list of truefx path strings" (vectorize (flatten (map (fn [x] (sort (filter #(str/includes? % x) (map #(.getPath %) (file-seq (clojure.java.io/file "/zata/truefx")))))) ["1D" "4H" "1H"]))))

(defn adjacent-confluence "determines oc confluence behavior of adjacent bars" [fp] (let [dohlc (mk-dohlc fp) fn (filename-from-filepath fp) pf (pip-factor fn) m (to-matrix (ds/remove-columns dohlc [:d])) o (sel m :cols 0) h (sel m :cols 1) l (sel m :cols 2) c (sel m :cols 3) co (map pips c o (repeat pf)) pn (map #(pos? (* %1 %2)) co (drop 1 co)) p% (Math/round (/ (count (filter true? pn)) (float (count pn)) 0.01))] (printf "%s %d %n" fn p%) p%))