[{"content":" Overview jobq is a standalone, production-inspired distributed job queue server written in Go. It provides durable job ingestion, worker leases, retries, and failure recovery backed by PostgreSQL.\nArchitecture \u0026amp; Design Decoupled System: Queue server acts as standalone infrastructure (like RabbitMQ or SQS) over HTTP; workers run as independent binaries executing business logic. Durable Persistence: PostgreSQL-backed job storage with atomic leases, SKIP LOCKED worker claims, and exponential backoff retries. Fault Recovery: Automatic lease reaper for expired or abandoned worker leases, heartbeat keep-alives, and Dead Letter Queue (DLQ). Go SDK Client: Clean client interface for job submission, lease renewal, acknowledgements (ack), and failures (nack). Progress \u0026amp; Roadmap Phase 1: Job model, REST API ingestion, SDK client Phase 2: Worker claim protocol, leases \u0026amp; reaper Phase 3: Acknowledgement, retries \u0026amp; dead letter queue Phase 4: Heartbeats \u0026amp; lease renewal ","permalink":"https://arjun118.github.io/tinkers/jobq/","summary":"\u003cascii-art piece=\"spinners\"options='{\u0026#34;names\u0026#34;:[],\u0026#34;labels\u0026#34;:true,\u0026#34;speed\u0026#34;:1}'\u003e\u003c/ascii-art\u003e\n\n\u003ch3 id=\"overview\"\u003eOverview\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003ejobq\u003c/strong\u003e is a standalone, production-inspired distributed job queue server written in Go. It provides durable job ingestion, worker leases, retries, and failure recovery backed by PostgreSQL.\u003c/p\u003e\n\u003ch3 id=\"architecture--design\"\u003eArchitecture \u0026amp; Design\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eDecoupled System\u003c/strong\u003e: Queue server acts as standalone infrastructure (like RabbitMQ or SQS) over HTTP; workers run as independent binaries executing business logic.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDurable Persistence\u003c/strong\u003e: PostgreSQL-backed job storage with atomic leases, \u003ccode\u003eSKIP LOCKED\u003c/code\u003e worker claims, and exponential backoff retries.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFault Recovery\u003c/strong\u003e: Automatic lease reaper for expired or abandoned worker leases, heartbeat keep-alives, and Dead Letter Queue (DLQ).\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eGo SDK Client\u003c/strong\u003e: Clean client interface for job submission, lease renewal, acknowledgements (\u003ccode\u003eack\u003c/code\u003e), and failures (\u003ccode\u003enack\u003c/code\u003e).\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"progress--roadmap\"\u003eProgress \u0026amp; Roadmap\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cinput checked=\"\" disabled=\"\" type=\"checkbox\"\u003e \u003cstrong\u003ePhase 1\u003c/strong\u003e: Job model, REST API ingestion, SDK client\u003c/li\u003e\n\u003cli\u003e\u003cinput checked=\"\" disabled=\"\" type=\"checkbox\"\u003e \u003cstrong\u003ePhase 2\u003c/strong\u003e: Worker claim protocol, leases \u0026amp; reaper\u003c/li\u003e\n\u003cli\u003e\u003cinput checked=\"\" disabled=\"\" type=\"checkbox\"\u003e \u003cstrong\u003ePhase 3\u003c/strong\u003e: Acknowledgement, retries \u0026amp; dead letter queue\u003c/li\u003e\n\u003cli\u003e\u003cinput checked=\"\" disabled=\"\" type=\"checkbox\"\u003e \u003cstrong\u003ePhase 4\u003c/strong\u003e: Heartbeats \u0026amp; lease renewal\u003c/li\u003e\n\u003c/ul\u003e","title":"jobq"},{"content":" Overview OCR Lab is a high-throughput local document intelligence and hybrid presentation OCR engine powered by Datalab\u0026rsquo;s Indra OCR (~1M parameter ONNX Runtime model).\nIt runs 100% locally on CPU with zero GPU requirement, unlimited batch file processing, and persistent SQLite storage.\nKey Highlights Local \u0026amp; Private: Runs entirely on local CPU with ONNX Runtime. No cloud APIs, no external telemetry, zero data leakage. Hybrid Native + Vision Extraction: Fast digital text extraction with selective vision OCR fallback for complex diagram/slide layouts. Slide \u0026amp; Presentation Intelligence: Handles presentation decks, code snippets, architecture diagrams, and multi-column tables. Batch Processing Engine: Multi-file queue with progress tracking, stage metrics, and instant preview. Persistent Storage: SQLite WAL with content-addressable hash deduplication. Architecture Model: Datalab Indra OCR (~1M params, ONNX CPU) Backend: Python 3.12+, FastAPI, SQLite WAL Deployment: Docker \u0026amp; Docker Compose ready ","permalink":"https://arjun118.github.io/tinkers/ocr-lab/","summary":"\u003cascii-art piece=\"skeleton\"options='{\u0026#34;lines\u0026#34;:3,\u0026#34;image\u0026#34;:true,\u0026#34;avatar\u0026#34;:true}'\u003e\u003c/ascii-art\u003e\n\n\u003ch3 id=\"overview\"\u003eOverview\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eOCR Lab\u003c/strong\u003e is a high-throughput local document intelligence and hybrid presentation OCR engine powered by Datalab\u0026rsquo;s \u003cstrong\u003eIndra OCR\u003c/strong\u003e (~1M parameter ONNX Runtime model).\u003c/p\u003e\n\u003cp\u003eIt runs 100% locally on CPU with zero GPU requirement, unlimited batch file processing, and persistent SQLite storage.\u003c/p\u003e\n\u003ch3 id=\"key-highlights\"\u003eKey Highlights\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eLocal \u0026amp; Private\u003c/strong\u003e: Runs entirely on local CPU with ONNX Runtime. No cloud APIs, no external telemetry, zero data leakage.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eHybrid Native + Vision Extraction\u003c/strong\u003e: Fast digital text extraction with selective vision OCR fallback for complex diagram/slide layouts.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSlide \u0026amp; Presentation Intelligence\u003c/strong\u003e: Handles presentation decks, code snippets, architecture diagrams, and multi-column tables.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eBatch Processing Engine\u003c/strong\u003e: Multi-file queue with progress tracking, stage metrics, and instant preview.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ePersistent Storage\u003c/strong\u003e: SQLite WAL with content-addressable hash deduplication.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"architecture\"\u003eArchitecture\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eModel\u003c/strong\u003e: Datalab Indra OCR (~1M params, ONNX CPU)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eBackend\u003c/strong\u003e: Python 3.12+, FastAPI, SQLite WAL\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDeployment\u003c/strong\u003e: Docker \u0026amp; Docker Compose ready\u003c/li\u003e\n\u003c/ul\u003e","title":"OCR Lab"},{"content":" in this blog, i will try to explain how i built a 3-tier search autocomplete system using the google books ngram dataset. the dataset is not an actual search queries dataset, so i will also explain how i curated it.\nwe will be building around the problem of designing top k most searched queries.\nFull Code on my github\nDesign Decisions we will only be having an english corpus, and we will use unicode code points. our dataset will contain at max 10 mil queries (no reasoning behind this, just a random number) Dataset Format every dataset will be formatted into\nphrase\\tfreq where phrase is the actual phrase and freq is the number of times that phrase is searched\ni have tried this with multiple datasets, but the versions of my dataset are roughly 2 V1 [AOL QUERIES] I have used aol_queries you can read about the dataset here - https://en.wikipedia.org/wiki/AOL_search_log_release please note that the dataset is not filtered and has nsfw words. i haven\u0026rsquo;t filtered it either, and used it as-is. please use it at your discretion one good thing about this dataset is that i don\u0026rsquo;t need to format it into phrase\\tfreq format, so we can use it directly sample records in the dataset:\ngoogle\t68513 mapquest\t37252 google.com\t29023 ebay\t28715 yahoo.com\t27669 yahoo\t26930 myspace.com\t22000 internet\t21996 map quest\t13366 myspace\t13005 there are exactly 433614 records in this dataset. this should be comfortably doable on my machine (16GB RAM)\nV2 [GOOGLE BOOKS NGRAM DATASET] now this one is a bit tricky i have downloaded all these from - https://storage.googleapis.com/books/ngrams/books/datasetsv3.html raw datasets: googlebooks-eng-all-5gram-20090715-0 googlebooks-eng-all-5gram-20090715-248 googlebooks-eng-all-5gram-20090715-374 googlebooks-eng-all-5gram-20090715-795 googlebooks-eng-fiction-all-2gram-20090715-7 googlebooks-eng-fiction-all-2gram-20090715-24 googlebooks-eng-fiction-all-2gram-20090715-46 googlebooks-eng-fiction-all-2gram-20090715-51 googlebooks-eng-fiction-all-2gram-20090715-59 googlebooks-eng-fiction-all-3gram-20090715-7 googlebooks-eng-fiction-all-3gram-20090715-56 googlebooks-eng-fiction-all-3gram-20090715-156 googlebooks-eng-fiction-all-3gram-20090715-165 googlebooks-eng-fiction-all-3gram-20090715-168 googlebooks-eng-fiction-all-4gram-20090715-125 googlebooks-eng-fiction-all-4gram-20090715-154 googlebooks-eng-fiction-all-4gram-20090715-285 googlebooks-eng-fiction-all-4gram-20090715-307 googlebooks-eng-fiction-all-4gram-20090715-340 googlebooks-eng-fiction-all-4gram-20090715-352 googlebooks-eng-us-all-5gram-20090715-178 googlebooks-eng-us-all-5gram-20090715-344 googlebooks-eng-us-all-5gram-20090715-492 they contain 2-gram, 3-gram, 4-gram, and 5-gram data i talked to deepseek and claude on how to convert these into my dataset format phrase\\tfreq you can find the dataset cleaning and creation code here - github i also have another stage, performed a single time just before my seeding to mongo (will discuss what this is in the arch section) - but it goes something like this if the start of the phrase is not a letter (unicode) or a digit, we remove that phrase - as i observed, this helped me get rid of some illegible phrases starting with parentheses, etc. the existence of these makes sense in this dataset, because the dataset is derived from the google books ngram dataset we convert everything to lowercase - meaning our implementation is not case-sensitive func normalize(s string) string { runes := []rune(strings.TrimSpace(s)) i := 0 for i \u0026lt; len(runes) { r := runes[i] if unicode.IsLetter(r) || unicode.IsDigit(r) { break } i++ } return strings.ToLower(string(runes[i:])) } let\u0026rsquo;s look at how we get to the clean canonical_sorted.txt, which is in our desired dataset format for direct seeding 1. take the file generated by `preprocess_ngrams` code.eg: `ds.txt` 2. normalize each phrase in `ds.txt` and write to a file `normalized_ds.txt` 3. sort this file lexographically using linux `sort` command 4. aggregate the final file\u0026#39;s duplicate queries and write to a final file: `canonical_sorted.txt` sample from canonical_sorted.txt:\n0 \u0026#34;\u0026#34; for the\u0026#34;\t134862 0 ( no pain\t166498 0 ) and after the\t40000 0 ) and the corresponding\t131572 0 ) can be\t783047 0 ) cannot be\t65536 0 ) does not satisfy\t57135 0 ) for the first\t108436 0 ) is equivalent to\t100856 0 ) is positive definite\t58893 we have around 9.6 mil records for our experiment\nwc -l canonical_sorted.txt # 9646759 canonical_sorted.txt Architecture I have iteratively implemented the architecture. i will split it into phases below\nPhase 1: [BASIC TRIE - V1 Dataset] In this phase, we will take the V1 dataset\n❯ wc -l aol_queries.txt # 433614 aol_queries.txt and build a trie. before going into any implementation details, let\u0026rsquo;s look at how our trie is implemented in go\ntype Reco struct { Word string Freq int64 } type Node struct { Val rune Children map[rune]*Node IsWord bool Freq int64 Top []Reco } type Trie struct { Root *Node K int CacheDepth int } this is just like a basic tree, but a node can have many children. field specifics - what we store - is below\nNode a node in our trie Val: the character this node holds (redundant, no need to store, i just kept this because it gave me clarity while debugging) Children: the children of this node IsWord: will be true if a word is formed until the node Freq: the frequency of this word/phrase from our dataset Top: we cache the top-K most occurring words at this node (using the Reco type, which has the word/phrase and its freq). this is an optimization Trie Root: also a Node with Val=\u0026quot;\u0026quot; K: for this phase 1 implementation, we initialize this trie with a number K, the number of suggestions/auto-completes to be returned per query CacheDepth: see point 6 from Node, we limit the depth past which no Top is populated at the node in the trie, so we only cache the shallow nodes. this can be set to the number x, where x is the average number of characters a user types before selecting a suggestion. this helps us save memory when all the queries are answered from an in-memory trie (or prefix-index hash map) Building the Trie func BuildTrie(ipfile string) *Trie { trie := NewTrie() file, err := os.Open(ipfile) if err != nil { panic(\u0026#34;error opening file,cannot build trie\u0026#34;) } defer file.Close() scanner := bufio.NewScanner(file) for scanner.Scan() { line := strings.TrimSpace(scanner.Text()) phrase, freq, ok := ParseLine(line) if ok { trie.Insert(phrase, freq) } } if err := scanner.Err(); err != nil { panic(fmt.Sprintf(\u0026#34;failed build trie..%s\u0026#34;, err.Error())) } return trie } we traverse our dataset line by line, parsing each line to give us a phrase and a frequency our insert is also simple func (t *Trie) Insert(s string, freq int64) { // insert and return the freq of the word node := t.Root for _, x := range s { if node.Children == nil { // if the node doesnot have children -lets initialize here node.Children = make(map[rune]*Node) } nextnode, ok := node.Children[x] if !ok { // add new nextnode = NewNode() nextnode.Val = x node.Children[x] = nextnode } node = nextnode } node.IsWord = true node.Freq = freq } with this we can easily search our trie using\nfunc (t *Trie) Search(s string) (bool, bool, int64, *Node) { // search for a string in the trie // first bool -\u0026gt; if the string exists // second bool -\u0026gt; if the string is a word or not node := t.Root for _, x := range s { nextnode, ok := node.Children[x] if !ok { return false, false, 0, nil } node = nextnode } return true, node.IsWord, node.Freq, node } you can find the whole trie implementation here: github_trie.\nPhase 2: [PREFIX INDEX HASH MAP - V1 Dataset] what we will include here is a simple hashmap. every prefix that can be formed while traversing the trie will be a key for the hashmap. the value will be what we want to return when that prefix is typed (the search query), i.e., a list of top-K words. type PrefixIndex map[string][]Reco func NewPrefixIndex() PrefixIndex { return PrefixIndex{} } that\u0026rsquo;s it, our prefix index is that simple. we can easily build the prefix index from our trie using\nfunc (t *Trie) walkForIndex(node *Node, prefix string, pi PrefixIndex) []Reco { cands := make([][]Reco, 0, len(node.Children)+1) if node.IsWord { cands = append(cands, []Reco{{Word: prefix, Freq: node.Freq}}) } for key, child := range node.Children { cands = append(cands, t.walkForIndex(child, prefix+string(key), pi)) } var top []Reco if len(cands) == 1 { top = cands[0] // already ≤ k meaning sorted } else if len(cands) \u0026gt; 1 { top = mergeTopK(cands, t.K) } pi[prefix] = top return top } notice the mergeTopK, we won\u0026rsquo;t re-calculate the top-K words for every node, instead we will perform a mergesort on the children\u0026rsquo;s top-k words\nfunc mergeTopK(lists [][]Reco, k int) []Reco { h := \u0026amp;headHeap{} heap.Init(h) for _, l := range lists { if len(l) \u0026gt; 0 { heap.Push(h, head{l, 0}) } } out := make([]Reco, 0, k) for h.Len() \u0026gt; 0 \u0026amp;\u0026amp; len(out) \u0026lt; k { t := heap.Pop(h).(head) // take the largest current head out = append(out, t.list[t.idx]) if t.idx+1 \u0026lt; len(t.list) { // that list still has elements? heap.Push(h, head{t.list, t.idx + 1}) // put the next one in the race } } return out } please find the heap definitions here: heap\nsearching and suggesting from our prefix index becomes extremely simple now. we don\u0026rsquo;t need to traverse anything to get the words to suggest, we just need to ask\nfunc (pi PrefixIndex) Suggest(prefix string) []string { recos, ok := pi[prefix] if !ok { return nil } result := make([]string, 0, len(recos)) for _, r := range recos { result = append(result, r.Word) } return result } but as our dataset becomes large, the number of prefixes we store increases very rapidly, and the performance of prefix-index might become even worse than using the optimized trie.\nwe will discuss these at length in the benchmarks section\nPhase 3: [PREFIX INDEX HASH MAP - SAMPLE OF V2 Dataset (3 mil queries)] i have worked on a sample of the V2 dataset (which has 9.6 mil records), specifically 3 mil queries, and put the trie and prefix-index hash map into memory to test the implementation. while it worked, it almost consumed all the RAM on my machine, which is understandable because of the nature of the data we store (many prefixes for a single phrase and their cache - the top-k words for each prefix)\nso instead of working on that architecture, i have moved to a 3-tier serving architecture, explained below\nPhase 4: [3-TIER SERVING] The three tiers in-memory: prefix-index hash map of the n most frequently occurring prefixes (this should ideally be the n most searched queries, by the user, if implementing at a user level) redis: an lfu (least frequently used) cache of the m most frequently occurring prefixes - this should be the most searched in a real-world case db: mongodb collection containing all the required data of our 9.6 mil queries (we will have a robust seeder to seed the db) obviously, m \u0026raquo; n. we keep a very small subset of queries in memory and a considerable number of prefixes in redis.\nMongoDB we seed our mongodb collection using our V2 dataset, similar to how we did in phases 1 and 2, where we seeded our implementation (trie / prefix-index) using the V1 dataset a record in our mongodb collection will look like [ { _id: ObjectId(\u0026#39;6a9d1523913c9ac792f99562\u0026#39;), prefix: \u0026#39;t\u0026#39;, top: [ { word: \u0026#39;to the\u0026#39;, freq: Long(\u0026#39;6720554310\u0026#39;) }, { word: \u0026#39;to his\u0026#39;, freq: Long(\u0026#39;3844455287\u0026#39;) }, { word: \u0026#39;to him\u0026#39;, freq: Long(\u0026#39;3484776154\u0026#39;) }, { word: \u0026#39;the second\u0026#39;, freq: Long(\u0026#39;2717181751\u0026#39;) }, { word: \u0026#39;the fact\u0026#39;, freq: Long(\u0026#39;2654935890\u0026#39;) }, { word: \u0026#39;to that\u0026#39;, freq: Long(\u0026#39;2565932938\u0026#39;) }, { word: \u0026#39;their\u0026#39;, freq: Long(\u0026#39;2328023435\u0026#39;) }, { word: \u0026#39;this\u0026#34;\u0026#39;, freq: Long(\u0026#39;2323063288\u0026#39;) }, { word: \u0026#39;tell you\u0026#39;, freq: Long(\u0026#39;2035448917\u0026#39;) }, { word: \u0026#39;they are\u0026#39;, freq: Long(\u0026#39;1903789632\u0026#39;) } ] } ] you can easily draw parallels between our prefix-index hash map version and how we store our records in mongodb.\nSeeder - Working Definitions our seeder is simple, recursive, and concurrent. let\u0026rsquo;s introduce 3 terms now\nbucket: we will have a bucket for a prefix bucket metrics: the metrics that define the bucket type BucketMetrics struct { Prefix string Start int64 End int64 DistinctPrefixes int64 } as you can see, we can identify which prefix a bucket belongs to using the Prefix field.\nbuild: building a prefix-index hash map for each bucket. this is what we will execute concurrently (using goroutines to build BucketMetrics for each prefix) merge: merging the prefix-index we built into mongo (why merge instead of write/insert is explained below) Why Recursive all the buckets, say all the phrases that start with a, b, c \u0026hellip; z, do not contribute a similar number of prefixes. we will have a zipf-ish distribution in our V2 dataset, look at the number of phrases that start with each character:\n0 1294 1 35060 2 15410 3 9243 4 6518 5 5534 6 4372 7 3567 8 3480 9 2906 a 1081054 b 463271 c 481223 d 315831 e 264935 f 383647 g 185792 h 498113 i 605340 j 58559 k 55729 l 263414 m 404064 n 219218 o 611757 p 396727 q 23578 r 271870 s 692986 t 1420702 u 117667 v 83098 w 570821 x 2504 y 82832 z 4643 now the distinct prefixes contributed by each bucket will definitely be more, and if we proceed to build the prefix index for that bucket - where we need to store all the distinct prefixes and their top-k words - that definitely won\u0026rsquo;t fit in my system memory.\nsolution?: split each bucket until we can fit the prefix-index hash map built for that bucket in memory\nbut how do we split, and how do we estimate memory? enter our tiny experiment to find out whether we can proceed to build a bucket or split it further\nMemory Estimation you can find the end to end code here: mem_est\nwhat we do is,\ntake a sample from the V2 dataset (here i took 2 samples - 1 mil and 3 mil records) goal: roughly estimate the bytes per prefix needed to store a prefix + its top-k words (the words here will be Word and Freq, exactly what we use in our seeder).\nthis makes life easier, because we only need to calculate the distinct prefixes found in a bucket, multiply that by bytesPerPrefix (an estimate we get from above), and check if it falls under our memory budget. if yes, build the prefix index for that bucket, else split further below are the findings\n1M-row dataset K Distinct Prefixes HeapAlloc HeapInuse Sys Bytes / Prefix Total Recos Avg Recos / Prefix 10 9,769,612 1,180.09 MB 1,370.91 MB 1,441.41 MB 126.66 12,298,420 1.26 20 9,769,612 1,207.09 MB 1,431.37 MB 1,495.03 MB 129.56 12,935,621 1.32 30 9,769,612 1,207.10 MB 1,456.12 MB 1,621.05 MB 129.56 13,300,024 1.36 40 9,769,612 1,227.30 MB 1,489.61 MB 1,612.36 MB 131.73 13,555,907 1.39 50 9,769,612 1,227.30 MB 1,369.29 MB 1,594.04 MB 131.73 13,751,263 1.41 3M-row dataset K Distinct Prefixes HeapAlloc HeapInuse Sys Bytes / Prefix Total Recos Avg Recos / Prefix 10 26,369,788 2,931.63 MB 3,585.84 MB 3,746.15 MB 116.57 33,943,880 1.29 20 26,369,788 3,013.93 MB 3,573.92 MB 3,716.96 MB 119.85 35,879,985 1.36 30 26,369,788 3,013.91 MB 3,623.60 MB 3,771.91 MB 119.85 36,991,688 1.40 40 26,369,788 3,075.59 MB 3,744.35 MB 3,900.10 MB 122.30 37,769,573 1.43 50 26,369,788 3,075.47 MB 3,838.52 MB 4,011.54 MB 122.29 38,369,444 1.46 we can roughly estimate that bytesPerPrefix will be 125-140. you can choose a number based on your machine\u0026rsquo;s config. if you want to be conservative, you can also choose 150\nnow, our bucket processing becomes increasingly simple\nfunc processBucket(f *os.File, prefix string, start int64, end int64, collection *mongo.Collection, ctx context.Context) { scanResult := Scan(f, prefix, start, end) for _, child := range scanResult.Children { estimated := float64(child.DistinctPrefixes) * bytesPerPrefix if estimated \u0026lt;= memoryBudget { indexBuildJobs \u0026lt;- child } else { fmt.Printf( \u0026#34;SPLIT %-10s prefixes=%d estimated=%.2f MB\\n\u0026#34;, child.Prefix, child.DistinctPrefixes, estimated/(1024*1024), ) processBucket( f, child.Prefix, child.Start, child.End, collection, ctx, ) } } } Concurrency we will use go-routines to build our prefix-indexes for different prefixes. but spinning up a new go-routine for every child of a prefix defeats the purpose, and we cannot run our seeder locally like that. so we will have bounded concurrency here\nchannels two buffered channels: 1 for processBucket to enqueue build jobs, and 1 for the index building workers to enqueue results\nvar ( indexBuildJobs = make(chan BucketMetrics, 5) indexResults = make(chan PI, 5) ) workers IndexBuildingWorkers consumes from indexBuildJobs and sends onto indexResults 6 workers at anytime i used const memoryBudget = 500 * MB, a memory budget of 500mb in our seeder. 6 index building workers running at any time ensure that the seeder executes comfortably, leaving headroom for the mongo process and other processes (i will detail these in a future blog) with this config, my system memory usage never exceeds 10GB (i still need to explain why it is 10gb; i will write that in a future blog as learnings, sorry :) ) func IndexBuildingWorkers(path string, wg *sync.WaitGroup) { for range 6 { // there are 6 workers at any time - working on building prefix indexes // each\u0026#39;s memory limit is capped to 500mb (done via estimated memory from the processBucket) wg.Go(func() { f, err := os.Open(path) // one handle per worker if err != nil { panic(err) } defer f.Close() for child := range indexBuildJobs { estimated := float64(child.DistinctPrefixes) * bytesPerPrefix fmt.Printf( \u0026#34;BUILD %-10s prefixes=%d estimated=%.2f MB\\n\u0026#34;, child.Prefix, child.DistinctPrefixes, estimated/(1024*1024), ) pi := buildPrefixIndex(f, child) fmt.Printf(\u0026#34;QUEUEING - MERGE %-10s prefixes=%d\\n\u0026#34;, child.Prefix, len(pi), ) indexResults \u0026lt;- pi } }) } } the buildPrefixIndex is similar to how we built our prefix-index in phase 2, but this time without a trie as an intermediate step func buildPrefixIndex(f *os.File, child BucketMetrics) PI { _, err := f.Seek(child.Start, io.SeekStart) if err != nil { panic(err) } limited := io.LimitReader(f, child.End-child.Start) scanner := bufio.NewScanner(limited) pi := make(PI) for scanner.Scan() { phrase, freq, ok := trie.ParseLine(scanner.Text()) if !ok { continue } runeList := []rune(phrase) for i := 1; i \u0026lt;= len(runeList); i++ { pref := string(runeList[:i]) if pi[pref] == nil { hp := \u0026amp;trie.RecHeap{} heap.Init(hp) pi[pref] = hp } if pi[pref].Len() \u0026lt; k { heap.Push(pi[pref], trie.Reco{Word: phrase, Freq: freq}) } else { if pi[pref].Top().Freq \u0026lt; freq { heap.Pop(pi[pref]) heap.Push(pi[pref], trie.Reco{Word: phrase, Freq: freq}) } } } } if scanner.Err() != nil { panic(scanner.Err()) } return pi } MongoMergeWorker func MongoMergeWorker(ctx context.Context, collection *mongo.Collection, done chan struct{}) { for pi := range indexResults { mergeIntoMongo(ctx, collection, pi) } close(done) } our mergeIntoMongo function merges the existing doc for the prefix (if already present in our mongodb collection) with the incoming one, preserving the top-k words. it does this in batches of batchSize=1000. code here: mergeIntoMongo Why only 1 MongoMergeWorker consider this: you are processing with 2 mongo merge workers. one worker is building the prefix-index for bucket th, and the other is building the prefix-index for bucket to.\nboth will update the document for prefix=t (they will update its top-k words). without proper locking semantics they will race, causing lost updates and race conditions. this can be solved by partitioning our mongo instance and having different mongo merge workers update different shards. i haven\u0026rsquo;t done that in this architecture, so i will include it in the next steps.\nScan given a prefix, our Scan function scans the file (our V2 dataset) to find the children bucket boundaries and returns a ScanResult which processBucket can use\ntype ScanResult struct { Children []BucketMetrics } func Scan(f *os.File, prefix string, start int64, end int64) ScanResult { _, err := f.Seek(start, io.SeekStart) if err != nil { panic(err) } limited := io.LimitReader(f, end-start) scanner := bufio.NewScanner(limited) prefixRunes := []rune(prefix) prefixLen := len(prefixRunes) var children []BucketMetrics var currentChild string var childStart int64 var childDP int64 childDP = 1 prevPhrase := \u0026#34;\u0026#34; var offset int64 for scanner.Scan() { line := scanner.Text() //advance offset lineSize := int64(len(line) + 1) offset += lineSize phrase, _, ok := trie.ParseLine(line) if !ok { continue } runes := []rune(phrase) // processing children, if have no extra text after the prefix for the current prefix, lets continue if len(runes) \u0026lt;= prefixLen { continue } // if the prefix of the current phrase doesnot equal to prefix // continue - ideally this should not happend if we have built our canonical sorted dataset correclty - safety check if prefixLen \u0026gt; 0 \u0026amp;\u0026amp; !strings.HasPrefix(phrase, prefix) { panic(fmt.Sprintf( \u0026#34;phrase %q outside prefix bucket %q\u0026#34;, phrase, prefix, )) } // child= prefix + next rune. child := string(runes[:prefixLen+1]) // new child bucket // but here the currentCHild is techncially previousChild isnt it? if child != currentChild { // complete prev child if currentChild != \u0026#34;\u0026#34; { children = append(children, BucketMetrics{ Prefix: currentChild, Start: childStart, End: start + offset - lineSize, DistinctPrefixes: childDP, }) } //new child currentChild = child // since we already increased the offset, we need to substract the lineSize from offset to get the start of the // new child childStart = start + offset - lineSize childDP = 0 // reset lcp for every child. prevPhrase = \u0026#34;\u0026#34; } // count new distinct prefixes childDP += int64( len(runes) - max(prefixLen+1, lcp(phrase, prevPhrase)), ) prevPhrase = phrase } if err := scanner.Err(); err != nil { panic(err) } //close final child if currentChild != \u0026#34;\u0026#34; { children = append(children, BucketMetrics{ Prefix: currentChild, Start: childStart, End: start + offset, DistinctPrefixes: childDP, }) } return ScanResult{ Children: children, } } Buckets Processed I have tried to visualize part of the bucket recursive splitting work using graphviz. hope the images below make sense\nonly the buckets shown in this gallery are split and processed; all the rest are processed in one go, because they fit the memory estimate check out the t bucket for a detailed split Benchmarks [will populate] - you can have a peek at: https://github.com/arjun118/system_design_implementations/blob/main/search_autocomplete/benchmarks.md\nplease note that the benchmarks are only for phases 1 and 2. i will update this section with all the phases\nRequest flow flowchart TD Start([Client requests Suggest prefix]) --\u003e CheckMem{\"1. In-Memory Cache?s.memory.Get prefix\"} %% L1 Memory Hit CheckMem -- \"Hit: ok == true\" --\u003e RetMem[\"Set source = in-memory\"] %% L2 Redis CheckMem -- \"Miss\" --\u003e CheckRedis{\"2. Redis Cache?s.redis.Get prefix\"} CheckRedis -- \"Hit: ok == true\" --\u003e RetRedis[\"Set source = cache\"] %% L3 MongoDB CheckRedis -- \"Miss\" --\u003e CheckDB[\"3. MongoDB Stores.mongo.Get prefix\"] CheckDB --\u003e DBStatus{\"Result?\"} DBStatus -- \"Error / Not Found\" --\u003e RetErr[\"Return nil\"] DBStatus -- \"Success\" --\u003e RetDB[\"Set source = db\"] %% Async write RetDB -.-\u003e Async[\"Async Goroutines.redis.Set prefix, result\"] Async -.-\u003e RedisNode[(Redis)] %% Final formatting RetMem --\u003e Format[\"Extract []string from []Reco\"] RetRedis --\u003e Format RetDB --\u003e Format Format --\u003e Done([Return suggestions, source]) classDef cache fill:#e1f5fe,stroke:#0288d1,stroke-width:1px; classDef db fill:#e8f5e9,stroke:#388e3c,stroke-width:1px; class CheckMem,CheckRedis cache; class CheckDB,DBStatus db; Demo Next Steps query logging + adaptive learning (Mongo/Redis) sharding and parallelizing seeding, and its impact on reads working on an atomic replace architecture where a worker keeps the in-memory state and cache in sync, updating every x mins or something like that ","permalink":"https://arjun118.github.io/technical/search_autocomplete/","summary":"\u003cascii-art piece=\"earthrise\"\u003e\u003c/ascii-art\u003e\n\n\u003cp\u003ein this blog, i will try to explain how i built a 3-tier search autocomplete system using the google books ngram dataset. the dataset is not an actual search queries dataset, so i will also explain how i curated it.\u003c/p\u003e\n\u003cp\u003ewe will be building around the problem of \u003ccode\u003edesigning top k most searched queries\u003c/code\u003e.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eFull Code on my \u003ca href=\"https://github.com/arjun118/system_design_implementations/tree/main/search_autocomplete\"\u003egithub\u003c/a\u003e\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch1 id=\"design-decisions\"\u003eDesign Decisions\u003c/h1\u003e\n\u003col\u003e\n\u003cli\u003ewe will only be having an english corpus, and we will use unicode code points.\u003c/li\u003e\n\u003cli\u003eour dataset will contain at max 10 mil queries (no reasoning behind this, just a random number)\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch1 id=\"dataset\"\u003eDataset\u003c/h1\u003e\n\u003ch2 id=\"format\"\u003eFormat\u003c/h2\u003e\n\u003cp\u003eevery dataset will be formatted into\u003c/p\u003e","title":"Search Autocomplete"},{"content":"we will be introducing nginx now, as a reverse proxy (acts on behalf of servers to field incoming client requests securely)\nimma keep the name of the approaches simple\nNginx Reverse Proxy Nginx Reverse Proxy With Cache X-Accel-Redirect we will be seeing an extra file, nginx.conf and some code changes in the application server.\nLet\u0026rsquo;s start then\nImplementations Nginx Reverse Proxy (branch: nginx_reverse_proxy) The Provider and the Delivery Will still remain with our minio-instance.\nIf you remember in our Public Minio storage and delivery (branch: public_minio_hls), we directly access the video from the minio\u0026rsquo;s playbackURL. eg: http://minio:9000/storage/video/...../object_key.m3u8\nOur storage bucket will be public in this impl\nIt will be the same here, but instead the client makes a request to the nginx URI path. Nginx will fetch the video requested resource for you from our minio-instance.\nlets take a look at the nginx config nginx.conf\nNginx Configuration file in: nginx/nginx.conf\nevents {} http { server{ listen 80; client_max_body_size 500M; location /api/videos { proxy_pass http://backend:8080; proxy_set_header HOST $host; proxy_set_header X-Real-IP $remote_addr; proxy_read_timeout 600s; proxy_send_timeout 600s; } location /media/ { proxy_pass http://minio:9000/; } } } Docker Compose file services: minio: image: quay.io/minio/minio container_name: minio restart: always # Expose these ports during local development/debugging. # In a typical production deployment MinIO lives on a private network # and is accessed only by trusted services (Nginx, backend, workers, etc). # The MinIO Console (9001) is almost never exposed publicly. # The S3 API port (9000) may be exposed if MinIO is being used as the # public delivery layer, but is commonly hidden behind Nginx, a CDN, # or another gateway. [comment- written by AI] # ports: # - \u0026#34;9000:9000\u0026#34; # - \u0026#34;9001:9001\u0026#34; environment: MINIO_ROOT_USER: adminpass MINIO_ROOT_PASSWORD: adminpass command: server --console-address \u0026#34;:9001\u0026#34; /data volumes: - ~/minio-data:/data nginx: image: nginx:latest ports: - \u0026#34;8080:80\u0026#34; volumes: - ./nginx/nginx.conf:/etc/nginx/nginx.conf depends_on: - backend - minio backend: build: context: . dockerfile: internal/Dockerfile depends_on: - minio any request that matches the uri path - /api/videos will be proxied by nginx to our backend - without stripping the matched prefix: 1. Request url by client: POST `http://localhost:8080/api/videos/.....` 2. Nginx matches the incoming request URI with `/api/videos/` and proxies to our backend on behalf of the client 3. Request made to our application server by nginx: POST `http://backend:8080/api/videos/....` any request that matches the uri path - /media/ will be proxied by nginx to our minio-instance- by stripping the matched prefix Request Lifecycle video bytes never touches our application server. Working run: docker compose up --build curl -H \u0026quot;Authorization:Bearer secret\u0026quot; -X POST http://localhost:8080/api/videos -F \u0026quot;file=@/path/to/you/video.mp4\u0026quot; i have implemented a simple middle to mimic authentication which i will cover in the last part when we get to cookie for video-access. notice the server as `nginx:1.31.2` from our response headers in for the request to `playlist0.ts` - reverse proxy in action Nginx Reverse Proxy Cache (branch: nginx_reverse_proxy_cache) docker files, provider, delivery - all else remain the same except for nginx.conf The only change for this implementation will be with the file nginx/nginx.conf\nhere is the configuration file\nevents {} http { log_format cachelog \u0026#39;$remote_addr \u0026#39; \u0026#39;$request_uri \u0026#39; \u0026#39;$status \u0026#39; \u0026#39;$upstream_cache_status\u0026#39;; access_log /var/log/nginx/cache.log cachelog; proxy_cache_path /var/cache/nginx/hls_cache levels=1:2 keys_zone=hls_cache:20m max_size=50g inactive=24h use_temp_path=off; server{ listen 80; client_max_body_size 500M; location / { proxy_pass http://backend:8080/; } location /api/videos { proxy_pass http://backend:8080; proxy_set_header HOST $host; proxy_set_header X-Real-IP $remote_addr; proxy_read_timeout 600s; proxy_send_timeout 600s; } location /media/ { proxy_cache hls_cache; proxy_cache_key \u0026#34;$scheme$proxy_host$request_uri\u0026#34;; proxy_ignore_headers Cache-Control Expires Set-Cookie; proxy_cache_valid 200 24h; proxy_cache_valid 206 24h; proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504; proxy_cache_background_update on; proxy_cache_lock on; proxy_cache_lock_timeout 10s; proxy_force_ranges on; add_header X-Cache-Status $upstream_cache_status always; proxy_pass http://minio:9000/; } } } Request Lifecycle when the requested object/resource results in a cache miss , nginx will proxy the request to minio-instance and fetch from the source (minio) on proxying response to the client nginx caches the response data. in the diagram , in delivery the green lines show a cache hit and red lines show a cache miss Working observe the first request log - it\u0026rsquo;s a cache MISS now when i refreshed the page, essentially requesting the same resource again (manifest and video segements) - it\u0026rsquo;s a cache HIT. hence proxy cache in action X-Accel-Redirect (branch: x_accel_redirect) Provider , Delivery the provider and delivery will remain the same Nginx Configuration events {} http { log_format cachelog \u0026#39;$remote_addr \u0026#39; \u0026#39;$request_uri \u0026#39; \u0026#39;$status \u0026#39; \u0026#39;$upstream_cache_status\u0026#39;; access_log /var/log/nginx/cache.log cachelog; error_log /var/log/nginx/error.log debug; proxy_cache_path /var/cache/nginx/hls_cache levels=1:2 keys_zone=hls_cache:20m max_size=50g inactive=24h use_temp_path=off; server{ listen 80; client_max_body_size 500M; location / { proxy_pass http://backend:8080/; } location /api/videos { proxy_pass http://backend:8080; proxy_set_header HOST $host; proxy_set_header X-Real-IP $remote_addr; proxy_read_timeout 600s; proxy_send_timeout 600s; } location /media/{ # this will be a validating/auth backend now # responds with x-accel-redirect header for nginx to know # that the request is valid and it can serve the file proxy_pass http://backend:8080; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; } location /_protected/ { internal; proxy_cache hls_cache; proxy_cache_key \u0026#34;$scheme$proxy_host$uri\u0026#34;; proxy_ignore_headers Cache-Control Expires Set-Cookie; proxy_cache_valid 200 24h; proxy_cache_valid 206 24h; proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504; proxy_cache_background_update on; proxy_cache_lock on; proxy_cache_lock_timeout 10s; proxy_force_ranges on; add_header X-Cache-Status $upstream_cache_status always; add_header X-Debug-Uri $uri always; proxy_pass http://minio:9000/; } } } Middleware we will introduce authentication middleware.\nin file: internal/middleware/auth.go\npackage middleware import ( \u0026#34;encoding/json\u0026#34; \u0026#34;maps\u0026#34; \u0026#34;net/http\u0026#34; \u0026#34;strings\u0026#34; ) func writeJson(w http.ResponseWriter, status int, data any, headers http.Header) error { js, err := json.Marshal(data) if err != nil { return err } js = append(js, \u0026#39;\\n\u0026#39;) maps.Copy(w.Header(), headers) w.Header().Set(\u0026#34;Content-Type\u0026#34;, \u0026#34;application/json\u0026#34;) w.WriteHeader(status) w.Write(js) return nil } func TokenAuthenticate(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { authHeader := r.Header.Get(\u0026#34;Authorization\u0026#34;) parts := strings.SplitN(authHeader, \u0026#34; \u0026#34;, 2) if len(parts) != 2 { http.Error(w, \u0026#34;missing auth header\u0026#34;, http.StatusUnauthorized) return } if parts[0] != \u0026#34;Bearer\u0026#34; { http.Error(w, \u0026#34;invalid auth scheme\u0026#34;, http.StatusUnauthorized) return } if parts[1] != \u0026#34;supersecret\u0026#34; { http.Error(w, \u0026#34;invalid token\u0026#34;, http.StatusUnauthorized) return } next.ServeHTTP(w, r) }) } func CookieAuthenticate(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Read token from cookie cookie, err := r.Cookie(\u0026#34;media_access\u0026#34;) if err != nil { http.Error(w, \u0026#34;missing auth cookie\u0026#34;, http.StatusUnauthorized) return } if cookie.Value != \u0026#34;supersecret\u0026#34; { http.Error(w, \u0026#34;invalid token in cookie\u0026#34;, http.StatusUnauthorized) return } next.ServeHTTP(w, r) }) } we will look at the explanation and usage of these in our server code\nServer lets change see how integrate the middleware and scoped authentication in our application server\nr := chi.NewRouter() r.Use(middleware.Logger) r.Use(middleware.Recoverer) r.Get(\u0026#34;/health\u0026#34;, func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) w.Write([]byte(\u0026#34;alive\\n\u0026#34;)) }) //our video delivery access will be a cookie-based authentication r.With(routingmiddleware.CookieAuthenticate).Handle(\u0026#34;/media/*\u0026#34;, \u0026amp;handlers.RedirectHandler{ BucketName: bucketName, }) //and our core application server routes will be token(jwt) based authentication r.Route(\u0026#34;/api/videos\u0026#34;, func(r chi.Router) { r.Use(routingmiddleware.TokenAuthenticate) r.Post(\u0026#34;/auth-cookie\u0026#34;, videoHandler.GetAuthCookie) r.Post(\u0026#34;/\u0026#34;, videoHandler.Upload) // POST /api/videos r.Get(\u0026#34;/url\u0026#34;, videoHandler.GetURL) // GET /api/videos/url?key=... r.Delete(\u0026#34;/\u0026#34;, videoHandler.Delete) // DELETE /api/videos?key=... }) server := \u0026amp;http.Server{ Addr: \u0026#34;:8080\u0026#34;, Handler: r, } in a production implementation, the flow will ideally looks like this\nThe goal of this approach is to keep media assets private while ensuring that the Go application is responsible only for authorization, not for serving video bytes.\nAuthentication Flow The client authenticates using their email and password. (we have skipped this part) The backend issues a signed JWT access token. Media Access Cookie Procuring Flow The client exchanges this token for a short-lived, scoped cookie that grants access to video content. The browser automatically sends this cookie with every HLS request (.m3u8 playlists and .ts segments). Client ↓ POST /api/videos/auth-cookie Authorization: Bearer \u0026lt;jwt\u0026gt; ↓ Go Application ↓ Set-Cookie: media_access=\u0026lt;signed-cookie\u0026gt; Media Request Flow Client ↓ GET /media/.../playlist.m3u8 ↓ CookieAuthenticate Middleware ↓ RedirectHandler ↓ X-Accel-Redirect: /_protected/videos/... ↓ Nginx Internal Redirect: http://mini:900/videos/... ↓ MinIO ↓ Nginx ↓ Client The Go application:\nValidates the scoped media cookie. Verifies that the user is authorized to access the requested resource. Returns a successful response containing an X-Accel-Redirect header pointing to the internal media location. Example:\nHTTP/1.1 200 OK X-Accel-Redirect: /_protected/videos/.../playlist.m3u8 Internal Media Delivery Nginx is configured with an internal-only location block:\nlocation /_protected/ { internal; # all the caching related stuff proxy_pass http://minio:9000/; } Because the location is marked as internal, clients cannot access it directly.\nAfter receiving the X-Accel-Redirect header, Nginx:\nPerforms an internal redirect to the protected location. Fetches the requested object from MinIO. Streams the response back to the client. Tradeoffs Media objects remain private in storage. Clients never learn the underlying MinIO object URLs. Authorization logic stays centralized in the Go application. Video bytes never flow through the application server. Nginx handles streaming, buffering, range requests, and caching efficiently. Easy migration path to signed URLs, signed cookies, or CDN-based delivery in the future. Request Lifecycle Working no video will be streamed without media_access cookie after cookie is issued, client can access the resource\nClosing That sums up all the implementations. We will look at core architectural changes in the next blogs\n","permalink":"https://arjun118.github.io/technical/building_a_vod_backend_in_golang_part_2/","summary":"\u003cp\u003ewe will be introducing nginx now, as a reverse proxy (acts on behalf of servers to field incoming client requests securely)\u003c/p\u003e\n\u003cp\u003eimma keep the name of the approaches simple\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eNginx Reverse Proxy\u003c/li\u003e\n\u003cli\u003eNginx Reverse Proxy With Cache\u003c/li\u003e\n\u003cli\u003eX-Accel-Redirect\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003ewe will be seeing an extra file, \u003ccode\u003enginx.conf\u003c/code\u003e and some code changes in the application server.\u003c/p\u003e\n\u003cp\u003eLet\u0026rsquo;s start then\u003c/p\u003e\n\u003ch1 id=\"implementations\"\u003eImplementations\u003c/h1\u003e\n\u003ch2 id=\"nginx-reverse-proxy-branch-nginx_reverse_proxy\"\u003eNginx Reverse Proxy (\u003ccode\u003ebranch: nginx_reverse_proxy\u003c/code\u003e)\u003c/h2\u003e\n\u003cp\u003eThe Provider and the Delivery Will still remain with our minio-instance.\u003c/p\u003e","title":"Building a Vod Backend in Golang Part 2"},{"content":"In this series (hope i continue to build this) we wil learn about building a scalable (eventually) video-on-demand backend in golang.\nWe will go through a few naive implementations until we reach some architecture which is actually scalable (we won\u0026rsquo;t be using any CDN here, but our final pattern will be resembling that, say you can swap out local imlementation of video delivery with cdn easily)\nCode will be available at github\nSetup this is gonna be a long section. we will set up the directory structure, define some terms which we will use to simplify the explantion\n . ├──  docker-compose.yaml ├──  go.mod ├──  go.sum ├──  internal │ ├──  handlers │ │ └──  video.go │ └──  media │ ├──  delivery │ │ ├──  cloudfront.go │ │ └──  local.go │ ├──  local │ │ └──  provider.go │ ├──  minio │ │ └──  provider.go │ ├──  provider.go │ └──  s3 │ └──  provider.go ├──  main.go ├──  service │ ├──  ffmpeg.go │ └──  video.go ├──  storage └──  vids └──  vid.mp4 main.go: routes ,spinning the server,entry point internal: our core application logic internal/handlers: api handlers for our routes internal/media: where we implement our various delivery and provider interface internal/media/provider.go: delivery, provider interface will be defined here service: video.go has convenience structs which will encapsulate the delivery and provider implementations for us to access common methods across our api handlers irrespective of the provider and delivery implementations we use.ffmpeg.go just have ffmpeg functionality storage: used to store the uploaded videos when our local is the provider vids: you don\u0026rsquo;t really need this, this is the video i will be uploading to test out the functionality. Terms Provider when i say provider i mean the underlying storage where we store our uploaded videos to the backend. we will look at 2 providers in this series\nLocal Minio Delivery when i say delivery i mean the underlying delivery where we enable our client to get access to the videos uploaded. we will look at 2 deliveries in this series\nLocal Minio Interaface Details in : internal/media/provider.go\npackage media import ( \u0026#34;context\u0026#34; \u0026#34;io\u0026#34; ) type FileMetaData struct { Filename string ContentType string Size int64 OwnerID string } type MediaObject struct { ID string `json:\u0026#34;id\u0026#34;` ObjectKey string `json:\u0026#34;key\u0026#34;` Provider string `json:\u0026#34;provider\u0026#34;` ContentType string `json:\u0026#34;content_type\u0026#34;` Size int64 `json:\u0026#34;size\u0026#34;` PlaybackURL string `json:\u0026#34;playback_url\u0026#34;` ThumbnailURL string `json:\u0026#34;thumbnail_url\u0026#34;` } type StorageProvider interface { Save(ctx context.Context, objectKey string, r io.Reader, meta FileMetaData) (int64, error) Delete(ctx context.Context, objectKey string) error } type DeliveryProvider interface { URL(ctx context.Context, objectKey string) (string, error) } our local and minio implementations for delivery and provider will satisfy these interfaces, hence abstracting complexity when used with VideoService\nVideo Service package service import ( \u0026#34;context\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;io\u0026#34; \u0026#34;log\u0026#34; \u0026#34;os\u0026#34; \u0026#34;path/filepath\u0026#34; \u0026#34;sync\u0026#34; \u0026#34;time\u0026#34; \u0026#34;github.com/arjun118/fileupload/internal/media\u0026#34; \u0026#34;github.com/google/uuid\u0026#34; ) type TranscodeJob struct { ID string TempSourcePath string FinalBasePath string Ctx context.Context } type VideoService struct { storage media.StorageProvider delivery media.DeliveryProvider jobQueue chan TranscodeJob workerWg sync.WaitGroup maxWorkers int } func NewVideoService(storage media.StorageProvider, delivery media.DeliveryProvider, maxWorkers int) *VideoService { svc := \u0026amp;VideoService{ storage: storage, delivery: delivery, //buffer size for queued jobs jobQueue: make(chan TranscodeJob, 100), maxWorkers: maxWorkers, } svc.StartWorkerPool() return svc } func (v *VideoService) StartWorkerPool() { for i := range v.maxWorkers { v.workerWg.Add(1) go func(workerID int) { defer v.workerWg.Done() log.Printf(\u0026#34;starting transcode worker %d\u0026#34;, workerID) for job := range v.jobQueue { log.Printf(\u0026#34;Worker %d starting job %s\u0026#34;, workerID, job.ID) jobCtx, cancel := context.WithTimeout(context.Background(), 10*time.Minute) err := v.processAndSaveHLS(jobCtx, job.ID, job.TempSourcePath, job.FinalBasePath) if err != nil { log.Printf(\u0026#34;[Error] Job %s failed: %v\u0026#34;, job.ID, err) //prod: update database job status - failed } else { log.Printf(\u0026#34;Job %s completed successfully\u0026#34;, job.ID) } cancel() } }(i) } } func (v *VideoService) StopWorkerPool() { close(v.jobQueue) v.workerWg.Wait() } func (v *VideoService) generateHLSFolderPath(videoID string) string { now := time.Now() year := now.Format(\u0026#34;2006\u0026#34;) month := now.Format(\u0026#34;01\u0026#34;) return filepath.ToSlash(filepath.Join(\u0026#34;videos\u0026#34;, year, month, videoID)) } func (v *VideoService) processAndSaveHLS(ctx context.Context, videoID string, tempSourcePath string, finalBasePath string) (err error) { tempHLSDir := filepath.Join(\u0026#34;/tmp\u0026#34;, \u0026#34;hls_\u0026#34;+videoID) if err := os.MkdirAll(tempHLSDir, 0755); err != nil { return fmt.Errorf(\u0026#34;failed to create HLS temp dir: %w\u0026#34;, err) } //clean the temporary folders defer func() { os.RemoveAll(tempHLSDir) os.Remove(tempSourcePath) if r := recover(); r != nil { err = fmt.Errorf(\u0026#34;panic recovered during transcoding: %v\u0026#34;, r) } }() _, err = TranscodeToHLS(ctx, tempSourcePath, tempHLSDir) if err != nil { return fmt.Errorf(\u0026#34;transcode failed: %w\u0026#34;, err) } entries, err := os.ReadDir(tempHLSDir) if err != nil { return fmt.Errorf(\u0026#34;failed to read output directory %w\u0026#34;, err) } for _, entry := range entries { if entry.IsDir() { continue } //playlist.m3u8 or playlist0.ts fileName := entry.Name() tempFilePath := filepath.Join(tempHLSDir, fileName) finalObjectKey := filepath.ToSlash(filepath.Join(finalBasePath, fileName)) if err := v.saveSegment(ctx, tempFilePath, finalObjectKey, fileName); err != nil { return fmt.Errorf(\u0026#34;failed to save segment %s: %w\u0026#34;, fileName, err) } } return nil } func (v *VideoService) saveSegment(ctx context.Context, srcPath, destKey, fileName string) error { file, err := os.Open(srcPath) if err != nil { return err } defer file.Close() contentType := \u0026#34;video/mp2t\u0026#34; if filepath.Ext(fileName) == \u0026#34;.m3u8\u0026#34; { contentType = \u0026#34;application/vnd.apple.mpegurl\u0026#34; } meta := media.FileMetaData{ Filename: fileName, ContentType: contentType, } _, err = v.storage.Save(ctx, destKey, file, meta) if err != nil { return fmt.Errorf(\u0026#34;storage save failed: %w\u0026#34;, err) } return nil } func (v *VideoService) Upload(ctx context.Context, file io.Reader, meta media.FileMetaData) (*media.MediaObject, error) { //a uuid string videoID := uuid.NewString() // save the raw file to the disk right now tempRawDir := filepath.Join(\u0026#34;/tmp\u0026#34;, \u0026#34;raw_videos\u0026#34;) if err := os.MkdirAll(tempRawDir, 0755); err != nil { return nil, err } tempSourcePath := filepath.Join(tempRawDir, videoID+\u0026#34;.mp4\u0026#34;) tempFile, err := os.Create(tempSourcePath) if err != nil { return nil, err } log.Println(\u0026#34;writing the raw file to temp file: \u0026#34;, tempSourcePath) size, err := io.Copy(tempFile, file) log.Println(\u0026#34;size of the raw file: \u0026#34;, float64(size)/float64(1024*1024)) tempFile.Close() if err != nil { os.Remove(tempSourcePath) return nil, fmt.Errorf(\u0026#34;failed to save raw file upload: %w\u0026#34;, err) } // videos/2026/06/video_uuid finalBasePath := v.generateHLSFolderPath(videoID) // videos/2026/06/video_uuid/playlist.m3u8 playlistKey := filepath.ToSlash(filepath.Join(finalBasePath, \u0026#34;playlist.m3u8\u0026#34;)) playBackURL, err := v.delivery.URL(ctx, playlistKey) if err != nil { os.Remove(tempSourcePath) return nil, fmt.Errorf(\u0026#34;failed to generate playback URL: %w\u0026#34;, err) } job := TranscodeJob{ ID: videoID, TempSourcePath: tempSourcePath, FinalBasePath: finalBasePath, Ctx: context.Background(), } select { case v.jobQueue \u0026lt;- job: log.Printf(\u0026#34;Job %s successfully queued: \u0026#34;, videoID) default: //queue is full, handle backpressure os.Remove(tempSourcePath) return nil, fmt.Errorf(\u0026#34;server queue is full, please retry again\u0026#34;) } return \u0026amp;media.MediaObject{ ID: videoID, ObjectKey: playlistKey, Provider: \u0026#34;local\u0026#34;, ContentType: \u0026#34;application/vnd.apple.mpegurl\u0026#34;, Size: size, PlaybackURL: playBackURL, }, nil } func (s *VideoService) GetPlaybackURL(ctx context.Context, objectKey string) (string, error) { // Business logic goes here: // e.g., s.db.GetVideoOwner(objectKey) -\u0026gt; check if current user is allowed to view it. url, err := s.delivery.URL(ctx, objectKey) if err != nil { return \u0026#34;\u0026#34;, fmt.Errorf(\u0026#34;failed to get playback URL for key %s: %w\u0026#34;, objectKey, err) } return url, nil } func (s *VideoService) Delete(ctx context.Context, objectKey string) error { // Business logic goes here: // e.g., check if the user requesting the delete actually owns the video. // 1. Delete from physical storage err := s.storage.Delete(ctx, objectKey) if err != nil { return fmt.Errorf(\u0026#34;failed to delete video from storage: %w\u0026#34;, err) } // 2. Future: Delete metadata from database // err = s.db.DeleteVideo(ctx, objectKey) // if err != nil { ... } return nil } we will do less of a code explanation but more of how the request flows from end to end\nStreaming and Transcoding I will eventually confuse you if i try to explain this.Please, I beg you to read this series of articles to understand video streaming, transcoding to hls segments\nthis helped me a lot in understanding streaming and hls in general\nPlease skim-through that series before continuing further\nImplementations Local Storage and Delivery (branch: main) we will save the files to uploaded to our backend will be stored to our local file system and when client requests we will deliver from our local system (network i/o of the video segments will be done our backend) Lets look at the local storage provider implementation\nProvider in: internal/media/local/provider.go\npackage local import ( \u0026#34;context\u0026#34; \u0026#34;io\u0026#34; \u0026#34;os\u0026#34; \u0026#34;path/filepath\u0026#34; \u0026#34;github.com/arjun118/fileupload/internal/media\u0026#34; ) type Storage struct { RootDir string } func NewStorage(rootDIR string) *Storage { return \u0026amp;Storage{ RootDir: rootDIR, } } func (p *Storage) Save(ctx context.Context, objectKey string, r io.Reader, meta media.FileMetaData) (int64, error) { fullPath := filepath.Join(p.RootDir, filepath.FromSlash(objectKey)) if err := os.MkdirAll(filepath.Dir(fullPath), 0o755); err != nil { return 0, err } f, err := os.Create(fullPath) if err != nil { return 0, err } defer f.Close() size, err := io.Copy(f, r) if err != nil { return 0, err } return size, nil } func (p *Storage) Delete(ctx context.Context, objectKey string) error { fullPath := filepath.Join(p.RootDir, filepath.FromSlash(objectKey)) if err := os.Remove(fullPath); err != nil \u0026amp;\u0026amp; !os.IsNotExist(err) { return err } return nil } Delivery and the delivery implementation\nin: internal/media/delivery/local.go\npackage delivery import ( \u0026#34;context\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;strings\u0026#34; ) type LocalDelivery struct { BaseURL string } func NewLocalDelivery(baseURL string) *LocalDelivery { return \u0026amp;LocalDelivery{ BaseURL: baseURL, } } func (d *LocalDelivery) URL(ctx context.Context, objectKey string) (string, error) { return d.playbackURL(objectKey), nil } func (d *LocalDelivery) playbackURL(objectKey string) string { return fmt.Sprintf(\u0026#34;%s/%s\u0026#34;, strings.TrimRight(d.BaseURL, \u0026#34;/\u0026#34;), objectKey) } Server let see how our main.go acts here, i am only posting the routes and help you understand how our fileserver helps\nserver implementation, in : main.go\nstorageDir := \u0026#34;./storage\u0026#34; storageProvider := local.NewStorage(storageDir) deliverProvider := delivery.NewLocalDelivery(\u0026#34;http://localhost:8080/media\u0026#34;) videoService := service.NewVideoService(storageProvider, deliverProvider, 3) videoHandler := handlers.NewVideoHandler(videoService) r := chi.NewRouter() r.Use(middleware.Logger) r.Use(middleware.Recoverer) r.Route(\u0026#34;/api/videos\u0026#34;, func(r chi.Router) { r.Post(\u0026#34;/\u0026#34;, videoHandler.Upload) r.Get(\u0026#34;/url\u0026#34;, videoHandler.GetURL) r.Delete(\u0026#34;/\u0026#34;, videoHandler.Delete) }) workDir, _ := os.Getwd() filesDir := http.Dir(filepath.Join(workDir, \u0026#34;storage\u0026#34;)) r.Handle(\u0026#34;/media/*\u0026#34;, http.StripPrefix(\u0026#34;/media\u0026#34;, http.FileServer(filesDir))) server := \u0026amp;http.Server{ Addr: \u0026#34;:8080\u0026#34;, Handler: r, } In local development, the LocalStorageProvider persists uploaded media to the storage/ directory on disk. We then mount that directory using http.FileServer, making it our delivery mechanism. This allows the generated playback URLs to point to /media/..., while the actual files are served directly from the filesystem without requiring custom streaming logic.\nRequest Lifecycle when client (browser or any frontend) requests for this particular file - it will do as below client request: http://localhost:8080/media/videos/2026/06/abc123/playlist.m3u8 after StripPrefix: /videos/2026/06/abc123/playlist.m3u8 (ignorning the http://\u0026hellip;.) resolved on disk: ./storage/videos/2026/06/abc123/playlist.m3u8 - where our actual hls related files are stored Tradeoffs This implementation is very naive and simple and has several limitations.\n1. Application Server Handles Everything Our Go application is responsible for:\nAccepting uploads Writing files to disk Running FFmpeg transcoding jobs Serving HLS playlists Serving HLS segments This means both CPU-intensive work (transcoding) and network-intensive work (video delivery) happen on the same machine.\nAs concurrent viewers increase, the application server starts spending more time moving video bytes than executing business logic.\n2. Video Traffic Flows Through The Backend Every HLS request:\nGET playlist.m3u8 GET segment000.ts GET segment001.ts GET segment002.ts ... hits our Go server. For a single viewer this is fine. For hundreds or thousands of viewers, our backend becomes a media relay rather than an API server.\n3. Application Restarts Become Risky The local filesystem becomes the source of truth. If:\nthe server is replaced the disk is lost a container is recreated without persistent volumes uploaded videos disappear. Object storage systems such as MinIO or S3 are designed specifically to solve this problem. i hope that the request lifecycle is clear. it wil almost remain the same in our upcoming implementations but the responsibilities of video-delivery will shift away from our golang application server.\nWorking start the server using go run main.go send a request : curl -X POST http://localhost:8080/api/videos -F \u0026quot;file=@/path/to/your-video\u0026quot; open the playback url in the browser of your choice you can see the browser handles fetching the master file and the subsequent segments for streaming Public Minio storage and delivery (branch: public_minio_hls) In this implementation we will introduce Minio\nour storage and delivery will be handeld by Minio leaving our golang application server handle the actual business logic\nProvider in: internal/media/minio/provider.go\npackage minio import ( \u0026#34;context\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;io\u0026#34; \u0026#34;log\u0026#34; \u0026#34;path/filepath\u0026#34; \u0026#34;github.com/arjun118/fileupload/internal/media\u0026#34; \u0026#34;github.com/minio/minio-go/v7\u0026#34; ) // put object // remove object // stat object type Storage struct { Client *minio.Client BucketName string } func NewStorage(client *minio.Client, bucketName string) *Storage { return \u0026amp;Storage{ Client: client, BucketName: bucketName, } } func (s *Storage) EnsureBucket(ctx context.Context) error { bucketExists, err := s.Client.BucketExists(ctx, s.BucketName) if err != nil { return fmt.Errorf(\u0026#34;failed to check bucket existence: %w\u0026#34;, err) } if !bucketExists { log.Printf(\u0026#34;bucket doesnot exists, creating bucket: %s\\n\u0026#34;, s.BucketName) err := s.Client.MakeBucket(ctx, s.BucketName, minio.MakeBucketOptions{}) if err != nil { return fmt.Errorf(\u0026#34;failed to create bucket: %w\u0026#34;, err) } } //you might want to configure cors rules for your bucket while using the backend with a frontend publicPolicy := fmt.Sprintf(`{ \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Sid\u0026#34;: \u0026#34;PublicRead\u0026#34;, \u0026#34;Principal\u0026#34;: {\u0026#34;AWS\u0026#34;: [\u0026#34;*\u0026#34;]}, \u0026#34;Action\u0026#34;: [\u0026#34;s3:GetObject\u0026#34;], \u0026#34;Resource\u0026#34;: [\u0026#34;arn:aws:s3:::%s/*\u0026#34;] } ] }`, s.BucketName) log.Println(\u0026#34;Configuring public read policy for bucket:\u0026#34;, s.BucketName) if err = s.Client.SetBucketPolicy(ctx, s.BucketName, publicPolicy); err != nil { return fmt.Errorf(\u0026#34;failed to set public policy: %w\u0026#34;, err) } return nil } func (s *Storage) Save(ctx context.Context, objectKey string, r io.Reader, meta media.FileMetaData) (int64, error) { //save this to minio bucket // videos/year/month/filename.ext filePathWithFolder := filepath.FromSlash(objectKey) var contentType string switch filepath.Ext(objectKey) { case \u0026#34;.m3u8\u0026#34;: contentType = \u0026#34;application/vnd.apple.mpegurl\u0026#34; case \u0026#34;.ts\u0026#34;: contentType = \u0026#34;video/mp2t\u0026#34; case \u0026#34;.vtt\u0026#34;: contentType = \u0026#34;text/vtt\u0026#34; case \u0026#34;.jpg\u0026#34;, \u0026#34;.jpeg\u0026#34;: contentType = \u0026#34;image/jpeg\u0026#34; case \u0026#34;.png\u0026#34;: contentType = \u0026#34;image/png\u0026#34; default: contentType = \u0026#34;application/octet-stream\u0026#34; } uploadInfo, err := s.Client.PutObject(ctx, s.BucketName, filePathWithFolder, r, -1, minio.PutObjectOptions{ ContentType: contentType, }) if err != nil { return 0, err } size := uploadInfo.Size return size, nil } func (s *Storage) Delete(ctx context.Context, objectKey string) error { //delete from bucket filePathWithFolder := filepath.FromSlash(objectKey) bucketExists, err := s.Client.BucketExists(ctx, s.BucketName) if err != nil { return err } if bucketExists { _, err := s.Client.StatObject(ctx, s.BucketName, filePathWithFolder, minio.StatObjectOptions{}) if err != nil { return err } } err = s.Client.RemoveObject(ctx, s.BucketName, filePathWithFolder, minio.RemoveObjectOptions{}) return err } our minio porvider is simple, it contains functions to ensure a particular bucket, save objects to that bucket,delete from the bucket EnsureBucket: it will create a bucket if it doesnot exist. if it exits it will be left as is. while creating the bucket we will set it to be accessible by anyone who has the playback url. i.e public. no access control is implemented. Save: we wills save the objects to our minio storage preserving their contenttype so that while minio acts as a provider it doesnot send ContentType: application/octet-stream which will break our video (browser cannot make sense of the master manifest(.m3u8) and segments(.ts) content without the correct ContentType header). setting this while saving our objects to minio directly helps us to stream the video correctly from minio Delivery in: internal/media/delivery/minio.go\ndelivery is very simple, we just put together the URL\npackage delivery import ( \u0026#34;context\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;log\u0026#34; \u0026#34;strings\u0026#34; ) type MinioDelivery struct { BucketName string EndPoint string } func NewMinioDelivery(bucketName string, endpoint string) *MinioDelivery { return \u0026amp;MinioDelivery{ BucketName: bucketName, EndPoint: endpoint, } } func (m *MinioDelivery) URL(ctx context.Context, objectKey string) (string, error) { return m.playbackURL(objectKey), nil } func (m *MinioDelivery) playbackURL(objectKey string) string { url := fmt.Sprintf(\u0026#34;http://%s/%s/%s\u0026#34;, strings.TrimRight(m.EndPoint, \u0026#34;/\u0026#34;), m.BucketName, objectKey) log.Println(\u0026#34;playback url called with objectkey: \u0026#34;, objectKey, \u0026#34;url: \u0026#34;, url) return url } Server bucketName := \u0026#34;storage\u0026#34; endpoint := \u0026#34;minio:9000\u0026#34; accessKeyID := \u0026#34;adminpass\u0026#34; secretAccessKey := \u0026#34;adminpass\u0026#34; useSSL := false minioClient, err := miniosdk.New(endpoint, \u0026amp;miniosdk.Options{ Creds: credentials.NewStaticV4(accessKeyID, secretAccessKey, \u0026#34;\u0026#34;), Secure: useSSL, }) if err != nil { log.Fatalln(err) } fmt.Println(\u0026#34;connected to minio successfully\u0026#34;) storageProvider := minio.NewStorage(minioClient, bucketName) for i := 0; i \u0026lt; 30; i++ { err = storageProvider.EnsureBucket(context.Background()) if err == nil { break } log.Printf(\u0026#34;waiting for minio (%d/30): %v\u0026#34;, i+1, err) time.Sleep(2 * time.Second) } if err != nil { log.Fatalf(\u0026#34;failed to initialize bucket: %v\u0026#34;, err) } else { log.Println(\u0026#34;ensured bucket...\u0026#34;) } deliverProvider := delivery.NewMinioDelivery(bucketName, \u0026#34;localhost:9000\u0026#34;) videoService := service.NewVideoService(storageProvider, deliverProvider, 3, \u0026#34;minio\u0026#34;) videoHandler := handlers.NewVideoHandler(videoService) r := chi.NewRouter() r.Use(middleware.Logger) r.Use(middleware.Recoverer) r.Route(\u0026#34;/api/videos\u0026#34;, func(r chi.Router) { r.Post(\u0026#34;/\u0026#34;, videoHandler.Upload) // POST /api/videos r.Get(\u0026#34;/url\u0026#34;, videoHandler.GetURL) // GET /api/videos/url?key=... r.Delete(\u0026#34;/\u0026#34;, videoHandler.Delete) // DELETE /api/videos?key=... }) server := \u0026amp;http.Server{ Addr: \u0026#34;:8080\u0026#34;, Handler: r, } notice that there is no fileserver in our main.go, that is because our application server doesnot handle the video delivery directly. it is totally handled by our minio instance the api endpoint and the playback url are different, because our minio delivery endpoint and api endpoint don\u0026rsquo;t talk to each other or not routed in anyway we will introduce docker files from now, to get our backend and minio instance up and running\ndocker manifest for our application api (backend)\nin: internal/Dockerfile\nFROM golang:1.26.1 AS builder WORKDIR /app COPY . . COPY go.mod main.go ./ RUN go mod download COPY internal ./internal COPY service ./service RUN CGO_ENABLED=0 GOOS=linux go build -o server . FROM alpine:latest WORKDIR /app RUN apk add --no-cache ffmpeg COPY --from=builder /app/server . CMD [\u0026#34;./server\u0026#34;] in: docker-compose-yaml\nservices: minio: image: quay.io/minio/minio container_name: minio restart: always ports: - \u0026#34;9000:9000\u0026#34; - \u0026#34;9001:9001\u0026#34; environment: MINIO_ROOT_USER: adminpass MINIO_ROOT_PASSWORD: adminpass command: server --console-address \u0026#34;:9001\u0026#34; /data volumes: - ~/minio-data:/data backend: build: context: . dockerfile: internal/Dockerfile depends_on: - minio ports: - \u0026#34;8080:8080\u0026#34; Request Lifecycle Tradeoffs raw videos are still stored on the application server public-minio bucket will have unrestricted access to the videos - which is not desired in most of the cases our in-memory buffered channel is not durable - the transcoding job is tightly coupled with our application logic (we will focus on this much later in this series). i will not highlight this as a con in the upcoming implementation because this will stay the same for all our implementation until we get to our actual architectural changes, which we will do. transcoding is still coupled with our application logic and our application server, while network-io video delivery is completely offloaded to our minio instance. Working 1.1. start the app using docker compose up --build 2. send a request : curl -X POST http://localhost:8080/api/videos -F \u0026quot;file=@/path/to/your-video\u0026quot; 3. open the playback url in the browser of your choice 4. notice that the playback url will be different from that of our application url (port is different) because the delivery is entirely handled by minio.eg: http://localhost:9000/storage/videos/2026/06/0a1ba2c8-7e89-43e2-9edc-1f43d2df6fd2/playlist.m3u8 5. you can see the browser handles fetching the mainfest file and the subsequent segments for streaming\nPrivate Minio Storage - Go Delivery (branch: private_go_streaming) Provider the provider will remain the exact same as our previous public implementation but the only will be that we won\u0026rsquo;t change our bucket to be public. we leave it as it is - default: private\nthe change is only in the Save method\nfunc (s *Storage) Save(ctx context.Context, objectKey string, r io.Reader, meta media.FileMetaData) (int64, error) { //save this to minio bucket // videos/year/month/filename.ext filePathWithFolder := filepath.FromSlash(objectKey) var contentType string switch filepath.Ext(objectKey) { case \u0026#34;.m3u8\u0026#34;: contentType = \u0026#34;application/vnd.apple.mpegurl\u0026#34; case \u0026#34;.ts\u0026#34;: contentType = \u0026#34;video/mp2t\u0026#34; case \u0026#34;.vtt\u0026#34;: contentType = \u0026#34;text/vtt\u0026#34; case \u0026#34;.jpg\u0026#34;, \u0026#34;.jpeg\u0026#34;: contentType = \u0026#34;image/jpeg\u0026#34; case \u0026#34;.png\u0026#34;: contentType = \u0026#34;image/png\u0026#34; default: contentType = \u0026#34;application/octet-stream\u0026#34; } uploadInfo, err := s.Client.PutObject(ctx, s.BucketName, filePathWithFolder, r, -1, minio.PutObjectOptions{ ContentType: contentType, }) if err != nil { return 0, err } size := uploadInfo.Size return size, nil } Delivery our minio delivery will also remain the same.\nServer this is where the major change will be seen.\nwe will be essentially be routing the requests for playback through our application server.\nbucketName := \u0026#34;storage\u0026#34; endpoint := \u0026#34;minio:9000\u0026#34; accessKeyID := \u0026#34;adminpass\u0026#34; secretAccessKey := \u0026#34;adminpass\u0026#34; useSSL := false minioClient, err := miniosdk.New(endpoint, \u0026amp;miniosdk.Options{ Creds: credentials.NewStaticV4(accessKeyID, secretAccessKey, \u0026#34;\u0026#34;), Secure: useSSL, }) if err != nil { log.Fatalln(err) } storageProvider := minio.NewStorage(minioClient, bucketName) for i := 0; i \u0026lt; 30; i++ { err = storageProvider.EnsureBucket(context.Background()) if err == nil { break } log.Printf(\u0026#34;waiting for minio (%d/30): %v\u0026#34;, i+1, err) time.Sleep(2 * time.Second) } if err != nil { log.Fatalf(\u0026#34;failed to initialize bucket: %v\u0026#34;, err) } else { log.Println(\u0026#34;ensured bucket...\u0026#34;) } deliverProvider := delivery.NewMinioDelivery(bucketName, \u0026#34;localhost:8080/media\u0026#34;) videoService := service.NewVideoService(storageProvider, deliverProvider, 3, \u0026#34;minio\u0026#34;) videoHandler := handlers.NewVideoHandler(videoService) r := chi.NewRouter() r.Use(middleware.Logger) r.Use(middleware.Recoverer) r.Route(\u0026#34;/api/videos\u0026#34;, func(r chi.Router) { r.Post(\u0026#34;/\u0026#34;, videoHandler.Upload) // POST /api/videos r.Get(\u0026#34;/url\u0026#34;, videoHandler.GetURL) // GET /api/videos/url?key=... r.Delete(\u0026#34;/\u0026#34;, videoHandler.Delete) // DELETE /api/videos?key=... }) r.Handle(\u0026#34;/media/*\u0026#34;, \u0026amp;handlers.ProxyHandler{ MinioClient: minioClient, BucketName: bucketName, }) server := \u0026amp;http.Server{ Addr: \u0026#34;:8080\u0026#34;, Handler: r, } look at the ProxyHandler. this is our handler which handles the media delivery requests. lets look at the ServeHTTP method on our handler\nin: internal/handlers/proxy.go\npackage handlers import ( \u0026#34;io\u0026#34; \u0026#34;net/http\u0026#34; \u0026#34;path/filepath\u0026#34; \u0026#34;strconv\u0026#34; \u0026#34;strings\u0026#34; \u0026#34;github.com/minio/minio-go/v7\u0026#34; ) //cache control to be implemented type ProxyHandler struct { MinioClient *minio.Client BucketName string } func (p *ProxyHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { if r.Method != http.MethodGet \u0026amp;\u0026amp; r.Method != http.MethodHead { http.Error( w, \u0026#34;method not allowed\u0026#34;, http.StatusMethodNotAllowed, ) return } objectKey := strings.TrimPrefix(r.URL.Path, \u0026#34;/media/\u0026#34;) if objectKey == \u0026#34;\u0026#34; { http.Error(w, \u0026#34;missing object key\u0026#34;, http.StatusBadRequest) return } obj, err := p.MinioClient.GetObject(r.Context(), p.BucketName, objectKey, minio.GetObjectOptions{}) if err != nil { http.Error(w, \u0026#34;file not found\u0026#34;, http.StatusNotFound) return } defer obj.Close() info, err := obj.Stat() if err != nil { http.Error(w, \u0026#34;not found\u0026#34;, http.StatusNotFound) return } contentType := info.ContentType if contentType == \u0026#34;\u0026#34; { switch filepath.Ext(objectKey) { case \u0026#34;.m3u8\u0026#34;: contentType = \u0026#34;application/vnd.apple.mpegurl\u0026#34; case \u0026#34;.ts\u0026#34;: contentType = \u0026#34;video/mp2t\u0026#34; default: contentType = \u0026#34;application/octet-stream\u0026#34; } } //segment not modified if r.Header.Get(\u0026#34;If-None-Match\u0026#34;) == info.ETag { w.WriteHeader(http.StatusNotModified) return } w.Header().Set(\u0026#34;Content-Type\u0026#34;, contentType) w.Header().Set(\u0026#34;Content-Length\u0026#34;, strconv.FormatInt(info.Size, 10)) w.Header().Set(\u0026#34;ETag\u0026#34;, info.ETag) w.Header().Set(\u0026#34;Last-Modified\u0026#34;, info.LastModified.UTC().Format(http.TimeFormat)) if r.Method == http.MethodHead { w.WriteHeader(http.StatusOK) return } _, _ = io.Copy(w, obj) } i will explain how the request the video segments will be sent to the client when requested - through our application server - refer to request lifecycle diagram below\nRequest Lifecycle Tradeoffs while the minio bucket is private now, the go proxy handles our video delivery. request for every video segment goes through our application server - hence , the network i/o overhead of video delivery has came back. Working observe the playback url is : http://localhost:8080/media/videos/2026/06/6b40bb6d-1782-4298-b5e3-ee2f95057f48/playlist.m3u8 Closing we will see our nginx implementations in the part-2 of this series and part-3 will contain architectural improvements to make our backend scalable and seperate responsibilities (low coupling)\n","permalink":"https://arjun118.github.io/technical/building_a_vod_backend_in_golang_part_1/","summary":"\u003cp\u003eIn this series (hope i continue to build this) we wil learn about building a scalable (eventually) video-on-demand backend in golang.\u003c/p\u003e\n\u003cp\u003eWe will go through a few naive implementations until we reach some architecture which is actually scalable (we won\u0026rsquo;t be using any CDN here, but our final pattern will be resembling that, say you can swap out local imlementation of video delivery with cdn easily)\u003c/p\u003e\n\u003cp\u003eCode will be available at \u003ca href=\"https://github.com/arjun118/vod\"\u003egithub\u003c/a\u003e\u003c/p\u003e","title":"Building a Vod Backend in Golang - Part 1"},{"content":"Let\u0026rsquo;s look at some sampling methods used to control the randomness of an LLM\u0026rsquo;s text generation.\nBackground LLMs by nature are auto-regressive. Meaning the future values (tokens) are predicted by learning from its own past values\nnext step in the sequence depends directly on the steps that came before it\neach new word is generated based on the context of all the words that were generated previously\nYou can control this exact behaviour, how randomly you want your LLM to generate the next token\nLets draw an analogy. Lets say you and I are speaking and it\u0026rsquo;s my turn to talk and i start talking. to simplify my thinking process, lets consider that the next word that comes out of my mouth depends on all the previous words that came out of my mouth at the start of my turn.\nnow i would pay attention to all the previous words i uttered and speak the next word.\nIf I am not in the mood to talk to you , i will make small talk to end my sentences quickly and you could predict my response easily\nIf i enjoy talking to you, i pull up all my creativity to make my sentence more interesting.\nNow remember, any word that comes out of my mouth is just from the dictionary of bag of words that i know until that point of time,also called my vocabulary\nWhen i am speaking my next word i can say ,\ni am sampling my next word from my vocabulary basis all my previous words spoken until that point\nNow the same way, LLMs also have their vocabulary: essentially a bunch of words. If you prompt them, they will also sample their next word from their vocabulary (obviously, unless your tokenizer is broken)\nthe way you speak depends upon your mood, the same way the LLM responds depends upon a bunch of numbers which you can change anytime you want\nLets look at some of those sampling methods. Below is a list of methods we are going to discuss\nGreedy Temperature Scaling Top-k Top-P min-P Setup For some hands-on we are gonna use : refer to awesome-chatper-4 and awesome-chapter-5 and weight-loading-from-pytorch-state-dicts\nBelow is the whole setup\nimport torch import torch.nn as nn import tiktoken import numpy as np class LayerNorm(nn.Module): def __init__(self, emb_dim): super().__init__() self.eps = 1e-5 self.scale = nn.Parameter(torch.ones(emb_dim)) self.shift = nn.Parameter(torch.zeros(emb_dim)) def forward(self, x): mean = x.mean(dim=-1, keepdim=True) var = x.var(dim=-1, keepdim=True, unbiased=False) norm_x = (x - mean) / torch.sqrt(var + self.eps) return self.scale * norm_x + self.shift class GELU(nn.Module): def forward(self, x): return 0.5 * x * ( 1 + torch.tanh( torch.sqrt(torch.tensor(2.0 / torch.pi)) * (x + 0.044715 * torch.pow(x, 3)) ) ) class FeedForward(nn.Module): def __init__(self, cfg): super().__init__() self.layers = nn.Sequential( nn.Linear(cfg[\u0026#34;emb_dim\u0026#34;], 4 * cfg[\u0026#34;emb_dim\u0026#34;]), GELU(), nn.Linear(4 * cfg[\u0026#34;emb_dim\u0026#34;], cfg[\u0026#34;emb_dim\u0026#34;]), ) def forward(self, x): return self.layers(x) class MultiHeadAttention(nn.Module): def __init__( self, d_in, d_out, context_length, dropout, num_heads, qkv_bias=False, ): super().__init__() assert d_out % num_heads == 0 self.d_out = d_out self.num_heads = num_heads self.head_dim = d_out // num_heads self.W_query = nn.Linear(d_in, d_out, bias=qkv_bias) self.W_key = nn.Linear(d_in, d_out, bias=qkv_bias) self.W_value = nn.Linear(d_in, d_out, bias=qkv_bias) self.out_proj = nn.Linear(d_out, d_out) self.dropout = nn.Dropout(dropout) self.register_buffer( \u0026#34;mask\u0026#34;, torch.triu( torch.ones(context_length, context_length), diagonal=1 ) ) def forward(self, x,attention_mask=None,past_k=None,past_v=None,use_cache=False): b, num_tokens, d_in = x.shape keys = self.W_key(x) queries = self.W_query(x) values = self.W_value(x) keys = keys.view( b, num_tokens, self.num_heads, self.head_dim ) values = values.view( b, num_tokens, self.num_heads, self.head_dim ) queries = queries.view( b, num_tokens, self.num_heads, self.head_dim ) keys = keys.transpose(1, 2) queries = queries.transpose(1, 2) values = values.transpose(1, 2) attn_scores = queries @ keys.transpose(2, 3) mask_bool = self.mask.bool()[:num_tokens, :num_tokens] # (batch, n_heads, seq_len,seq_len) attn_scores.masked_fill_(mask_bool, -torch.inf) if attention_mask is not None: # shape - (batch,seq_len) # expand to (batch,1,1,seq_len) padding_mask=(attention_mask.unsqueeze(1).unsqueeze(2)) attn_scores=attn_scores.masked_fill( padding_mask==0, -torch.inf ) attn_weights = torch.softmax( attn_scores / keys.shape[-1]**0.5, dim=-1 ) attn_weights = self.dropout(attn_weights) context_vec = (attn_weights @ values).transpose(1, 2) context_vec = context_vec.contiguous().view( b, num_tokens, self.d_out ) context_vec = self.out_proj(context_vec) return context_vec class TransformerBlock(nn.Module): def __init__(self, cfg): super().__init__() self.att = MultiHeadAttention( d_in=cfg[\u0026#34;emb_dim\u0026#34;], d_out=cfg[\u0026#34;emb_dim\u0026#34;], context_length=cfg[\u0026#34;context_length\u0026#34;], num_heads=cfg[\u0026#34;n_heads\u0026#34;], dropout=cfg[\u0026#34;drop_rate\u0026#34;], qkv_bias=cfg[\u0026#34;qkv_bias\u0026#34;], ) self.ff = FeedForward(cfg) self.norm1 = LayerNorm(cfg[\u0026#34;emb_dim\u0026#34;]) self.norm2 = LayerNorm(cfg[\u0026#34;emb_dim\u0026#34;]) self.drop_shortcut = nn.Dropout(cfg[\u0026#34;drop_rate\u0026#34;]) def forward(self, x,attention_mask=None): shortcut = x x = self.norm1(x) x = self.att(x,attention_mask=attention_mask) x = self.drop_shortcut(x) x = x + shortcut shortcut = x x = self.norm2(x) x = self.ff(x) x = self.drop_shortcut(x) x = x + shortcut return x class GPTModel(nn.Module): def __init__(self, cfg): super().__init__() self.tok_emb = nn.Embedding( cfg[\u0026#34;vocab_size\u0026#34;], cfg[\u0026#34;emb_dim\u0026#34;] ) self.pos_emb = nn.Embedding( cfg[\u0026#34;context_length\u0026#34;], cfg[\u0026#34;emb_dim\u0026#34;] ) self.drop_emb = nn.Dropout(cfg[\u0026#34;drop_rate\u0026#34;]) self.trf_blocks = nn.Sequential( *[TransformerBlock(cfg) for _ in range(cfg[\u0026#34;n_layers\u0026#34;])] ) self.final_norm = LayerNorm(cfg[\u0026#34;emb_dim\u0026#34;]) self.out_head = nn.Linear( cfg[\u0026#34;emb_dim\u0026#34;], cfg[\u0026#34;vocab_size\u0026#34;], bias=False ) def forward(self, in_idx,attention_mask=None): batch_size, seq_len = in_idx.shape tok_embeds = self.tok_emb(in_idx) pos_embeds = self.pos_emb( torch.arange(seq_len, device=in_idx.device) ) x = tok_embeds + pos_embeds x = self.drop_emb(x) for block in self.trf_blocks: x=block(x,attention_mask=attention_mask) x = self.final_norm(x) logits = self.out_head(x) return logits # GPT2 config GPT2_SMALL_124M = { \u0026#34;vocab_size\u0026#34;: 50257, \u0026#34;context_length\u0026#34;: 1024, \u0026#34;emb_dim\u0026#34;: 768, \u0026#34;n_heads\u0026#34;: 12, \u0026#34;n_layers\u0026#34;: 12, \u0026#34;drop_rate\u0026#34;: 0.0, \u0026#34;qkv_bias\u0026#34;: True, } GPT2_MEDIUM_355M = { \u0026#34;vocab_size\u0026#34;: 50257, \u0026#34;context_length\u0026#34;: 1024, \u0026#34;emb_dim\u0026#34;: 1024, \u0026#34;n_heads\u0026#34;: 16, \u0026#34;n_layers\u0026#34;: 24, \u0026#34;drop_rate\u0026#34;: 0.0, \u0026#34;qkv_bias\u0026#34;: True, } GPT2_LARGE_774M = { \u0026#34;vocab_size\u0026#34;: 50257, \u0026#34;context_length\u0026#34;: 1024, \u0026#34;emb_dim\u0026#34;: 1280, \u0026#34;n_heads\u0026#34;: 20, \u0026#34;n_layers\u0026#34;: 36, \u0026#34;drop_rate\u0026#34;: 0.0, \u0026#34;qkv_bias\u0026#34;: True, } GPT2_XL_1558M = { \u0026#34;vocab_size\u0026#34;: 50257, \u0026#34;context_length\u0026#34;: 1024, \u0026#34;emb_dim\u0026#34;: 1600, \u0026#34;n_heads\u0026#34;: 25, \u0026#34;n_layers\u0026#34;: 48, \u0026#34;drop_rate\u0026#34;: 0.0, \u0026#34;qkv_bias\u0026#34;: True, } BASE_CONFIG = { \u0026#34;vocab_size\u0026#34;: 50257, # Vocabulary size \u0026#34;context_length\u0026#34;: 1024, # Context length \u0026#34;drop_rate\u0026#34;: 0.0, # Dropout rate \u0026#34;qkv_bias\u0026#34;: True # Query-key-value bias } model_configs = { \u0026#34;gpt2-small (124M)\u0026#34;: {\u0026#34;emb_dim\u0026#34;: 768, \u0026#34;n_layers\u0026#34;: 12, \u0026#34;n_heads\u0026#34;: 12}, \u0026#34;gpt2-medium (355M)\u0026#34;: {\u0026#34;emb_dim\u0026#34;: 1024, \u0026#34;n_layers\u0026#34;: 24, \u0026#34;n_heads\u0026#34;: 16}, \u0026#34;gpt2-large (774M)\u0026#34;: {\u0026#34;emb_dim\u0026#34;: 1280, \u0026#34;n_layers\u0026#34;: 36, \u0026#34;n_heads\u0026#34;: 20}, \u0026#34;gpt2-xl (1558M)\u0026#34;: {\u0026#34;emb_dim\u0026#34;: 1600, \u0026#34;n_layers\u0026#34;: 48, \u0026#34;n_heads\u0026#34;: 25}, } CHOOSE_MODEL = \u0026#34;gpt2-small (124M)\u0026#34; BASE_CONFIG.update(model_configs[CHOOSE_MODEL]) file_name = \u0026#34;gpt2-small-124M.pth\u0026#34; import os import requests url = f\u0026#34;https://huggingface.co/rasbt/gpt2-from-scratch-pytorch/resolve/main/{file_name}\u0026#34; if not os.path.exists(file_name): response = requests.get(url, timeout=60) response.raise_for_status() with open(file_name, \u0026#34;wb\u0026#34;) as f: f.write(response.content) print(f\u0026#34;Downloaded to {file_name}\u0026#34;) # load weights gpt = GPTModel(BASE_CONFIG) gpt.load_state_dict(torch.load(file_name, weights_only=True)) gpt.eval() device = torch.device(\u0026#34;cuda\u0026#34; if torch.cuda.is_available() else \u0026#34;cpu\u0026#34;) gpt.to(device); gpt import tiktoken torch.manual_seed(123) tokenizer = tiktoken.get_encoding(\u0026#34;gpt2\u0026#34;) import torch PAD_TOKEN_ID=0 def texts_to_token_ids(texts, tokenizer): encoded = [ tokenizer.encode(text) for text in texts ] max_len = max(len(seq) for seq in encoded) padded=[] attention_masks=[] for seq in encoded: pad_len=max_len-len(seq) padded_seq=[PAD_TOKEN_ID]*pad_len+seq mask=[0]*pad_len+[1]*len(seq) padded.append(padded_seq) attention_masks.append(mask) return ( torch.tensor(padded), torch.tensor(attention_masks) ) def token_ids_to_texts(token_ids, tokenizer): return [ tokenizer.decode(seq.tolist()) for seq in token_ids ] i have made some changes to token to texts and text to tokens to add padding and all, but that is still wip\nyou can just use those from chapter-4 and chapter-5 from Build a Large Language Model From Scratch book by Sebastian Raschka\nSampling Methods Greedy this is the simplest, most deterministic approach (boringggg)\nhere ,there is less of sampling happening and more of choose the most likely answer\nWorking model scans the probability distribution and strictly selects the token with the highest probability Features Sounds robotically logical, highly predictable fast, computationally cheap,and excellent for tasks requiring strict factual accuracy Code def generate(model, idx,attention_mask,max_new_tokens, context_size, eos_id=None): batch_size=idx.shape[0] finished=torch.zeros( batch_size, dtype=torch.bool, device=idx.device ) for _ in range(max_new_tokens): active_mask=~finished active_indices=torch.nonzero(active_mask,as_tuple=False).squeeze(-1) if len(active_indices)==0: break active_idx=idx[active_indices] active_attention_mask=attention_mask[active_indices] idx_cond=active_idx[:,-context_size:] attn_cond=active_attention_mask[:, -context_size:] with torch.no_grad(): logits=model(idx_cond,attention_mask=attn_cond) # only select the last token - newly generated token logits=logits[:,-1,:] idx_next=torch.argmax(logits,dim=-1,keepdim=True) new_tokens=torch.full( (batch_size,1), eos_id if eos_id is not None else 0, dtype=idx.dtype, device=idx.device ) new_tokens[active_indices]=idx_next idx=torch.cat((idx,new_tokens),dim=1) new_attention=torch.zeros( (batch_size,1), dtype=attention_mask.dtype, device=attention_mask.device ) new_attention[active_indices]=1 attention_mask=torch.cat((attention_mask,new_attention),dim=1) if eos_id is not None: newly_finished = ( idx_next.squeeze(-1) == eos_id ) finished[active_indices]|=newly_finished if finished.all(): break return idx pay attention to this line:\nidx_next=torch.argmax(logits,dim=-1,keepdim=True)\nthat is the whole sampling in the greedy method\ni have made some changes to include early stopping and including attention masks, different from what that is given in sebastian\u0026rsquo;s github. but yeah you can ignore them, not needed for now\nlet\u0026rsquo;s generate some text, the greedy way\ninput_ids, attention_mask = texts_to_token_ids( [ \u0026#34;Every effort moves\u0026#34;, \u0026#34;practice makes man\u0026#34; ], tokenizer ) input_ids = input_ids.to(device) attention_mask = attention_mask.to(device) token_ids = generate( model=gpt, idx=input_ids, attention_mask=attention_mask, max_new_tokens=30, context_size=BASE_CONFIG[\u0026#34;context_length\u0026#34;], eos_id=tokenizer.eot_token ) print(\u0026#34;generated output: \\n\u0026#34;) for i,out in enumerate(token_ids_to_texts(token_ids,tokenizer)): print(f\u0026#34;\u0026#34;\u0026#34;{i+1}. \\n{out}\\n\u0026#34;\u0026#34;\u0026#34;) generated output: 1. Every effort moves forward, but it\u0026#39;s not enough. \u0026#34;I\u0026#39;m not going to sit here and say, \u0026#39;I\u0026#39;m not going to do this,\u0026#39; 2. practice makes man\u0026#39;s life easier. The first step is to understand the concept of \u0026#34;self-awareness.\u0026#34; Self-awareness is the ability to recognize Temperature Scaling this does not actually select the token, as greedy does it changes the shape of the probability distribution before we select the token parameter: T will be used in our softmax equation Working if T=1.0 -\u0026gt; its the default state - probabilities remain unchanged T\u0026lt;1.0 -\u0026gt; makes distribution sharper - amplifies the differences between the logits rich gets richer, and top choice dominates approaches greedy as T nears 0 T\u0026gt;1.0 - flattens the distribution, the math minimizes the differences boosting the probabilites of obscure , low-ranked tokens Features this is bascially the model\u0026rsquo;s creativity or hallucination dial gives granual control over the predictability of the output High temperatures (e.g., T \u0026gt; 1.2) will often cause the model to output complete gibberish by assigning too much probability to nonsensical tokens. Standard sampling with temperature still leaves a tiny probability that the model might pick the absolute worst token in the vocabulary Code def generate(model, idx, attention_mask,max_new_tokens, context_size,temperature=0.0, eos_id=None): batch_size=idx.shape[0] finished=torch.zeros( batch_size, dtype=torch.bool, device=idx.device ) for _ in range(max_new_tokens): active_mask=~finished active_indices=torch.nonzero(active_mask,as_tuple=False).squeeze(-1) if len(active_indices)==0: break active_idx=idx[active_indices] active_attention_mask=attention_mask[active_indices] idx_cond=active_idx[:,-context_size:] attn_cond=active_attention_mask[:, -context_size:] with torch.no_grad(): logits=model(idx_cond,attention_mask=attn_cond) # only select the last token - newly generated token logits=logits[:,-1,:] ###### code block change ###### if temperature\u0026gt;0.0: logits=logits/temperature logits=logits-logits.max(dim=-1,keepdim=True).values probs=torch.softmax(logits,dim=-1) idx_next=torch.multinomial(probs,num_samples=1) else: # it temp is 0. its same as greedy idx_next=torch.argmax(logits,dim=-1,keepdim=True) ###### code block change ###### new_tokens=torch.full( (batch_size,1), eos_id if eos_id is not None else 0, dtype=idx.dtype, device=idx.device ) new_tokens[active_indices]=idx_next idx=torch.cat((idx,new_tokens),dim=1) new_attention=torch.zeros( (batch_size,1), dtype=attention_mask.dtype, device=attention_mask.device ) new_attention[active_indices]=1 attention_mask=torch.cat((attention_mask,new_attention),dim=1) if eos_id is not None: newly_finished = ( idx_next.squeeze(-1) == eos_id ) finished[active_indices]|=newly_finished if finished.all(): break return idx focus on\nif temperature\u0026gt;0.0: logits=logits/temperature logits=logits-logits.max(dim=-1,keepdim=True).values probs=torch.softmax(logits,dim=-1) idx_next=torch.multinomial(probs,num_samples=1) we scale the logits and substract the max (for softmax stability) and calculate the probabilities by applying softmax\nusage:\ninput_ids, attention_mask = texts_to_token_ids( [ \u0026#34;Every effort moves\u0026#34;, \u0026#34;practice makes man\u0026#34; ], tokenizer ) input_ids = input_ids.to(device) attention_mask = attention_mask.to(device) token_ids = generate( model=gpt, idx=input_ids, attention_mask=attention_mask, max_new_tokens=30, context_size=BASE_CONFIG[\u0026#34;context_length\u0026#34;], temperature=0.5, eos_id=tokenizer.eot_token ) print(\u0026#34;generated output: \\n\u0026#34;) for i,out in enumerate(token_ids_to_texts(token_ids,tokenizer)): print(f\u0026#34;\u0026#34;\u0026#34;{i+1}. \\n{out}\\n\u0026#34;\u0026#34;\u0026#34;) generated output: 1. Every effort moves forward, but a little time needs to be made to get it done. The project is currently in its early stages, and has been a 2. practice makes man-made disasters much more likely. But if you want to know how to prevent a disaster, you need to know how to do it right Top-K Sampling you set a number K the model sorts its output (basically a probability for each of the token) probabilities and throws away everything from K+1 rank downwards redistributes the remaining probability mass among the top K tokens. It then samples from that truncated list. Features controlled randomness it eliminates the long tail of garbage tokens by preventing model from suddenly going completely off the rails - it does so by being very rigid with the prob cutoff if the model is highly confident (one token as 90% prob), the top-k still forces the model to consider the next K-1 terrible options if the model is deeply uncertain , low K might prematurely cutoff perfectly valid words Code def generate(model, idx, attention_mask,max_new_tokens, context_size,top_k=None,temperature=0.0, eos_id=None): batch_size=idx.shape[0] finished=torch.zeros( batch_size, dtype=torch.bool, device=idx.device ) for _ in range(max_new_tokens): active_mask=~finished active_indices=torch.nonzero(active_mask,as_tuple=False).squeeze(-1) if active_indices.numel()==0: break active_idx=idx[active_indices] active_attention_mask=attention_mask[active_indices] idx_cond=active_idx[:,-context_size:] attn_cond=active_attention_mask[:, -context_size:] with torch.no_grad(): logits=model(idx_cond,attention_mask=attn_cond) # only select the last token - newly generated token logits=logits[:,-1,:] ###### code block change ###### if temperature\u0026gt;0.0: logits=logits/temperature if top_k is not None: top_logits,_=torch.topk(logits,top_k) min_val=top_logits[:,-1].unsqueeze(-1) logits=torch.where(logits\u0026lt;min_val,torch.tensor(float(\u0026#34;-inf\u0026#34;)).to(logits.device),logits) ###### code block change ###### if temperature\u0026gt;0.0: logits=logits-logits.max(dim=-1,keepdim=True).values probs=torch.softmax(logits,dim=-1) idx_next=torch.multinomial(probs,num_samples=1) else: # it temp is 0. its same as greedy idx_next=torch.argmax(logits,dim=-1,keepdim=True) new_tokens=torch.full( (batch_size,1), eos_id if eos_id is not None else 0, dtype=idx.dtype, device=idx.device ) new_tokens[active_indices]=idx_next idx=torch.cat((idx,new_tokens),dim=1) new_attention=torch.zeros( (batch_size,1), dtype=attention_mask.dtype, device=attention_mask.device ) new_attention[active_indices]=1 attention_mask=torch.cat((attention_mask,new_attention),dim=1) if eos_id is not None: newly_finished = ( idx_next.squeeze(-1) == eos_id ) finished[active_indices]|=newly_finished if finished.all(): break return idx usage:\ninput_ids, attention_mask = texts_to_token_ids( [ \u0026#34;Every effort moves\u0026#34;, \u0026#34;practice makes man\u0026#34; ], tokenizer ) input_ids = input_ids.to(device) attention_mask = attention_mask.to(device) token_ids = generate( model=gpt, idx=input_ids, attention_mask=attention_mask, max_new_tokens=30, context_size=BASE_CONFIG[\u0026#34;context_length\u0026#34;], top_k=40, temperature=0.5, eos_id=tokenizer.eot_token ) print(\u0026#34;generated output: \\n\u0026#34;) for i,out in enumerate(token_ids_to_texts(token_ids,tokenizer)): print(f\u0026#34;\u0026#34;\u0026#34;{i+1}. \\n{out}\\n\u0026#34;\u0026#34;\u0026#34;) generated output: 1. Every effort moves forward. I\u0026#39;m hopeful that we will be able to do this,\u0026#34; said one of the people. The group plans to hold a meeting at 2. practice makes man a better leader. The problem is, there is no such thing as a perfect leader. You have to start somewhere. You have Top-P (Nucleus Sampling) this fixes the rigidity of the Top-K by using dynamic cutoff based on probability mass, rather than fixed number of tokens Working you set a percentage top_p:0.9, i.e. 90% model sorts the tokens by probability and keeps adding them to a pool until the cumulative sum of their probs hits 90% if model is highly confident: it will take less tokens to reach 90% if the model is uncertain: it will take more tokens to reach 90% Features this is kind of a context-aware filtering rather a rigid number of tokens superior and produces more natural sounding, diverse text Code def generate(model, idx, attention_mask,max_new_tokens, context_size,top_p=None,top_k=None,temperature=0.0, eos_id=None): batch_size=idx.shape[0] finished=torch.zeros( batch_size, dtype=torch.bool, device=idx.device ) for _ in range(max_new_tokens): active_mask=~finished active_indices=torch.nonzero(active_mask,as_tuple=False).squeeze(-1) if active_indices.numel()==0: break active_idx=idx[active_indices] active_attention_mask=attention_mask[active_indices] idx_cond=active_idx[:,-context_size:] attn_cond=active_attention_mask[:, -context_size:] with torch.no_grad(): logits=model(idx_cond,attention_mask=attn_cond) # only select the last token - newly generated token logits=logits[:,-1,:] if temperature\u0026gt;0.0: logits=logits/temperature if top_k is not None: # only cares about ranking not probs top_logits,_=torch.topk(logits,top_k) min_val=top_logits[:,-1].unsqueeze(-1) logits=torch.where(logits\u0026lt;min_val,torch.tensor(float(\u0026#34;-inf\u0026#34;)).to(logits.device),logits) ###### code block change ###### if top_p is not None: # renormalize the remaining tokens probs probs=torch.softmax(logits,dim=-1) sorted_probs,sorted_indices=torch.sort( probs, descending=True, dim=-1 ) cumulative_probs=torch.cumsum(sorted_probs,dim=-1) sorted_indices_to_remove=cumulative_probs\u0026gt;top_p # shift by one - include the token that actually crossed # the threshold sorted_indices_to_remove[..., 1:]=sorted_indices_to_remove[..., :-1].clone() sorted_indices_to_remove[..., 0] = False indices_to_remove=torch.zeros_like(logits,dtype=torch.bool) indices_to_remove.scatter_( dim=-1, index=sorted_indices, src=sorted_indices_to_remove ) logits = logits.masked_fill( indices_to_remove, float(\u0026#34;-inf\u0026#34;) ) ###### code block change ###### if temperature\u0026gt;0.0: logits=logits-logits.max(dim=-1,keepdim=True).values probs=torch.softmax(logits,dim=-1) idx_next=torch.multinomial(probs,num_samples=1) else: # it temp is 0. its same as greedy idx_next=torch.argmax(logits,dim=-1,keepdim=True) new_tokens=torch.full( (batch_size,1), eos_id if eos_id is not None else 0, dtype=idx.dtype, device=idx.device ) new_tokens[active_indices]=idx_next idx=torch.cat((idx,new_tokens),dim=1) new_attention=torch.zeros( (batch_size,1), dtype=attention_mask.dtype, device=attention_mask.device ) new_attention[active_indices]=1 attention_mask=torch.cat((attention_mask,new_attention),dim=1) if eos_id is not None: newly_finished = ( idx_next.squeeze(-1) == eos_id ) finished[active_indices]|=newly_finished if finished.all(): break return idx usage:\ninput_ids, attention_mask = texts_to_token_ids( [ \u0026#34;Every effort moves\u0026#34;, \u0026#34;practice makes man\u0026#34; ], tokenizer ) input_ids = input_ids.to(device) attention_mask = attention_mask.to(device) token_ids = generate( model=gpt, idx=input_ids, attention_mask=attention_mask, max_new_tokens=30, context_size=BASE_CONFIG[\u0026#34;context_length\u0026#34;], # top_k=40, top_p=0.8, temperature=0.3, eos_id=tokenizer.eot_token ) print(\u0026#34;generated output: \\n\u0026#34;) for i,out in enumerate(token_ids_to_texts(token_ids,tokenizer)): print(f\u0026#34;\u0026#34;\u0026#34;{i+1}. \\n{out}\\n\u0026#34;\u0026#34;\u0026#34;) generated output: 1. Every effort moves forward, but the next step is to try to get the best of the best. \u0026#34;We have to make sure that we\u0026#39;re not going 2. practice makes man\u0026#39;s life more difficult, but it\u0026#39;s also a good thing. The best way to learn to be a better man is to be a better min-p sampling scales the cutoff relative to the top choice effectively does the job of temperature and top-p combined without needing constant tuning Working you set a min_p:0.1 The model looks at the probability of the absolute top token. The cutoff for all other tokens becomes min_p multiplied by the top token\u0026rsquo;s probability. If the top token has an 80% chance, any other token must have at least an 8% chance to be considered. If the top token only has a 20% chance, other tokens only need a 2% chance to be considered Features balances the strictness and creativity. aggressively prunes bad options when the model is confident def generate(model, idx, attention_mask,max_new_tokens, context_size,min_p=None,top_p=None,top_k=None,temperature=0.0, eos_id=None): batch_size=idx.shape[0] finished=torch.zeros( batch_size, dtype=torch.bool, device=idx.device ) for _ in range(max_new_tokens): active_mask=~finished active_indices=torch.nonzero(active_mask,as_tuple=False).squeeze(-1) if active_indices.numel()==0: break active_idx=idx[active_indices] active_attention_mask=attention_mask[active_indices] idx_cond=active_idx[:,-context_size:] attn_cond=active_attention_mask[:, -context_size:] with torch.no_grad(): logits=model(idx_cond,attention_mask=attn_cond) # only select the last token - newly generated token # batchsize, 1, logits logits=logits[:,-1,:] if temperature\u0026gt;0.0: logits=logits/temperature if top_k is not None: top_logits,_=torch.topk(logits,top_k) min_val=top_logits[:,-1].unsqueeze(-1) logits=torch.where(logits\u0026lt;min_val,torch.tensor(float(\u0026#34;-inf\u0026#34;)).to(logits.device),logits) if top_p is not None: # renormalize the remaining tokens probs probs=torch.softmax(logits,dim=-1) sorted_probs,sorted_indices=torch.sort( probs, descending=True, dim=-1 ) cumulative_probs=torch.cumsum(sorted_probs,dim=-1) sorted_indices_to_remove=cumulative_probs\u0026gt;top_p # shift by one - include the token that actually crossed # the threshold sorted_indices_to_remove[..., 1:]=sorted_indices_to_remove[..., :-1].clone() sorted_indices_to_remove[..., 0] = False indices_to_remove=torch.zeros_like(logits,dtype=torch.bool) indices_to_remove.scatter_( dim=-1, index=sorted_indices, src=sorted_indices_to_remove ) logits = logits.masked_fill( indices_to_remove, float(\u0026#34;-inf\u0026#34;) ) ###### code block change ###### if min_p is not None: # (batchsize,1,1) probs=torch.softmax(logits,dim=-1) max_prob_values,_=torch.max(probs, dim=-1,keepdim=True) threshold=max_prob_values*min_p indices_to_remove= probs \u0026lt; threshold logits=logits.masked_fill( indices_to_remove, float(\u0026#34;-inf\u0026#34;) ) ###### code block change ###### # token selection if temperature\u0026gt;0.0: logits=logits-logits.max(dim=-1,keepdim=True).values probs=torch.softmax(logits,dim=-1) idx_next=torch.multinomial(probs,num_samples=1) else: # it temp is 0. its same as greedy idx_next=torch.argmax(logits,dim=-1,keepdim=True) new_tokens=torch.full( (batch_size,1), eos_id if eos_id is not None else 0, dtype=idx.dtype, device=idx.device ) new_tokens[active_indices]=idx_next idx=torch.cat((idx,new_tokens),dim=1) new_attention=torch.zeros( (batch_size,1), dtype=attention_mask.dtype, device=attention_mask.device ) new_attention[active_indices]=1 attention_mask=torch.cat((attention_mask,new_attention),dim=1) if eos_id is not None: newly_finished = ( idx_next.squeeze(-1) == eos_id ) finished[active_indices]|=newly_finished if finished.all(): break return idx usage\ninput_ids, attention_mask = texts_to_token_ids( [ \u0026#34;Every effort moves\u0026#34;, \u0026#34;practice makes man\u0026#34; ], tokenizer ) input_ids = input_ids.to(device) attention_mask = attention_mask.to(device) token_ids = generate( model=gpt, idx=input_ids, attention_mask=attention_mask, max_new_tokens=30, context_size=BASE_CONFIG[\u0026#34;context_length\u0026#34;], min_p=0.1, top_p=0.9, temperature=0.7, eos_id=tokenizer.eot_token ) print(\u0026#34;generated output: \\n\u0026#34;) for i,out in enumerate(token_ids_to_texts(token_ids,tokenizer)): print(f\u0026#34;\u0026#34;\u0026#34;{i+1}. \\n{out}\\n\u0026#34;\u0026#34;\u0026#34;) generated output: 1. Every effort moves forward, and we will continue to build on our strong partnership with the United States,\u0026#34; the statement said. \u0026#34;We will continue to work closely with our 2. practice makes man\u0026#39;s life easier, and it\u0026#39;s the best way to get rid of the stress. The best way to get rid of stress is to get Operating Space Temperature and Top-k operate on logits\nalthough top-k can technically operate on probabilities since the rank order doesnot change after softmax but operating on logits is industry standard because it skips calculating exponentials for the grabage tokens you are about to throw away anyway.\nTop-p and min-p operate on probabilities\nand\nTop-p, top-k, min-p won\u0026rsquo;t actually make a difference while greedy decoding, they only make a difference when temperature \u0026gt; 0.0\nthe general order of applying these sampling methods is as below\nLOGIT SPACE ↓ PROBABILITY SPACE ↓ SAMPLING logits ↓ temperature scaling ↓ top_k ↓ top_p ↓ min_p ↓ softmax ↓ multinomial I plan to cover,\nimplementing KV-cache in the above GPTModel implementation IDK whatever comes to mind after that Sampling Methods\nrepetition penalty presence penalty frequency penalty logit bias bad-word masking forced tokens Decoding Methods\nBeam Search Speculative Decoding Contrastive Search/Decoding in the upcoming blogs.\n","permalink":"https://arjun118.github.io/technical/sampling_methods/","summary":"\u003cp\u003eLet\u0026rsquo;s look at some sampling methods used to control the randomness of an LLM\u0026rsquo;s text generation.\u003c/p\u003e\n\u003ch1 id=\"background\"\u003eBackground\u003c/h1\u003e\n\u003cp\u003eLLMs by nature are auto-regressive. Meaning the future values (tokens) are predicted by learning from its own past values\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003enext step in the sequence depends directly on the steps that came before it\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cblockquote\u003e\n\u003cp\u003eeach new word is generated based on the context of  all the words that were generated previously\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eYou can control this exact behaviour, how randomly you want your LLM to generate the next token\u003c/p\u003e","title":"Sampling Methods"},{"content":"As the title suggests, we will look at some self-attention mechanisms in this blog. These are by no means all the known attention mechanisms. I’m writing these based on my understanding while trying to keep the implementations accurate.\nCausal Scaled dot product attention This is the original (“vanilla”) attention mechanism proposed in the Attention Is All You Need paper.\nwe will implement Causal (future tokens masked) scaled dot product self-attention\nlets get our dummy inputs right\nDummy input setup import torch import torch.nn as nn inputs=torch.rand(2048,512) batch=torch.stack((inputs,inputs),dim=0) print(batch.shape) # torch.Size([2, 2048, 512]) 2 is our batch size and each batch has 2048 tokens and each token has an embedding of dimension of 512 (of course all random, nothing real)\nlets also define our config dictionary, just to keep things organized\nconfig={ \u0026#34;context_length\u0026#34;:2048*2, \u0026#34;dropout\u0026#34;:0.1, } we initialize the maximum supported context length to 4096, although our current input sequence length is 2048 and dropout factor is 0.1 (to be used in our self-attention block)\nAll inputs, batch,config will stay same for all the variants we are gonna learn\nbelow are some other variables we initialize so that we don`t keep repeating ourselves\nd_in=inputs.shape[1] d_out=512 #input dimension to output dimension -\u0026gt; generally preferred to preserve input dimensionality first lets see the code, i\u0026rsquo;ve also included comments,shapes beside the code lines but we will also see some explanation below\nCode class CausalSelfAttention(nn.Module): def __init__(self, d_in,d_out,context_length, dropout,qkv_bias=False): super().__init__() self.W_query= nn.Linear(d_in,d_out,bias=qkv_bias) self.W_key= nn.Linear(d_in,d_out,bias=qkv_bias) self.W_value= nn.Linear(d_in,d_out,bias=qkv_bias) self.d_out=d_out self.dropout=nn.Dropout(dropout) self.register_buffer(\u0026#34;mask\u0026#34;,torch.triu(torch.ones(context_length,context_length),diagonal=1)) self.scale=d_out**-0.5 def forward(self,x): # batch size, number of tokens, input dimension b,num_tokens,d_in=x.shape # find out projections using the weight matrices queries,keys,values=self.W_query(x),self.W_key(x),self.W_value(x) # keys -\u0026gt; b, num_tokens, d_out # queries -\u0026gt; b, num_tokens, d_out # values -\u0026gt; b,num_tokens, d_out attn_scores=queries@keys.transpose(1,2) # this is QK^T part of the attention calculation attn_scores.masked_fill_(self.mask.bool()[:num_tokens,:num_tokens],-torch.inf) # apply causal mask and don`t let the current tokens attend to future tokens, only past tokens attn_weights=torch.softmax(attn_scores*self.scale,dim=-1) # (1/sqrt(dk)) part of attention calc attn_weights=self.dropout(attn_weights) context_vec=attn_weights@values return context_vec if you are familiar with PyTorch, the class declaration should make sense along with the usage of Linear, Dropout, transpose, view etc. refer to pytorch documentation if it doesn\u0026rsquo;t, they are pretty self explanatory\nExplanation constructor:W_query, W_key, W_value are linear projection layers which we will need to calcuate the projections from the input matrix - x dropout is for us to apply some sort of regularization - achieved via randomly dropping attention weights at a factor of 0.1 3.forward: in the forward method you determine the batch size, input sequence length, and input dimensionality then you calculate queries, keys, values matrices which are basically projections of the input matrix x calculated using the corresponding weight matrices we calculate attention scores by making the keys matrix multiplication compatible with our queries matrix, refer the shapes from the comments in the code Apply causal mask (mask future tokens) by setting their attention score as negative infinity so that they approximate to zero after softmax. Apply softmax and scale by the inverse square root of the key dimension (head_dim or d_k) scaling factor is used because, 1.keep the variance of these attention scores(dot products) stable 2. as dimension increases , attention logits become larger and larger 3.without it, softmax becomes extremely peaky, gradients become tiny, training becomes unstable apply dropout (for regularization and good generalization) finally calcuate the context vectors by multiplying attention weights with values check the outputs by running the below code\ntorch.manual_seed(123) ca=CausalSelfAttention(d_in,d_out,config[\u0026#39;context_length\u0026#39;],config[\u0026#39;dropout\u0026#39;]) context_vecs=ca(batch) print(context_vecs.shape) # torch.Size([2, 2048, 512]) Multi Head Attention now lets move onto multi head attention, i am not going to draw any images, just code and explanation.\nFor visual intuition , please refer to this excellent blog - https://magazine.sebastianraschka.com/p/visual-attention-variants\nwe will use the same batch as before,\nCode class MultiHeadAttention(nn.Module): def __init__(self,d_in,d_out,context_length,dropout,n_heads,qkv_bias=False): super().__init__() assert d_out % n_heads == 0, \u0026#34;output dimension must be divisible by number of heads\u0026#34; self.W_query = nn.Linear(d_in, d_out, bias=qkv_bias) self.W_key = nn.Linear(d_in, d_out, bias=qkv_bias) self.W_value = nn.Linear(d_in, d_out, bias=qkv_bias) self.d_out=d_out self.n_heads=n_heads self.head_dim=d_out//self.n_heads self.out_proj=nn.Linear(d_out,d_out) # linear layer we use to combine our individual head outputs self.dropout=nn.Dropout(dropout) self.scale = self.head_dim ** -0.5 # Non-trainable tensors that are still part of model state. self.register_buffer(\u0026#34;mask\u0026#34;,torch.triu(torch.ones(context_length,context_length),diagonal=1)) def forward(self,x): # batch size, number of input tokens, input dimension b,input_tokens,d_in=x.shape # calculate query , key ,value projections queries,keys,values=self.W_query(x),self.W_key(x),self.W_value(x) # each of queries,keys, values has # size of b, input_tokens, d_out # split queries keys and values into different heads - only the last dimension queries=queries.view(b,input_tokens,self.n_heads,self.head_dim) values=values.view(b,input_tokens,self.n_heads,self.head_dim) keys=keys.view(b,input_tokens,self.n_heads,self.head_dim) # now each of queries, values, keys has # (batch size, input tokens, number of heads, each head dimension) # transpose 1 and 2 dimension of these matrices (0 based indexing) queries=queries.transpose(1,2) values=values.transpose(1,2) keys=keys.transpose(1,2) # now each of queries, values, keys has # (batch size, n_heads,input_tokens, head_dim) attn_scores=queries@keys.transpose(2,3) # above multiplication fares out as -\u0026gt; (b, n_heads,input_tokens,head_dim) @ (b,n_heads,head_dim,input_tokens) =\u0026gt; (b, n_heads,input_tokens,input_tokens) mask_bool=self.mask.bool()[:input_tokens,:input_tokens] attn_scores.masked_fill_(mask_bool,-torch.inf) attn_weights=torch.softmax(attn_scores*self.scale,dim=-1) attn_weights=self.dropout(attn_weights) # context vector # shape before transpose: batch_size, n_heads,num_tokens, head_dim # after transpose: batch_size, num_tokens,n_heads, head_dim # attn weights - (b, n_heads,input_tokens,input_tokens) # values- (b, n_heads,input_tokens, head_dim) # attn_weights @ values =\u0026gt; (b, n_heads,input_tokens, head_dim) context_vec=(attn_weights @ values).transpose(1,2) # shape of context vec =\u0026gt; (b, input_tokens, n_heads, head_dim) context_vec=context_vec.contiguous().view(b,input_tokens,self.d_out) context_vec=self.out_proj(context_vec) return context_vec Explanation lets consider our d_in = d_out = 256\nEvery single head sees the entire input sequence (all the tokens). They do not look at just a slice of the sentence. conceptually, What the heads split up is the embedding dimension (the features), not the number of tokens (the words). keep in mind that heads are not literally assigned fixed semantic feature ranges, this is just for our understanding. The Sequence vs. The Features Think about the shape of your tensor right before you calculate the attention scores: query shape: (batch_size, n_heads, num_tokens, head_dim). example numbers: (2, 8, 16, 32)\nLet\u0026rsquo;s isolate just one sequence in the batch and look at what Head 1 gets:\nHead 1\u0026rsquo;s slice: A matrix of 16 (tokens) by 32 (features).\nNotice that the 16 is completely intact. Head 1 has access to Token 1, Token 2, Token 3\u0026hellip; all the way to Token 16. It sees the whole sentence. What it doesn\u0026rsquo;t have the full picture of is the d_out dimension. It only gets 32 out of the 256 total features available for those words.\nEvery head sees every token by looking at the scaled dot-product calculation:\nattn_scores = query @ keys.transpose(2,3) The resulting attn_scores tensor has the shape (batch_size, n_heads, num_tokens, num_tokens). Using your numbers, that is (2, 8, 16, 16).\nThat 16 x 16 grid exists inside every single one of the 8 heads. That means inside Head 1, Token 1 calculates an attention score with Token 1, Token 2, Token 3, etc. Every token interacts with every other token, inside every single head.\nThe \u0026ldquo;Inspectors\u0026rdquo; Analogy Imagine your input sequence is a line of 16 cars.\nThe Embedding (d_out): A massive, 256-page spec sheet for each car (paint color, engine type, tire pressure, top speed, etc.).\nSingle-Head Attention: One general inspector looks at all 16 cars and tries to read the entire 256-page spec sheet for every car at the same time to figure out how the cars relate to each other. It\u0026rsquo;s overwhelming, and they average things out.\nMulti-Head Attention (8 Heads): You hire 8 highly specialized inspectors.\nInspector 1 (Head 1) looks at all 16 cars, but only cares about the paint and interior colors (features 0-31).\nInspector 2 (Head 2) looks at all 16 cars, but only cares about the engine and transmission (features 32-63).\nwe eventually combine them using .contiguous().view(\u0026hellip;) and self.out_proj. We do this because while Inspector 1 perfectly understands which cars have matching colors, and Inspector 2 perfectly understands which cars can race each other, neither of them has the full picture of the cars. The final linear projection layer is the \u0026ldquo;meeting room\u0026rdquo; where all 8 inspectors bring their 16 x 32 reports, staple them back together into a 16 x 256 report, and synthesize a final understanding of the entire sequence.\nusage:\ntorch.manual_seed(123) mha=MultiHeadAttention(d_in,d_out,config[\u0026#39;context_length\u0026#39;],config[\u0026#39;dropout\u0026#39;],8) context_vecs=mha(batch) print(context_vecs.shape) # torch.Size([2, 2048, 512]) Multi Query Attention multi query attention is very similar to multi head attention but all the query heads share the same key and value projections across all calculations\nCode class MultiQueryAttention(nn.Module): def __init__(self,d_in,d_out,context_length,dropout,n_heads,qkv_bias=False): super().__init__() assert d_out % n_heads == 0, \u0026#34;output dimension must be divisible by number of heads\u0026#34; self.d_out=d_out self.n_heads=n_heads self.head_dim=d_out//self.n_heads self.W_query = nn.Linear(d_in, self.d_out, bias=qkv_bias) self.W_key = nn.Linear(d_in, self.head_dim, bias=qkv_bias) self.W_value = nn.Linear(d_in, self.head_dim, bias=qkv_bias) self.out_proj=nn.Linear(d_out,d_out) # linear layer we use to combine our individual head outputs self.dropout=nn.Dropout(dropout) self.scale = self.head_dim ** -0.5 # Non-trainable tensors that are still part of model state. self.register_buffer(\u0026#34;mask\u0026#34;,torch.triu(torch.ones(context_length,context_length),diagonal=1)) def forward(self,x): # batch size, number of input tokens, input dimension b,input_tokens,d_in=x.shape # calculate query , key ,value projections queries,keys,values=self.W_query(x),self.W_key(x),self.W_value(x) # each of queries,keys, values has # size of b, input_tokens, d_out # split queries keys and values into different heads - only the last dimension queries=queries.view(b,input_tokens,self.n_heads,self.head_dim) values,keys=values.unsqueeze(1),keys.unsqueeze(1) # values=values.view(b,1,input_tokens,self.head_dim) # keys=keys.view(b,1,input_tokens,self.head_dim) # queries has(b, input tokens,n_heads, head_dim) # transpose 1 and 2 dimension only for queries queries=queries.transpose(1,2) # now queries has (batch size, n_heads,input_tokens, head_dim) # values, keys (batch size, 1,input_tokens, head_dim) attn_scores=queries@keys.transpose(2,3) # above multiplication fares out as -\u0026gt; (b, n_heads,input_tokens,head_dim) @ (b,1,head_dim,input_tokens) =\u0026gt; (b, n_heads,input_tokens,input_tokens) mask_bool=self.mask.bool()[:input_tokens,:input_tokens] attn_scores.masked_fill_(mask_bool,-torch.inf) attn_weights=torch.softmax(attn_scores*self.scale,dim=-1) attn_weights=self.dropout(attn_weights) # attn weights - (b, n_heads,input_tokens,input_tokens) # values- (b, 1,input_tokens, head_dim) # attn_weights @ values =\u0026gt; (b, n_heads,input_tokens, head_dim) context_vec=(attn_weights @ values).transpose(1,2) # shape of context vec =\u0026gt; (b, input_tokens, n_heads, head_dim) context_vec=context_vec.contiguous().view(b,input_tokens,self.d_out) context_vec=self.out_proj(context_vec) return context_vec usage:\ntorch.manual_seed(123) mqa=MultiQueryAttention(d_in,d_out,config[\u0026#39;context_length\u0026#39;],config[\u0026#39;dropout\u0026#39;],8) context_vecs=mqa(batch) Group Query Attention please read this awesome explanatory blog for the theory - https://www.ibm.com/think/topics/grouped-query-attention\nCode class GroupedQueryAttentionV2(nn.Module): def __init__(self,d_in,d_out,context_length,dropout,n_q_heads,n_kv_heads,qkv_bias=False): # n_q_heads -\u0026gt; number of query heads # n_kv_heads -\u0026gt; number of key/value heads or number of groups # n_q_heads//n_kv_heads implies the number of query heads that share a key and value head super().__init__() assert d_out % n_q_heads == 0, \u0026#34;output dimension must be divisible by number of heads\u0026#34; assert n_q_heads%n_kv_heads==0, \u0026#34;number of query heads must be divisible by number of kv heads to form groups\u0026#34; self.d_out=d_out self.n_q_heads=n_q_heads self.n_kv_heads=n_kv_heads self.groupsize=self.n_q_heads//self.n_kv_heads self.head_dim=self.d_out//self.n_q_heads self.W_query = nn.Linear(d_in, self.d_out, bias=qkv_bias) self.W_key = nn.Linear( d_in, n_kv_heads * self.head_dim ) self.W_value = nn.Linear( d_in, n_kv_heads * self.head_dim ) self.out_proj=nn.Linear(d_out,d_out) # linear layer we use to combine our individual head outputs self.dropout=nn.Dropout(dropout) self.scale = self.head_dim ** -0.5 # Non-trainable tensors that are still part of model state. self.register_buffer( \u0026#34;mask\u0026#34;, torch.triu( torch.ones(context_length, context_length, dtype=torch.bool), diagonal=1 ) ) def forward(self,x): # batch size, number of input tokens, input dimension b,input_tokens,d_in=x.shape # calculate query , key ,value projections queries,keys,values=self.W_query(x),self.W_key(x),self.W_value(x) # queries -\u0026gt; b, input_tokens, d_out == b, input_tokens, n_q_heads*head_dim # keys, values of b, input_tokens, n_kv_heads*head_dim # split queries keys and values into different heads - only the last dimension queries=queries.view(b,input_tokens,self.n_q_heads,self.head_dim) values=values.view(b,input_tokens,self.n_kv_heads, self.head_dim) keys=keys.view(b,input_tokens,self.n_kv_heads,self.head_dim) # transpose 1 and 2 dimension queries=queries.transpose(1,2) values=values.transpose(1,2) keys=keys.transpose(1,2) # now queries has (batch size, n_q_heads,input_tokens, head_dim) # values, keys (batch size, n_kv_heads,input_tokens, head_dim) # lets split the queries into (batch size, n_kv_heads, group_size, input_tokens,head_dim) queries=queries.reshape(b,self.n_kv_heads,self.groupsize,input_tokens,self.head_dim) # lets add one another dimension to our key and value matrices, final dimension of keys and values = # (batch_size, n_kv_heads, 1 , input_tokens, head_dim) values=values.unsqueeze(2) keys=keys.unsqueeze(2) # (batch size, n_kv_heads, group_size, input_tokens,head_dim) @ (batch_size, n_kv_heads, 1 , head_dim, input_tokens) attn_scores=queries@keys.transpose(-2,-1) # above multiplication fares out as -\u0026gt; (b, n_kv_heads, group_size,input_tokens,head_dim) @ (b, n_kv_heads, 1,head_dim,input_tokens) =\u0026gt; (b, n_kv_heads, group_size,input_tokens,input_tokens) mask_bool=self.mask[:input_tokens,:input_tokens] attn_scores.masked_fill_(mask_bool,-torch.inf) attn_weights=torch.softmax(attn_scores*self.scale,dim=-1) attn_weights=self.dropout(attn_weights) # values- (batch_size, n_kv_heads, 1 , input_tokens, head_dim) # attn_weights @ values =\u0026gt; (b, n_kv_heads, group_size,input_tokens, head_dim) context_vec=(attn_weights @ values).reshape(b,self.n_q_heads,input_tokens,self.head_dim).transpose(1,2) # shape of context vec =\u0026gt; (b, input_tokens, n_q_heads, head_dim) context_vec=context_vec.contiguous().view(b,input_tokens,self.d_out) context_vec=self.out_proj(context_vec) return context_vec usage remains the same as above\nThe aim of this blog is to provide code level view of few attention implementations\nthere are many other implementations for attention and i will write about those when i learn about them.\n","permalink":"https://arjun118.github.io/technical/attention_variations/","summary":"\u003cp\u003eAs the title suggests, we will look at some self-attention mechanisms in this blog. These are by no means all the known attention mechanisms.\nI’m writing these based on my understanding while trying to keep the implementations accurate.\u003c/p\u003e\n\u003ch1 id=\"causal-scaled-dot-product-attention\"\u003eCausal Scaled dot product attention\u003c/h1\u003e\n\u003cp\u003e\u003cimg alt=\"attention_formula\" loading=\"lazy\" src=\"/images/attention_formula.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThis is the original (“vanilla”) attention mechanism proposed in the Attention Is All You Need paper.\u003c/p\u003e\n\u003cp\u003ewe will implement Causal (future tokens masked) scaled dot product self-attention\u003c/p\u003e","title":"Self-Attention Variations"},{"content":"I am not familiar with the interfaces that exists in other languages, so i will try my best to explain what interfaces are in golang\nIntroduction by now you should be knowing the terms , concrete type, and abstract type\nhere are some quick pointers\nconcrete type\nspecifies exact representation of its values (struct layout, slice header, pointer, etc.) exposes that the intrinsic operations of that representation (arithmetic for numbers, or indexing,append and range for slices) may also provide additional behaviour through its methods zero value is well defined abstract type\nreveals only some of their methods hide the underlying representation cannot see underlying nil pointers without assertion no built-in operations you have a value of interface type, means you know nothing about it until you assert it interfaces are of abstract types\nthere are different built-in interfaces that go uses extensively\nDeclaration type CustomInterface interface { CustomMethod(p []byte) string } an interface type specifies a set of methods that a concrete type must posses to be considered an instance of that interface\nto draw parallel from the above declaration, if a type has a method implemented named CustomMethod on it with the same exact signature, then that type can be considered an instance of that interface (or that type satisfies the CustomInterface)\nyou need not explicitly declare that a type satisfies or implements an interface.just implementing the methods on that type with the same signature and name that the interface declared is sufficient in go. The satisfaction is implicit. A type satisfies an interface if it possesses all the methods the interface requires. neat, right?\nyou can embed an interface inside an interface\nhere is an examples from the io package\npackage io type Reader interface { Read(p []byte) (n int, err error) } type Closer interface { Close() error } type ReadWriter interface { Reader Writer } type ReadWriteCloser interface { Reader Writer Closer } intuitively,for a type to satisfy ReadWriter interface should satisfy all the interfaces that are embedded in it.\nAn empty interface interface {}, places no demands on the types that satisfy it.we can assign any value to an empty interface\nUsage lets see some examples\npackage main import \u0026#34;fmt\u0026#34; type Vehicle interface { HasAWD() bool HasSufficientClearance() bool } type Bike struct { NoOfWheels int CC float64 GroundClearance int HasSpokeWheels bool } func (b Bike) HasAWD() bool { // technically current motorcycles does // not have all wheel drive,we are just mimicking here // treat spoke wheels as AWD if b.HasSpokeWheels { return true } return false } func (b Bike) HasSufficientClearance() bool { return b.GroundClearance \u0026gt;= 210 } func (b Bike) OnlyOnBike() bool { return true } type Car struct { NoOfWheels int CC float64 GroundClearance int AllWheelDrive bool } func (c Car) HasAWD() bool { return c.AllWheelDrive } func (c Car) HasSufficientClearance() bool { return c.GroundClearance \u0026gt;= 210 } func IsOffRoadSuitable(v Vehicle) bool { return v.HasAWD() \u0026amp;\u0026amp; v.HasSufficientClearance() } func main(){ Ktm390 := Bike{ 2, 398.63, 183, false, } Ktm390Adv := Bike{ 2, 398.63, 237, true, } TharRoxx := Car{ 4, 2148.0, 226, true, } fmt.Println(IsOffRoadSuitable(Ktm390)) fmt.Println(IsOffRoadSuitable(Ktm390Adv)) fmt.Println(IsOffRoadSuitable(TharRoxx)) veh:= Vehicle(Ktm390) fmt.Println(veh.OnlyOnBike()) // you cannot call this, explanation below } looking at the above code, we have an interface Vehicle which states that it requires 2 methods to be implemented by any type for them to satisfy it.\nwe also have two struct types Bike and Car which implements these methods hence satisfying the interface.\nIsOffRoadSuitable is a function which has a parameter of type Vehicle. since both Bike and Car satisfy this interface we can pass the instances of both Bike and Car as an argument to this function call and that is precisely what we are doing the following lines\nconsider the below code tho,\nveh.OnlyOnBike() here OnlyonBike is a method only defined on Bike and the interface does not specify anything about this method. so trying to call this method on the instance of an interface would result in a compile-time error. you can also think of this as, veh is of static type Vehicle hence the compiler only allows only those that are listed in the interface\nInterface Values what does an interface value, a value of an interface type actually hold?\nwell, it has 2 components.\nconcrete type value of that type or so called in golang\ndynamic type dynamic value the type component in the interface value is represented by the type descriptor of the concrete type\nlets see some examples and print out some types\nvar w io.Writer io.Writer is a very widely used interface in golang. when declared like above, it assumes the zero-value of the interface which is nil.\nfor an interface value to be nil, both the type and value components has to be nil.\nyou can test whether or not the interface is nil using w==nil and w!=nil\nlet\u0026rsquo;s change this\nw=os.Stdout os.Stdout is of type *os.File, you can check that using\nfmt.Printf(\u0026#34;%T\\n\u0026#34;, os.Stdout) // \u0026#34;*os.File\u0026#34; *os.File implements a Write(p []byte) (n int, err error) which is required by io.Writer interface and hence that satisfies the interface.\ncalling Write method on an interface value containing an *os.File pointer causes the (*os.File).Write methods to be called\nin this case\nw.Write([]byte(\u0026#34;ktm 390 duke\u0026#34;)) prints the same string to the stdout (terminal)\nlooking at some buffers\nw= new(bytes.Buffer) dynamic type is now *bytes.Buffer and the dynamic value is a pointer to the newly allocated buffer\nsince *bytes.Buffer also satisfies io.Writer, we can the Write method to append the string to the buffer\nw.Write([]byte(\u0026#34;wow what a string\u0026#34;)) fmt.Println(w.String()) the above Write method call appends the string to the buffer\nEmptiness before closing this lets truly understand when an interface value is truly nil and when it\u0026rsquo;s dynamic value is nil\na nil interface value contain no value at all\ni\u0026rsquo;ll try to explain the distinction with an example direct from the book\nconst debug = true func f(out io.Writer) { // when debug is true // an empty buffer is created // when called f(buf), there is // dynamic value of out != nil pointer // but a non-nil pointer referring to an empty buffer // and to which we can write to // hence this works // but if the debug is false // the zero value of buffer is passed (nil) to f // and the dynamic value of out be nil and // not some pointer pointing to an empty memory space,ready // to be written // hence the error fmt.Printf(\u0026#34;Dynamic Type of out: %T\\n\u0026#34;,out) fmt.Printf(\u0026#34;Dynamic Value of out: %#v\\n\u0026#34;,out) if out != nil { out.Write([]byte(\u0026#34;f is writing this\\n\u0026#34;)) } } func main() { var buf *bytes.Buffer // originally buf is totally nil,nil pointer, the zero value of the // buffer // fmt.Printf(\u0026#34;zero value of the buffer: %#v\u0026#34;, buf) if debug { buf = new(bytes.Buffer) // non-nil empty buffer // now buffer is empty, but not nil // buf points to something which is empty, // that something has no data yet } f(buf) if debug{ fmt.Println(\u0026#34;content of buffer: \u0026#34; ,buf.String()) } } lets see the results with debug=true\nDynamic Type of out: *bytes.Buffer Dynamic Value of out: \u0026amp;bytes.Buffer{buf:[]uint8(nil), off:0, lastRead:0} content of buffer: f is writing this when debug=false\nDynamic Type of out: *bytes.Buffer Dynamic Value of out: (*bytes.Buffer)(nil) panic: runtime error: invalid memory address or nil pointer dereference [signal 0xc0000005 code=0x1 addr=0x20 pc=0x7ff6b4c0b977] goroutine 1 [running]: bytes.(*Buffer).Write(0x7ff6b4c7f8c8?, {0xc000012198?, 0x7ff6b4c5abe7?, 0x1a?}) C:/Program Files/Go/src/bytes/buffer.go:182 +0x17 main.f({0x7ff6b4c7f8a8, 0x0}) C:/Users/Vishnu/Desktop/DEV/golang/the_go_programming_language_book/interfaces/main.go:255 +0x122 main.main() C:/Users/Vishnu/Desktop/DEV/golang/the_go_programming_language_book/interfaces/main.go:184 +0x1c exit status 2 take a look at the dynamic value of out\nwe can make our f more robust,\nfunc f(out io.Writer) { fmt.Printf(\u0026#34;%T\\n\u0026#34;, out) fmt.Printf(\u0026#34;%#v\\n\u0026#34;, out) if out == nil { fmt.Println(\u0026#34;truly nil interface\u0026#34;) return } if b, ok := out.(*bytes.Buffer); ok \u0026amp;\u0026amp; b == nil { fmt.Println(\u0026#34;out is a typed nil *bytes.Buffer\u0026#34;) return } if out != nil { out.Write([]byte(\u0026#34;done!\\n\u0026#34;)) } } here is some visualization to help you\nthanks for reading! see you later, once i learn more\n","permalink":"https://arjun118.github.io/technical/interfaces_golang/","summary":"\u003cp\u003eI am not familiar with the interfaces that exists in other languages, so i will try my best to explain what interfaces are in golang\u003c/p\u003e\n\u003ch1 id=\"introduction\"\u003eIntroduction\u003c/h1\u003e\n\u003cp\u003eby now you should be knowing the terms , \u003ccode\u003econcrete type\u003c/code\u003e, and \u003ccode\u003eabstract type\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003ehere are some quick pointers\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003econcrete type\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003especifies exact representation of its values (struct layout, slice header, pointer, etc.)\u003c/li\u003e\n\u003cli\u003eexposes that the intrinsic operations of that representation (arithmetic for numbers, or indexing,append and range for slices)\u003c/li\u003e\n\u003cli\u003emay also provide additional behaviour through its methods\u003c/li\u003e\n\u003cli\u003ezero value is well defined\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eabstract type\u003c/strong\u003e\u003c/p\u003e","title":"Interfaces in Golang"},{"content":"I am skipping functions and directly jumping in to methods. I will cover functions in detail but let\u0026rsquo;s get through methods quickly since it\u0026rsquo;s relatively a simple topic\nIf you are coming from any programming language, you are already familiar with methods.\nDeclaration A traditional functional declaration in golang is as below\nfunc Greet(name string) string{ return fmt.Sprintf(\u0026#34;hello, %s!\u0026#34;, name) } When functions are tied to a particular type, they are called Methods (just a funny way of thinking about them). For that to happen there has to be some changes in the way declare method from the way we declare a function\nHere is a sample method\nExample type Employee struct { ID int Salary int Designation string } func (e *Employee) Promote(newSalary int, newDesignation string){ e.Salary= newSalary e.Designation=newDesignation } Before moving forward, let\u0026rsquo;s about the type of receivers we can have for method\nPointer Receiver Value Receiver what we have above is a pointer receiver.\nBy using a value receiver, whenever we call a method which has a value receiver, it makes a copy of each argument value which might not be desirable if the arguments are large.\nwhen a method has a pointer receiver, any changes made on the fields of the receiver argument will reflect in the original receiver (because it is pass by receiver)\nLet\u0026rsquo;s create a sample employee\nemp:= Employee{ 1, 50000, \u0026#34;IC1\u0026#34; } Let\u0026rsquo;s call the method Promote\nemp:= Employee{ 1, 50000, \u0026#34;IC1\u0026#34; } \u0026amp;emp.Promote(100000, \u0026#34;IC2\u0026#34;) the above code, \u0026amp;emp.Promote(100000,\u0026quot;IC2\u0026quot;) does not work as intended because,in Go, method selection (emp.Promote) happens before any unary operator like \u0026amp; is applied.so the compiler first interprets emp.Promote(100000, \u0026ldquo;IC2\u0026rdquo;) as a complete function call expression, evaluates it, and then tries to apply and to the result.\nThere are few correct ways of using it\nCalling a method with pointer receiver Method 1 empptr:=\u0026amp;emp empptr.Promote(100000, \u0026#34;IC2\u0026#34;) Method 2 (\u0026amp;emp).Promote(100000, \u0026#34;IC2\u0026#34;) Method 3 the language helps us\nemp.Promote(100000, \u0026#34;IC2\u0026#34;) the compiler will perform an implicit \u0026amp;emp on variable emp\nCalling a method with value receiver calling a method with value receiver is simple and self explanatory\nlets declare a method with value receiver\nfunc (e Employee) IsFresher() bool{ return e.Designation == \u0026#34;IC1\u0026#34; } using this is simple\nfresher:= e.IsFresher() you can, quite possible call this method on a pointer to the emp instance and go will perform an implicit conversion\nempptr=\u0026amp;emp fresher:= empptr.IsFresher() Struct Embedding we had talked about struct embedding and anonymous fields in a struct and the ease of access they provide in my composite data types blog.\nthe idea here remains the same let\u0026rsquo;s see how we can compose structs and use the methods of other structs by embedding them\nExample package main import ( \u0026#34;fmt\u0026#34; \u0026#34;image/color\u0026#34; ) // Front -\u0026gt; front wheel width in mm // Rear -\u0026gt; rear wheel width in mm type Wheels struct { Front int Rear int } type Bike struct { Wheels Color color.RGBA } func (w *Wheels) UpgradeWheels(newWheels Wheels) { w.Front = newWheels.Front w.Rear = newWheels.Rear } func main(){ // let\u0026#39;s define my current bike - rtr 200 var Rtr200 Bike = Bike{ Wheels{90, 130}, color.RGBA{0, 0, 0, 255}, //it\u0026#39;s moslty black } // let\u0026#39;s define the bike i want to own in the // near future - ktm duke 390 var Ktm390 Bike = Bike{ Wheels{110, 150}, color.RGBA{255, 102, 0, 255}, //THAT ELECTRIC ORANGE LOOKS GORGEOUS } // let me quickly upgrade the types of my rtr200 to that to duke (don\u0026#39;t do this IRL) Rtr200.UpgradeWheels(Ktm390.Wheels) } now if you print what Rtr200 Wheels are you will see {110,150}\nlets look at a few things that are happening here\nwe are direclty accessing the method UpgradeWheels which has a pointer Wheels receiver from an instance of the struct Bike in which the Wheels struct is embedded The methods of Wheels have been promoted to Bike we can also access the methods of the Wheels in this format Rtr.Wheels.UpgradeWheels(Ktm390.Wheels)\nnotice that we are not passing the Ktm390 variable as an argument direclty to the UpgradeWheels call. that is because the argument that UpgradeWheels accepts is still the type of Wheels and not Bike to funnily put, Bike is not Wheels, but it has Wheels and it has one additional method UpgradeWheels promoted by Wheels here is the process by which compiler looks for the declared method named UpgradeWheels when resolving the call to the same method it first looks for a directly declared method named UpgradeWheels on Bike then for methods promoted once from Bike\u0026rsquo;s embedded fields like Wheels then for methods promoted twice from embedded fields within Wheels and so on Method Values When we want to call a method\nwe select and call a method in the same expression like\ne.IsFresher() the whole idea of method values is that we can seperate these 2 actions and they have nice benifits\nthe selector\ne.IsFresher yields a method value, a function that binds a method to a specific receiver value e. now the function can be invoked without a receiver value, it only needs the non-receiver arguments.\nlets look at an example with a struct which defines a point in a cartesian plane\nExample type Point struct {X, Y float64} func (p Point) Slope(q Point) float64 { return (q.Y-p.Y)/(q.X-p.X) } func main(){ p := Point{2, 4} q := Point{5, 11} slopeOfLineWithP:= p.Slope fmt.Println(slopeOfLineWithP(q)) } a silly way to think about this is, assigning functions to a variable in javascript. p.Slope becomes a closure like function, one where p is already captured similar to how js functions can close over variables.\ntake a look at the signature of slopeOfLineWithP\nfunc(main.Point) float64 Methods Expressions the usual way we do a method call is , we supply the receiver in a special way using the selector syntax\ne.IsFresher() now what a method expression is , it yields a function value with a regular first parameter taking the place of the receiver and rest all parameters carried so that we ca call it just like we would a function, without any selector syntax, here is an example\ni am adding another method to our Point struct named Translate\nExample func (p *Point) Translate(angle float64) { oldX := p.X p.X = p.X*math.Cos(angle) - p.Y*math.Sin(angle) p.Y = oldX*math.Sin(angle) + p.Y*math.Cos(angle) } p:=Point{1.5, 7} q:=Point{3, 9.3} slope := Point.Slope fmt.Println(slope(p, q)) // lets look at the signature of the function fmt.Printf(\u0026#34;%T\\n\u0026#34;, slope) translate := (*Point).Translate translate(\u0026amp;p, 2*math.Pi) fmt.Println(p) fmt.Printf(\u0026#34;%T\\n\u0026#34;, translate) take a look at the ouptut\n1.5333333333333339 func(main.Point, main.Point) float64 {1.5000000000000018 7} func(*main.Point, float64) the function signature of slope and translate are what we expected. since Translate is a method which has pointer receiver , it has a pointer to the struct Point as it\u0026rsquo;s first paramter\nThank you for reading!\n","permalink":"https://arjun118.github.io/technical/methods_golang/","summary":"\u003cp\u003eI am skipping functions and directly jumping in to methods. I will cover functions in detail but let\u0026rsquo;s get through methods\nquickly since it\u0026rsquo;s relatively a simple topic\u003c/p\u003e\n\u003cp\u003eIf you are coming from any programming language, you are already familiar with methods.\u003c/p\u003e\n\u003ch1 id=\"declaration\"\u003eDeclaration\u003c/h1\u003e\n\u003cp\u003eA traditional functional declaration in golang is as below\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-go\" data-lang=\"go\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#66d9ef\"\u003efunc\u003c/span\u003e \u003cspan style=\"color:#a6e22e\"\u003eGreet\u003c/span\u003e(\u003cspan style=\"color:#a6e22e\"\u003ename\u003c/span\u003e \u003cspan style=\"color:#66d9ef\"\u003estring\u003c/span\u003e) \u003cspan style=\"color:#66d9ef\"\u003estring\u003c/span\u003e{\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#66d9ef\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#a6e22e\"\u003efmt\u003c/span\u003e.\u003cspan style=\"color:#a6e22e\"\u003eSprintf\u003c/span\u003e(\u003cspan style=\"color:#e6db74\"\u003e\u0026#34;hello, %s!\u0026#34;\u003c/span\u003e, \u003cspan style=\"color:#a6e22e\"\u003ename\u003c/span\u003e)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eWhen functions are tied to a particular type, they are called Methods (just a funny way of thinking about them). For that to happen there has to be some changes in the way declare method from the way we declare a function\u003c/p\u003e","title":"Methods In Golang"},{"content":"Let\u0026rsquo;s go over some composite (aggregate data types) that go has to offer\nWe will be covering\narrays slices structs maps Arrays Arrays are homoegeneous in golang.Meaning, all elements have the same type\nArrays are fixed in size. once you declare it there is not way for you to increase the size of the said array without creating a new one\nindexing arrays in go is same as any other language\nDeclaration and Usage you declare an array in golang as\nvar a[3]int every element in the above array is initialzed to the empty value of it\u0026rsquo;s element type (0 in this case)\nLooping over an array for i,x := range a{ fmt.Println(i,x) } when yo loop over an array using range, for every element you will be able to use two values, the index of the element in the array and the array itself\nif you don\u0026rsquo;t want to the index, just use a _\nfor _,x := range a{ fmt.Println(i,x) } Array Literal var q [3]int= [3]int{1,2,34} there, you just initalized an array of 3 three elements. you donot need to mention the size of the array on the right side of the equality\nit can be replaced using ...\nvar q [3]int= [...]int{1,2,34} the compiler determines the array\u0026rsquo;s size during compile time\nthe size of an array is a part of it\u0026rsquo;s type. which means [3]int and [5]int are different types, meaning you cannot do this\nb:= [3]{1,2,3} b=[4]{1,2,3,4} array comparision works in the same way\na1 := [2]int{1, 2} b1 := [...]int{1, 2} c1 := [2]int{1, 3} fmt.Println(a1 == b1, a1 == c1, b1 == c1) // above is entirely possible d1 := [3]int{1, 2} fmt.Println(a1 == d1) // above is not possible, they are different types altogether in go, function arguments are not passed by reference they are passed by valuesince the arrays are inherently immutable their usage is very limited, even you pass them by reference there is no way to add or rempve array elements\nSlices slices are light weight data structures which give access to the subsequence of all the elements of an array (underlying array)\na slice is made up of three components\npointer length capacity Pointer : the pointer points to the first element of the underlying array that is reachable through the slic\nLength: number of elements in the slice which is always \u0026lt; capacity\nlen(s) Capacity: start of the slice and the end of the underlying array\ncap(s) Declaration and Usage let\u0026rsquo;s declare an array (underlying array for our slice)\ndays := [...]string{\u0026#34;sunday\u0026#34;, \u0026#34;monday\u0026#34;, \u0026#34;tuesday\u0026#34;, \u0026#34;wednesday\u0026#34;, \u0026#34;thursday\u0026#34;, \u0026#34;friday\u0026#34;, \u0026#34;saturday\u0026#34;, \u0026#34;weekend\u0026#34;, \u0026#34;weekday\u0026#34;} now we can get a slice\nslice1:=days[1:6] fmt.Printf(\u0026#34;len of s %d\\ncap of s %d\\nval %v\\n\u0026#34;, len(slice1), cap(slice1), slice1) len of s 5 cap of s 8 val [monday tuesday wednesday thursday friday] here length of slice1 is self explanatory -\u0026gt; you have 5 elements in the slice\nand the capacity as per definition is the start of the slice index to the end of the underlying array. since the slice starts from index 1, we will have 8 as capacity (one less than the length of the underlying array)\nlet\u0026rsquo;s get an another slice, an overlapping one with the previous one\nslice2 := days[4:8] here length of slice2 is self explanatory -\u0026gt; you have 4 elements in the slice\nand the capacity as per definition is the start of the slice index to the end of the underlying array. since the slice starts from index 4, we will have 5 as capacity\nfmt.Printf(\u0026#34;len of s %d\\ncap of s %d\\nval %v\\n\u0026#34;, len(slice2), cap(slice2), slice2) len of s 4 cap of s 5 val [thursday friday saturday weekend] you can always reslice a slice to expand it until it\u0026rsquo;s capacity allows\nfor examples you can do something like\nslice1=slice[:8] fmt.Printf(\u0026#34;len of s %d\\ncap of s %d\\nval %v\\n\u0026#34;, len(slice1), cap(slice1), slice1) output:\nlen of s 8 cap of s 8 val [monday tuesday wednesday thursday friday saturday weekend weekday] you cannot re-slice a slice past it\u0026rsquo;s capacity\nslice2=slice[:6] this will result in a panic because slice2 has a capacity of 5\nSlice Literal slice3 := []int{3, 4, 5, 2, 4, 5} fmt.Println(slice3) you can also create slices using make\nslice4:=make([]int, 3) removing the size from the array declaration makes it a slice\nwe cannot compare slices using == because slices are references and their elements are indirect\nnote that the element at index 0 still exists in the underlying array but just not reachable through slice1 or slice2 because they start at a later index\nchanging the values from either of slice1 or slice2 changes the values of underlying arrays and the other slices too\nhere is what the logical representation of the slices look like\nMaps in golang, map is a reference to a hash table. as in anyother language the keys of map should be hashable and comparable\nDeclaraiton and usage prices:=make(map[string]int) you can do a map literal as\nprices:= map[string][int]{ \u0026#34;banana\u0026#34;:23, \u0026#34;apple\u0026#34;:45 } adding elements to a map is straight forward, so is deleting\nprices[\u0026#34;mango\u0026#34;] = 56 delete(prices, \u0026#34;apple\u0026#34;) Looping over a map for fruit, price := range prices { fmt.Println(fruit, price) } when you try to access a key from a map, which does not exists, it returns the zero value of the type of value it holds which is not very helpful if you are trying to truly know whether or not this key exists in the map. you can do that with\nprice, exists := prices[\u0026#34;banana\u0026#34;] exists here will be a boolean. true if the element exists in the map or else false\nStruct aggregate data type that groups together zero or more names values of arbitrary types as a single entity\neach value is called a field\nDeclaration and usage let\u0026rsquo;s define a Employee Struct\ntype Employee struct { ID int Name string Address string DoB time.Time Position string Salary int ManagerID int } we can also group the field names which as the same type as\ntype Employee struct { ID,Salary,ManagerID int Name,Address string DoB time.Time Position string } i will use the first method here.\ninitalize an empty struct as\nvar employee1 Employee if provided nothing, every field\u0026rsquo;s value of an instance of a struct will be initialized to the zero of their respective types\nbut be sure to group those which are related because the field order is significant to the type identity\nAccessing fields you can access inidivudal struct using the dot notation\nfmt.Println(employee1.ID) we can also get the pointers to the individal fields\nsalary := \u0026amp;employee1.Salary *salary += 5000 fmt.Println(employee1.Salary) the dot notation also works with the pointers to the structs\nvar employee2 *Employee = \u0026amp;Employee{} fmt.Println(employee2.ID) employee2.Position = \u0026#34;Hard Worker\u0026#34; fmt.Println(employee2.Position) if you don\u0026rsquo;t want a field of a struct to be accessed beyond the package where it is defined, start the name of the field with a lower case letter\nby default go\u0026rsquo;s accesss control mechnaism allows fields that only begin with a captial letter to be exported\nStruct literal let\u0026rsquo;s define a Point type, to represent a cordinate in the cartesian plane\ntype Point struct{ X, Y int } you an initialize a point in 2 ways\npass the values to all the fields in the same order as they are declared in the struct\np := Point{1, 2} or just pass what you want or have\np1 := Point{X: 2} if you want your function to modify a struct which it receives it as an argument, you need to pass the pointer of the struct to that function\nyou can create a pointer to the instance of the struct in two ways\npointerp := \u0026amp;Point{1, 2} or\npp := new(Point) *pp = Point{1, 2} Struct Embedding and Anonymous fields when you declare a field with no name but a type in a struct, they are called anonymous fields\nto demonstrate their utility, let us define 2 structs\ntype Circle struct { X, Y, Radius int } type Wheel struct { X, Y, Radius, Spokes int } we can extract X and Y into a new struct called Point as we did above, now the declarations become\ntype Point struct { X, Y int } type Circle struct { Center Point1 Radius int } type Wheel struct { Circle Circle Spokes int } we kind of embedded Point into Circle and Circle into Wheel\nlet\u0026rsquo;s initialize a Wheel instance\n/ w := Wheel{} w.Circle.Center.X = 8 w.Circle.Center.Y = 5 w.Circle.Radius = 10 w.Spokes = 30 fmt.Printf(\u0026#34;%#v\\n\u0026#34;, w) to access X and Y we need to write very verbose statements accessing each of the embedded struct, this can be easily done and bypassed using anon field\nhere is how the type declarations change\ntype Point struct { X, Y int } type Circle struct { Point1 Radius int } type Wheel struct { Circle Spokes int } now initializing the same wheel changes to\nw := Wheel{} w.X = 8 w.Y = 5 w.Radius = 10 w.Spokes = 30 fmt.Printf(\u0026#34;%#v\\n\u0026#34;, w) we can just refer to the names of the individual fields of the structs which are embedded as anonymous fields. the anonymous fields still has a name, that is just equal to type name\nthe above operation is equivalent to,\nw := Wheel{} w.Circle.Point1.X = 8 w.Circle.Point1.Y = 5 w.Circle.Radius = 10 w.Spokes = 30 fmt.Printf(\u0026#34;%#v\\n\u0026#34;, w) That\u0026rsquo;s all for this blog.Thank you for reading.\n","permalink":"https://arjun118.github.io/technical/composite_data_types_golang/","summary":"\u003cp\u003eLet\u0026rsquo;s go over some composite (aggregate data types) that go has to offer\u003c/p\u003e\n\u003cp\u003eWe will be covering\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003earrays\u003c/li\u003e\n\u003cli\u003eslices\u003c/li\u003e\n\u003cli\u003estructs\u003c/li\u003e\n\u003cli\u003emaps\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch1 id=\"arrays\"\u003eArrays\u003c/h1\u003e\n\u003cp\u003eArrays are homoegeneous in golang.Meaning, all elements have the same type\u003c/p\u003e\n\u003cp\u003eArrays are fixed in size. once you declare it there is not way for you to increase the size of the said array without creating a new one\u003c/p\u003e\n\u003cp\u003eindexing arrays in go is same as any other language\u003c/p\u003e","title":"Composite Data Types Golang"},{"content":"In this blog, we will go over some basic data types of golang.\nThis is just me writing about what i learn\u0026rsquo;t in golang\nReference: The Go Programming Language book\nGo\u0026rsquo;s data types fall in to 4 types\nBasic types: numbers, strings, booleans (these are what we will be going over in this blog.) Aggregate types: arrays, structs, slices etc. (in the coming blog) Reference types: pointers, maps, functions, channels etc. (in the coming blogs) Interface type (in the coming blogs) Basic Data Types In this I\u0026rsquo;ll cover intergers, floats, strings, boolean. Nothing too fancy. just in brief. If you already experienced in Go. Please skip this blog and save your time\nIntegers Integers come in 2 flavours in golang, signed and unsigned\neach of them has again 4 different flavours\nint- int8, int16, int32, int64 uint - uint8, uint16, uint32, uint64\nas the name shows, they differ in their size\nNote: Keep in mind that, if you use just int and unit to intialize a variable, they assume their most efficient size for signed and unsigned integers on a particular platform. It will either be 32 or 64\nWhat is a rune? rune is an alias of int32 data type, they are the same underlying type which represents a 32 bit integer\nwhy do you need an alias why you already have int32 ? - just for convenience and clarity\nrune becomes incredingly useful and makes sense when we are working with strings. we will have a look at it there\nthey are used to represent unicode points (more about this in the strings section)\nWhat is a byte? a byte (you already know what a byte is if you are reading this), is an alias for uint8 data type\nit emphasizes the value of a piece of raw data but represents the same underlying data type as uint8\nSigned integers span negative numbers too but unsigned intergers don\u0026rsquo;t\nRange of signed and unsigned numbers int8: -127 to 127 , one bit reserved for sign uint8: 0 to 255 Note: Explicit conversion is required to convert a value from one type to another type, even for the same flavours, to convert an int8 to int32 -\u0026gt; int32(x) where x is a variable of type int8\nFloat floats come in just 2 flavours -\u0026gt; float32, float64\nfloat64 is preferred for most purposes, because float32 computation accumulates error rapidly\nBoolean boolean, as usual comes in 2 flavours\ntrue and false\nbooleans can be combined with \u0026amp;\u0026amp; (and), || (or) operations which have short circuit behaviour. \u0026amp;\u0026amp; has higher precendence than ||\nthere is no implicit conversion or interpreation of booleans to int in golang. meaning you cannot do\nint(1==1) this doesnot evaluate to 1\nif you want, you need to convert them explicitly.\nStrings in go, strings are simply a sequence of bytes\ntext strings are interpretedas UTF-8 encoded sequences of Unicode points (runes)\nlen(string) returns the number of bytes in a string and s[i] retreives the ith byte of the string\nthe number of characters in a string may not be equal to what will be returned by len(string) because some characters in the string might be taking more than 1 byte\ns := \u0026#34;hello, world\u0026#34; fmt.Println(len(s)) // \u0026#34;12\u0026#34; fmt.Println(s[0], s[7]) // \u0026#34;104 119\u0026#34; (\u0026#39;h\u0026#39; and \u0026#39;w\u0026#39;) the ith byte of the string is not necessarily the ith character of the string because UTF-8 encoding of a non -ASCII code point requries 2 or more bytes\nASCII AND UNICODE previously (very very previously, long time i mean), there was only the american standard code for information interchange (ASCII). ASCII uses 7 bits to represent 12 characters\nupper case, lower case english letters, digits, punctuation and device control characters\nto handle all the different data in myraid languages comes unicode\nassign each one of distinct characters a standard number called a unicode code point or rune what we call in golang\nUTF-8 an encoding standard which provides variable length encoding of unicode points as bytes. it uses 1-4 bytes to represent each rune but only 1 byte for ASCII characters as they just occupy 1 byte\nhigher order bit of the first byte of the encoding for a rune indicates how many bytes to follow\nI was confused about this, here\u0026rsquo;s what ChatGPT has to say\nIterating over strings You can iterate over strings in golang in 2 ways\nconventional for loop using range (also for loop) we need to talk about these differently \u0026lsquo;cause they behave differently\nUsing conventional for loop p := \u0026#34;Hello, नमस्ते🙏\u0026#34; when you use the loop like\nfor i:=0; i\u0026lt;len(p); i++{ fmt.Println(\u0026#34;%d \u0026#34;,p[i]) } the output will be\n72 101 108 108 111 44 32 224 164 168 224 164 174 224 164 184 224 165 141 224 164 164 224 165 135 240 159 153 143 we are iterating over the individual bytes of the string, in essence this is expected behaviour because string is a sequence of bytes\nUsing range in for loop When you use range in for loop\nfor _, r := range p { fmt.Printf(\u0026#34;%d \u0026#34;,r) } output will be\n72 101 108 108 111 44 32 2344 2350 2360 2381 2340 2375 128591 if you observe closely the output will be same for first seven 7 characters of the string. this is because the first 7 characters are ASCII which occupies 1 byte each and the rest are not\nYou can also see that after 32 or the \u0026rsquo; \u0026lsquo;, the numbers are not same. because conventional loop iterates over bytes and each rune can occupy multiple bytes.\nto be very specific, we are representing the decimal representation of rune or unicode point\nlet\u0026rsquo;s do another round of detailed printing\nfor _, r := range p { fmt.Printf(\u0026#34;rune: %q | unicode: %U | decimal: %d | bytes used: %d\\n\u0026#34;, r, r, r, utf8.RuneLen(r)) } utf8.RuneLen(r) return the number of bytes that a rune occupies, here r is a rune\nhere is the output\nrune: \u0026#39;H\u0026#39; | unicode: U+0048 | decimal: 72 | bytes used: 1 rune: \u0026#39;e\u0026#39; | unicode: U+0065 | decimal: 101 | bytes used: 1 rune: \u0026#39;l\u0026#39; | unicode: U+006C | decimal: 108 | bytes used: 1 rune: \u0026#39;l\u0026#39; | unicode: U+006C | decimal: 108 | bytes used: 1 rune: \u0026#39;o\u0026#39; | unicode: U+006F | decimal: 111 | bytes used: 1 rune: \u0026#39;,\u0026#39; | unicode: U+002C | decimal: 44 | bytes used: 1 rune: \u0026#39; \u0026#39; | unicode: U+0020 | decimal: 32 | bytes used: 1 rune: \u0026#39;न\u0026#39; | unicode: U+0928 | decimal: 2344 | bytes used: 3 rune: \u0026#39;म\u0026#39; | unicode: U+092E | decimal: 2350 | bytes used: 3 rune: \u0026#39;स\u0026#39; | unicode: U+0938 | decimal: 2360 | bytes used: 3 rune: \u0026#39;्\u0026#39; | unicode: U+094D | decimal: 2381 | bytes used: 3 rune: \u0026#39;त\u0026#39; | unicode: U+0924 | decimal: 2340 | bytes used: 3 rune: \u0026#39;े\u0026#39; | unicode: U+0947 | decimal: 2375 | bytes used: 3 rune: \u0026#39;🙏\u0026#39; | unicode: U+1F64F | decimal: 128591 | bytes used: 4 now here is a simple run down of the verbs\n%q print the quoted string which the rune represents %U the unicode code point %d the decimal representation here is some clarity over the number of bytes a string occupies and the number of runes it consists of.\nWe can get the number of runes a string consists of using utf8.RuneCountInString(p)\nfmt.Println(\u0026#34;Number of runes: \u0026#34;, utf8.RuneCountInString(p)) fmt.Println(\u0026#34;length of string (bytes): \u0026#34;, len(p)) Output\nNumber of runes: 14 length of string (bytes): 29 That\u0026rsquo;s all for this blog. I\u0026rsquo;ll write about the aggregate data types next.\nThank you for reading.\n","permalink":"https://arjun118.github.io/technical/basic_data_types_golang/","summary":"\u003cp\u003eIn this blog, we will go over some basic data types of golang.\u003c/p\u003e\n\u003cp\u003eThis is just me writing about what i learn\u0026rsquo;t in golang\u003c/p\u003e\n\u003cp\u003eReference: \u003ca href=\"https://www.oreilly.com/library/view/the-go-programming/9780134190570/\"\u003eThe Go Programming Language book\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eGo\u0026rsquo;s data types fall in to 4 types\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eBasic types\u003c/strong\u003e: numbers, strings, booleans (these are what we will be going over in this blog.)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAggregate types\u003c/strong\u003e: arrays, structs, slices etc. (in the coming blog)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eReference types\u003c/strong\u003e: pointers, maps, functions, channels etc. (in the coming blogs)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eInterface type\u003c/strong\u003e (in the coming blogs)\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch1 id=\"basic-data-types\"\u003eBasic Data Types\u003c/h1\u003e\n\u003cp\u003eIn this I\u0026rsquo;ll cover intergers, floats, strings, boolean. Nothing too fancy. just in brief. If you already experienced in Go. Please skip this blog and save your time\u003c/p\u003e","title":"Basic Data Types Golang"},{"content":"Let\u0026rsquo;s see what is the difference between Handler, HandlerFunc, Handle, HandleFunc in the net/http package in go and how to use them\nHandler type: interface\nas you can see Handler is an interface. this is the core contract for any object that serve or respond to an HTTP request in Go\nif a particular type wants to act as an HTTP handler, it must implement the ServeHTTP method\nUsage package main import ( \u0026#34;fmt\u0026#34; \u0026#34;log\u0026#34; \u0026#34;net/http\u0026#34; ) type myHandler struct { Text string } func (h *myHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, h.Text) } func main() { mux := http.NewServeMux() mycustomHandler := \u0026amp;myHandler{Text: \u0026#34;this is your custom handler\u0026#34;} mux.Handle(\u0026#34;/\u0026#34;, mycustomHandler) fmt.Println(\u0026#34;Starting server on :8081\u0026#34;) log.Fatal(http.ListenAndServe(\u0026#34;:8081\u0026#34;, mux)) } here our custom type, myHandler can be passed to Handle in place of a handler because myHandler satisfies the interface Handler by implementing ServeHTTP\nbut we haven\u0026rsquo;t seen what Handle does, right? hold that thought we are getting there, it will all make sense in some time\nHandlerFunc now if you are coming from a MVC background (like me, for ex: node, express for backend where you have controllers.js, routes.js etc) you can consider the HandlerFunc as parallel to controller functions\ntype: Function\nnow the purpose of HandlerFunc is to act as a bridge that allows us to use ordinary Go functions (provided it has the correct signature func(ResponseWriter, *Request))\nthe HandlerFunc itself implements Handler (i.e.,it has its own ServeHTTP method)\nrefer to this link\nso when ServeHTTP is called on the value of the type HandlerFunc it simply calls the underlying function that HandlerFunc wraps\npackage main import ( \u0026#34;fmt\u0026#34; \u0026#34;log\u0026#34; \u0026#34;net/http\u0026#34; ) func handlerHome(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, \u0026#34;Welcome to the home page!\u0026#34;) } func main() { mux := http.NewServeMux() mux.Handle(\u0026#34;/\u0026#34;, http.HandlerFunc(handlerHome)) fmt.Println(\u0026#34;Starting server on :8081\u0026#34;) log.Fatal(http.ListenAndServe(\u0026#34;:8081\u0026#34;, mux)) } Handle type: Function\nthis is a primary block,which is used to associate a specific URL pattern to a specific http.Handler\nWhen a multiplexer receives a request whose URL path matches the pattern, it will call the ServeHTTP method of the registered handler\nthe second argument has to be a value that satisfies the http.Handler interface\nwe saw Handle\u0026rsquo;s usage above\nHandleFunc type: Function\nthis is also used to register handler\u0026rsquo;s but the way to do it is a little bit different and convenient\nthis does the same job as http.Handle, but specifically designed to accept a regular function (with correct signature as http.HandlerFunc) directly as handler without requiring to manually convert this function into a http.HandlerFunc and pass it to http.Handle as an handler, more like a convenience wrapper\npackage main import ( \u0026#34;fmt\u0026#34; \u0026#34;log\u0026#34; \u0026#34;net/http\u0026#34; ) func homePage(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, \u0026#34;Welcome to the Home Page!\u0026#34;) } func contactPage(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, \u0026#34;Contact us at contact@example.com\u0026#34;) } func main() { mux := http.NewServeMux() mux.HandleFunc(\u0026#34;/\u0026#34;, homePage) mux.HandleFunc(\u0026#34;/contact\u0026#34;, contactPage) fmt.Println(\u0026#34;Starting server on :8081\u0026#34;) log.Fatal(http.ListenAndServe(\u0026#34;:8081\u0026#34;, mux)) } That\u0026rsquo;s the end of it, hope this helps.\n","permalink":"https://arjun118.github.io/technical/difference_between_handler_handlerfunc_handle_handlefunc_in_go/","summary":"\u003cp\u003eLet\u0026rsquo;s see what is the difference between Handler, HandlerFunc, Handle, HandleFunc in the net/http package in go and how to use them\u003c/p\u003e\n\u003ch1 id=\"handler\"\u003eHandler\u003c/h1\u003e\n\u003cp\u003e\u003cimg alt=\"Handler definition\" loading=\"lazy\" src=\"/images/handler.png\"\u003e\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003etype: \u003ccode\u003einterface\u003c/code\u003e\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eas you can see Handler is an interface. this is the core contract for any object that serve or respond to an HTTP request in Go\u003c/p\u003e\n\u003cp\u003eif a particular type wants to act as an HTTP handler, it must implement the \u003ccode\u003eServeHTTP\u003c/code\u003e method\u003c/p\u003e","title":"Difference Between Handler,Handlerfunc,Handle,Handlefunc in Go"},{"content":"Hey there, good to see you here.\nI\u0026rsquo;ve picked up golang to learn this summer. so I thought building toy projects would be a good way to start with. In the process I stumbled upon gophercises which is arguably one of the best resource to learn to while having fun\nI was trying my hand at building a command line todo tool or task list project in gophercises. tbh it\u0026rsquo;s pretty basic but i get to try out boltdb and the cobra tools which helps you in building command line applications faster.\nso this blog (not very technical) is just me reading about it. if you already know about endianness and how juggle with numbers in go, this one is probably not for you (you should skip it)\nTwo sides of Endianness by now, i think , you are familiar with what endianness is and all, if not, please go through this\nour application is very basic, to over simplify , here is the usage\nadd a task\ntodocli add \u0026lt;task_name\u0026gt;\ndo a task\ntodocli do \u0026lt;task_id\u0026gt;\nlist all tasks\ntodocli list\nthat\u0026rsquo;s it, what\u0026rsquo;d i tell you, toy projects!\nwe use boltdb to persist the tasks data.\nas you have already guessed, the only code we need write is to add (write to the tasks db), list (read from the tasks db), do (delete from the tasks db) for now,we are not storing any completed tasks,we just delete them\ni\u0026rsquo;ll just paste the relevant pieces of code we need to write for add functionaly, on which i want focus on right now.\nfunc Add(task string) error{ db, err := bolt.Open(\u0026#34;tasks.db\u0026#34;, 0600, nil) if err != nil { fmt.Println(\u0026#34;Error opening database:\u0026#34;, err) return err } defer db.Close() // add a new task to the db readWriteError := db.Update(func(tx *bolt.Tx) error { b, er := tx.CreateBucketIfNotExists([]byte(\u0026#34;tasks\u0026#34;)) if er != nil { return er } id, _ := b.NextSequence() fmt.Println(id) fmt.Println(itob(int(id))) err = b.Put(itob(int(id)), []byte(task)) if err != nil { fmt.Printf(\u0026#34;Could not write task to the database: %s\u0026#34;, err) return err } return nil }) if readWriteError != nil { fmt.Println(\u0026#34;Error writing to database:\u0026#34;, readWriteError) return readWriteError } fmt.Println(\u0026#34;Task added:\u0026#34;, task) return nil } func itob(v int) []byte { b := make([]byte, 8) binary.BigEndian.PutUint64(b, uint64(v)) fmt.Println(\u0026#34;itob:\u0026#34;, v, b) return b } i\u0026rsquo;ll simplify the things i don\u0026rsquo;t want you to focus on.\nid, _ := b.NextSequence() let\u0026rsquo;s just say you already have 3 tasks in your db, and the task_id of the last task is 3. now when you try to add a new task, the id variable will have the value 4 indicating the next task\u0026rsquo;s id that is to be added\nlet\u0026rsquo;s focus on this piece of code\nerr = b.Put(itob(int(id)), []byte(task)) if we take a step back and look at Put in boltdb\u0026rsquo;s documentation, we\u0026rsquo;d find\nlooking by the signature ,we can make out that we need to provide a slice of bytes as our key and as our value\nthis is where we need to convert our task_id which is an integer to a slice of bytes. we can convert this into big endian representation or small endian representation while storing in the db\nlet\u0026rsquo;s see how we can convert int into big endian and little endian representations\npackage main import ( \u0026#34;fmt\u0026#34; \u0026#34;encoding/binary\u0026#34; ) func itob(v int) ([]byte, []byte) { be := make([]byte, 8) le:= make([]byte, 8) binary.BigEndian.PutUint64(be, uint64(v)) binary.LittleEndian.PutUint64(le, uint64(v)) return be,le } func main(){ x:=255 be,le:= itob(x) fmt.Printf(\u0026#34;value: %d, big endian representation: %v\\n\u0026#34;,x,be) fmt.Printf(\u0026#34;value: %d, little endian representation: %v\\n\u0026#34;,x,le) } output\nvalue: 255, big endian representation: [0 0 0 0 0 0 0 255] value: 255, little endian representation: [255 0 0 0 0 0 0 0] this is straight forward cause, byte is a synonym of uint8 datatype in go, hence we can store a maximum of 255 in one instance of a byte\nsince 255 is represented as 1 1 1 1 1 1 1 1 1 now this byte when represented in big endian, (uint64, which occupies 8 bytes) 00000000 00000000 000000000 000000000 000000000 000000000 000000000 11111111\nin big endian,the msb (most significant byte) occupies the lowest memory address, while in little endian, the lsb (least significant byte) occupies the lowest memory address\nhence in big endian this becomes 0 0 0 0 0 0 0 255, the same way little endian comes out as 255 0 0 0 0 0 0 0\nlet\u0026rsquo;s see one more example, x=1921\nvalue: 1921, big endian representation: [0 0 0 0 0 0 7 129] value: 1921, little endian representation: [129 7 0 0 0 0 0 0] in uint64 1921 is represented as\n00000000 00000000 00000000 00000000 00000000 00000000 00000111 10000001\ni hope that, now it\u0026rsquo;s corresponding big and little endian representations make sense\nAlright, this is a very short, vauge blog. I wanted to write about something hence this exists.\nNote: big endian is used here. It\u0026rsquo;s often used in BoltDB for ordering purposes. BoltDB stores keys as []byte and compares them lexicographically (byte-by-byte from left to right). hence the numeric order of the keys is preserved naturally\n","permalink":"https://arjun118.github.io/technical/little_and_big_endian_in_go/","summary":"\u003cp\u003eHey there, good to see you here.\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;ve picked up golang to learn this summer. so I thought building toy projects would be a good way to start with. In the process I stumbled upon \u003ca href=\"https://courses.calhoun.io/courses/cor_gophercises\"\u003egophercises\u003c/a\u003e which is arguably one of the best resource to learn to while having fun\u003c/p\u003e\n\u003cp\u003eI was trying my hand at building a command line todo tool or task list \u003ca href=\"https://courses.calhoun.io/lessons/les_goph_35\"\u003eproject in gophercises\u003c/a\u003e. tbh it\u0026rsquo;s pretty basic but i get to try out boltdb and the cobra tools which helps you in building command line applications faster.\u003c/p\u003e","title":"Little and Big Endian in Go"},{"content":" This blog assumes that you are familiar with docker and it\u0026rsquo;s basic concepts such as volumes (that\u0026rsquo;s the only concept we gonna use either way 😅).\nWe have a flask application which is basically a social blogging site where users can register, login (ofcourse), make posts, comment, follow other users and with role based authentication implemented for granting appropriate permissions for users, moderators and admins.\nFollowing the good practice of using a separate python virtual environments to work on different projects, i\u0026rsquo;ve created a virtual environment using the venv python module and built my application in that environment so i could keep my projects\u0026rsquo; isolated\nThis is how my script.py, which is the starting point of the flask application looks like. Your\u0026rsquo;s might look different and it\u0026rsquo;s perfectly alright!. you can ignore FLASK_CONFIG for now, the \u0026lsquo;default\u0026rsquo; value for FLASK_CONFIG indicates that the current environment is a developement one and not a prod one\nimport os from app import create_app, db from app.models import User, Role from flask_migrate import Migrate app=create_app(os.getenv(\u0026#39;FLASK_CONFIG\u0026#39;) or \u0026#39;default\u0026#39;) migrate=Migrate(app,db) with app.app_context(): db.create_all() Role.insert_roles() @app.shell_context_processor def make_shell_context(): return dict(db=db, User=User, Role=Role) if __name__==\u0026#39;__main__\u0026#39;: app.run() look at these three lines of code\nwith app.app_context(): db.create_all() Role.insert_roles() as mentioned earlier, i\u0026rsquo;ve implemented role based authentication (RBAC), you dont need to do Role.insert_roles() if you do not have any roles to begin with. But you still need to create all the database tables if you are using a database\nThe way db.create_all() works is that if a particular table related to a model does not exist in the database, it creates that table in the database. However, if a table associated with the model already exists, it won\u0026rsquo;t create a new table.\nKeep in mind, we need to create the database tables inside the application context of our app or else simply running db.create_all() won\u0026rsquo;t work\nNow let\u0026rsquo;s take a look at the Dockerfile\nFROM python:alpine RUN adduser -D cicada # you can change ths user too! USER cicada ENV FLASK_APP=script.py # replace script.py with the appropriate filename in your project WORKDIR /home/cicada COPY requirements.txt requirements.txt RUN python -m venv venv RUN venv/bin/pip install -r requirements.txt COPY app app COPY migrations migrations COPY script.py config.py boot.sh ./ EXPOSE 5000 ENTRYPOINT [\u0026#34;./boot.sh\u0026#34;] a few points to note here,\nmy directory organization looks like this,(i have omitted out some files intentionally, just wanted to give you guys an idea about the organization)\n├── Dockerfile ├── app │ ├── __init__.py │ ├── auth │ │ ├── __init__.py │ │ ├── forms.py │ │ └── views.py │ ├── decorators.py │ ├── main │ │ ├── __init__.py │ │ ├── errors.py │ │ ├── forms.py │ │ └── views.py │ ├── models.py │ ├── static │ │ ├── favicon.ico │ │ └── styles.css │ └── templates │ ├── 403.html │ ├── 404.html │ ├── 500.html │ ├── auth │ │ ├── login.html │ │ └── register.html │ ├── base.html │ ├── index.html │ ├── profile.html │ └── single_post.html ├── boot.sh ├── config.py ├── data-dev.sqlite ├── migrations ├── requirements.txt ├── script.py we are creating a new python environment in the using python -m venv \u0026lt;name_of_the_environment\u0026gt; we are installing all our dependencies in the new environment here the app folder is my main folder where i have the code for my application the contents of boot.sh , which is the entrypoint of our docker image are #!/bin/sh source venv/bin/activate flask run --host 0.0.0.0 --port 5000 the requirements.txt file contains the list of modules/ dependencies that our flask application needs, you run generate this file by using the below command, just remember to activate your virtual environment before generating\nsource \u0026lt;your_environment_name\u0026gt;/bin/activate pip freeze \u0026gt; requirements.txt Now, it\u0026rsquo;s time to build the image. Navigate to the directory where you have your Dockerfile saved and execute this\ndocker build -t flask_app:latest . once the image is built, you can list out the images using (i\u0026rsquo;ve created a while ago)\n╰─$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE flask_app latest ca0ec641a957 42 minutes ago 129MB let\u0026rsquo;s spin up a container with this image\ndocker run -it -p 8000:5000 --name smapp flask_app:latest we are mapping the port 5000 which will be exposed inside the container to the port 8000 on our host machine, so that we can access the site from http://localhost:8000\nNow you can interact with the website, create a user, publish a post, comment etc.\nBut when you restart the container, all the users, posts, comment which you may have created earlier are lost. This is where the need for volumes come in\nTo ensure data persistence we mount the volume to the container\nlets create a volume\n╰─$ docker volume create smdata smdata list the available volumes\n╰─$ docker volume ls DRIVER VOLUME NAME local smdata mounting a volume,\ndocker run -it --name smapp --mount source=smdata,target=/home/cicada -p 8000:5000 flask_app:latest The source parameter specifies the name of the volume. The target parameter defines the folder path inside the container where the volume will be mounted. Now,even if you restart the container, your data will be persistent\nNext, I will try write about how we can do this using docker-compose and an external mysql db\n","permalink":"https://arjun118.github.io/technical/dockerize_a_flask_application/","summary":"\u003cblockquote\u003e\n\u003cp\u003eThis blog assumes that you are familiar with docker and it\u0026rsquo;s basic concepts such as volumes (that\u0026rsquo;s the only concept we gonna use either way 😅).\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eWe have a flask application which is basically a social blogging site where users can register, login (ofcourse), make posts, comment, follow other users and with role based authentication implemented for granting appropriate permissions for users, moderators and admins.\u003c/p\u003e\n\u003cp\u003eFollowing the good practice of using a separate python virtual environments to work on different projects, i\u0026rsquo;ve created a virtual environment using the \u003ccode\u003evenv\u003c/code\u003e python module and built my application in that environment so i could keep my projects\u0026rsquo; isolated\u003c/p\u003e","title":"Dockerize a Flask Application"}]