<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>LLM on Cicada's Blog</title><link>https://arjun118.github.io/tags/llm/</link><description>Recent content in LLM on Cicada's Blog</description><generator>Hugo</generator><language>en-us</language><lastBuildDate>Fri, 08 May 2026 20:46:06 +0530</lastBuildDate><atom:link href="https://arjun118.github.io/tags/llm/index.xml" rel="self" type="application/rss+xml"/><item><title>Self-Attention Variations</title><link>https://arjun118.github.io/technical/attention_variations/</link><pubDate>Fri, 08 May 2026 20:46:06 +0530</pubDate><guid>https://arjun118.github.io/technical/attention_variations/</guid><description>&lt;p&gt;As the title suggests, we will look at some self-attention mechanisms in this blog. These are by no means all the known attention mechanisms.
I’m writing these based on my understanding while trying to keep the implementations accurate.&lt;/p&gt;
&lt;h1 id="causal-scaled-dot-product-attention"&gt;Causal Scaled dot product attention&lt;/h1&gt;
&lt;p&gt;&lt;img alt="attention_formula" loading="lazy" src="https://arjun118.github.io/images/attention_formula.png"&gt;&lt;/p&gt;
&lt;p&gt;This is the original (“vanilla”) attention mechanism proposed in the Attention Is All You Need paper.&lt;/p&gt;
&lt;p&gt;we will implement Causal (future tokens masked) scaled dot product self-attention&lt;/p&gt;</description></item></channel></rss>