![](/posts/media/25-05-01-attending-iclr-2025/e6423653-265a-44c2-b525-04cd0fe94876.png) Attending the International Conference of Learning Representations (ICLR) in Singapore from April 24th to 28th last week was an incredible experience, made even more special as it was the first time such a top-tier AI conference came to our shores. In this post, I aim to cover my motivations for attending the event, what I set out to achieve and what I learnt from the experience. I plan to do a deeper dive into the specific workshops, posters and talks that I attended in subsequent posts, and will link them here once written! ## Motivations 1. Experience a top-tier AI conference to see the latest research trends and breakthroughs, 2. Understand the relationship between research and industrial development better especially in the context of Singapore, and 3. Validate / define exactly which field I want to dive deeper into within AI. For some context on point 3 – I have been working in AI for some time and have known that I am on the side of research / engineering that was leaning more towards the engineering, and also leaning more towards on-device use cases rather than on-cloud deployment for inference. ![](/posts/media/25-05-01-attending-iclr-2025/2c72b85a-2ea2-4591-958a-73dab0374297.png) *Where I exist along the spectrum of one developing AI* If we follow the [swyx / Latent.Space definition of an AI Engineer](https://www.latent.space/p/ai-engineer), perhaps it would look something like this too: ![](/posts/media/25-05-01-attending-iclr-2025/dfc0a519-b532-42dd-ad4a-ffe16d6f32b5.png) *Somewhere between a "ML Researcher" and a "ML Engineer", while jumping to tooling & infra too* I have always felt that this was still a bit vague to me, and I have been trying to define this more clearly with the guidance of my mentor at work, who tends more towards the research side. She previously gave me some advice about the sparsity / quantization / model efficiency space and [Lightweight AI, Embedded AI, Efficient AI]() being something that seems to suit me. However, I have been increasingly wondering if this is true since sometimes some of those derivations and papers make me like 😵‍. On the other hand, another mentor that is further down the engineering side guided me towards topics like [AIMET](), [CUDA](), by hand quantization etc. that definitely tends closer towards practical engineering topics. I have been trying to classify myself as someone that would exist in-between both of their approaches, to be somewhere in the middle – between software and hardware, between theory and practice, between research and engineering. This is why one of the main goals was to find out how I can exist at this space, what specific AI field I can dive even deeper into, and where I currently stand in this AI ecosystem. This then links to finding out more about the Singapore AI scene and what exists outside of Continental, especially in the fields I'm interested in. Anyway back to today's topic – ICLR and what I've learnt. ## My plan, and how it worked out In order to make the most use out of my time there, I wanted to carefully plan the best way to approach attending the conference – so I talked to my colleagues for advice, and [asked my friend chatgpt](https://chatgpt.com/share/6810f0bd-7734-800b-9693-bc0e683ac727). Armed with these new guiding principles to attending a conference, off I went down the rabbit hole of prep work. I first started off building this site! Since I knew that I would be networking, I wanted to have something like a [landing page](https://hello.dariustan.me) or my [main home page](https://dariustan.me) to have something to introduce my work and what I do. That took an entire long weekend to build and populate with information. Then, for the actual conference content, I first started off plotting out the posters that I wanted to visit and to ask questions on, and the workshops that interested me. ![](/posts/media/25-05-01-attending-iclr-2025/0add2111-4955-4be6-a373-26b05b943d58.png) *Overkill – I did follow this in the end, but ended up getting more value from other unplanned sessions too!* Next, I created a list of goals for what I wanted to achieve by the end: - [x] meet Singaporean in industry - [x] meet Singaporean in government - [x] meet someone in my domain of interest - [ ] connect with at least 5 people (did not hit 5) - [x] talk to / approach at least 5 people - [ ] attend 3 meetups (only went for 1) - [ ] use my website at least once (did not physically show anyone, but got some clicks for sure) and as you can see, not all of them were successful by the end of the 5 days. Beyond the main conference, I planned to attend various [satellite events](https://lu.ma/sg-ai) to connect with the local AI community. While I was waitlisted for some interesting meetups like the [Thinking Machines social](https://lu.ma/ecgmuhmx) [\[1\]](https://horace.io/brrr_intro.html), [\[2\]](https://lilianweng.github.io/) or [LatentSpace Live](https://lu.ma/e1kvpgcz) [\[1\]](https://www.latent.space/p/josephine-teo), [\[2\]](https://substack.tech-talk-cto.com/p/what-does-it-mean-to-be-an-ai-hub), [\[3\]](https://substack.tech-talk-cto.com/p/how-singapore-has-the-potential-to), I was fortunate to get an invite to [AI Tinkerers Singapore](https://singapore.aitinkerers.org/), which proved to be a valuable networking opportunity. ## So, what did I learn? To compress the knowledge gained across the 5 days in this already-too-long post would be pushing the limits of anyone's attention span (including mine), so I'd summarise the main points below. ### Where I found most value 1. [Workshop on Sparsity in LLMs](https://www.sparsellm.org/) 1. Chat with [Olivia Hsu](https://weiya711.github.io): Getting the opportunity to listen and talk to experts in the field to hear their opinions on my views of the state of the hardware/software interplay in the current state of AI 2. Panel discussion: Hearing about the struggles between academia and industry, and how sometimes my current role also somehow seems to straddle the challenges and benefits from both sides 3. Talk by [Amir Yazdanbakhsh](https://www.ayazdan.com/): Learning about the newer methods in sparsity and quantization, hardware-software co-design, and simulation tools that can simulate the effectiveness on Google TPUs in terms of scaling and runtime performance on-device. 2. Keynote by [Danqi Chen](https://www.cs.princeton.edu/~danqic/) 1. Challenges faced when working with an academic budget of hardware compute 2. How frontier SOTA results can still be obtained in academia / in hardware compute limited labs with their limited GPU compute 3. Networking in [AI Tinkerers Singapore](https://singapore.aitinkerers.org/) 1. Spoke to Tim Santos of [Graphcore](https://www.graphcore.ai/), who shared some insights on the hardware-software co-design issues in industry, and how a non-NVIDIA player can succeed. 2. Demos by non-AI presenters inspired me to be more proactive and intentional in learning in public, and to be accountable for the countless projects that I start and semi-complete. 3. The fact that they could build AI related features is a wake-up call for me to push more for what I want to do / where I want to be, and to make use of the free work-life balance that my current job allows me to have to work on things that excite me. ### Current trends & buzzwords - The standard few: LLMs, VLMs, MLLMs, RLHF (PPO, DPO etc.), AGI, AI Safety - Some newer ones: In-context learning, Test time scaling, VLAs, World models, Speculative decoding, Scaling laws (and all their variants) - On model compression / sparsity: - LoRA variants (and also optimizing LoRA) - SVD (surprisingly commonly used now) - 4 bit quantization being the sweet spot (though 1.58(?) bit quantization exists too) - On inference devices: - Hardware Lottery, Software Lottery, Hardware-Software Co-design ### Knowledge that I am missing | **Category** | **Topics** | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Model Architectures & Mechanisms** | - RoPE (Rotary Positional Encodings)
- Mamba / RWKV
- Linear Attention Models
- Diffusion Transformer
- Registers (ICLR 2024)
- Circuits (NeurIPS 2024 Spotlight)
- Pythia
- In-context Learning | | **Quantization & Compression** | - Hadamard / HadaCore
- STE (Straight-Through Estimator)
- SmoothQuant
- OmniQuant
- GPTQ
- AWQ
- FlashAttention 1/2/3
- 1.58 Bit Quantization
- 2:4 Sparsity | | **Scaling & Performance Optimization** | - Scaling Laws (and their variants)
- Test Time Scaling
- Kurtosis
- Perplexity Metric
- Tensor, Pipeline, Data Parallelism
- torchao | | **Inference & Decoding Techniques** | - Speculative Decoding
- Optimal Transport Distance | | **Hardware & Infrastructure** | - GPU Architecture / CUDA / Thunderkittens | ### Other conferences that might be relevant to me - [MLSys](https://mlsys.org/) - [ISCA](https://iscaconf.org/) I guess in summary of all these points, there is a seemingly endless pool of knowledge available in the research world and to sift through them to find practically applicable methods is a dauntless task. But, it is heartening to know of these areas of research listed above that also try to bridge the gap with actual real-world production use-cases. ## Last thoughts Attending ICLR truly helped me solidify my understanding of where I fit in the AI landscape and the specific areas at the intersection of research and practical engineering that excite me the most. I'm really glad that I managed to have the opportunity to go, thanks to all my bosses for approving and sponsoring my request for it. In subsequent posts, I will give more details and breakdown on what I learnt on each day of the conference, and I'm hoping to start writing more about my previous and upcoming personal projects in this field!