Research: A new world
Reflecting on my last two weeks at OpenAI, it’s been a pretty interesting time! I’m excited to be a part of the Scholars program and to be given an opportunity to dive into deep learning and research. I’ve been familiar with the idea of neural networks and training them, I’ve taken classes, but I’ve yet to really apply the ideas I’ve learned. Working as a software engineer, I felt good about having a reasonable take on many engineering problems, but ML/Deep Learning has been a black box with seemingly cool stuff inside. It’s been exciting to transition to the world of research, but as I’ve learned over the past two weeks, it’s a much different world than software engineering.
Upon starting the Scholars program, we had a day of orientation involving meeting my peers and learning about the company. I knew I was pretty ignorant about the field, but this was one of those special moments when you realize to what extent. I was pretty used to being a rather knowledgeable person as a software engineer, able to move throughout the stack with full context. But I was also pretty happy to be starting as a beginner in an environment full of experts. However, when looking at my fellow beginners I felt somewhat inadequate with my (lack of) preparation for this. But we are all approaching this program with varied and unique experiences, so no point in comparing myself 🙂
It’s definitely weird joining an organization during a global pandemic. At my old company, we were growing pretty aggressively and were bringing on new developers all the time. I didn’t really consider how strange it must have been for the new hires to have not met anyone at the company in person. But now being in those shoes, it’s strange trying to soak in a new work environment when your sitting in the same room you’ve been in for the last few months. Everyone I’ve met has been interesting and kind and open with sharing their knowledge and answering my questions. I do wish I was able to meet in person as there’s a certain energy that exists in person that’s hard to replicate at a distance.
Another thing that was new to me is working in research. I’ve been used to working in two-week sprints where we have a clear goal in mind. Having structure was a part of working in an agile team where you have daily stand-ups and weekly meetings about various things. Having my day to structure as I saw fit? It was initially overwhelming but liberating. My general working hours are 10a-6p; so I can have the morning to prime myself by biking or climbing, making breakfast, reading, catching up on sleep, etc. What do I do during these hours? It varies as I’m learning, but my daily work includes:
- following tutorials
- learning Pytorch
- configuring azure stuff
- configuring my dev environment
- talking/meeting people at OpenAI
- looking up all the stuff mentioned in chats
- reading papers
- watching videos/talks explaining concepts
- reading ML blogs
- considering experiments
- trying stuff in python
- syncs with my advisor
- contemplating why things work
- occasionally panicking about how much I don’t know
In general, it’s been nice to have this time to find a cadence that works for me being effective each day and over time.
So what have I learned in the last few weeks? I learned about convolutional neural networks and built them from scratch. I learned about different optimizations on them. I rebuilt several pytorch methods along the way. I learned about different initializations and what problems they solve. I also was able to take another look at some of the ideas I had coming into the program and see at them through a research lens. In considering research topics it’s more beneficial to consider ideas that build off of existing work, solve known problems, or introduce new techniques rather than to consider cool ideas off the cuff. It’s easier to get feedback from people when you have common references to discuss new ideas.
I also read a paper and found it fascinating and enjoyed discussing it with my peers. It was “The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks”. I’ve read few papers before joining OpenAI, as software engineers usually don’t read research papers in my experience. I read the abstract, watched a short video about the paper, then read the actual paper. I thought this was a nice approach, as reading papers always seems so daunting. The video gave really great context to the point of the paper and made reading it much easier as I knew what they were getting across. It was also nice getting my peers' perspective as I learned about subsequent papers that brought up new points, analogous ideas to topics in the paper, and new points of views. I captured some of my thoughts on that paper here!
I feel pretty good about my first two weeks at OpenAI, but ultimately I’m still pretty much a beginner to deep learning and the world of research. I’m excited to peer deeper into the field and to hopefully be able to contribute something meaningful!
