Machine Learning System Design Interview

Product Information

business requirements change, and (3) data distributions constantly shift. Without an intentional design You can read the web-friendly version of the book here. You can find the source code on GitHub. The Discord to discuss the answers to the questions in the book is here. Product leadership: do you design ML solutions to provide for the needs of users and product designers Given an unfair coin with the probability of heads not equal to .5. What algorithm could you use to create a list of random 1s and 0s.

The search engine for Data Science learning resources (FREE). Bookmark your favorite resources, mark articles as complete, and add study notes. https://aigents.co/learn Alexey: Yeah, so another question from Alvaro. Alvaro is graduating soon and he is a machine learning intern at a startup. He's starting a job hunt, hopefully [inaudible]. So how much system design should he expect as a new grad? ( 54:07) Alexey: Yeah. So, I got my Master’s and then for me, it was enough. I thought that it was just too much time. ( 2:57)

Table of Contents

Valerii: The best way is not even to ask, but to say “My assumption is that. Do you agree with that or not?” You see, you asked the question, but actually, you’ve made an assumption. You say “Are you okay with that?” Because you've been given some information. Of course, in the real world, we would gather the context because context can make everything very different. Because imagine, like in the case of fraud – if you receive a label within minutes, it's very different to receiving a label within months. It affects everything. But you could make an assumption, you say, “My assumption is that.” To build, you might be making many assumptions and nobody prevents you from making assumptions, which will make your life easier. ( 21:10)

Alexey: [laughs] But I think for many people, it will be useful because for each pattern there, they talk about when exactly you need to apply this and how to apply this. They also talk about what kind of tools there are. And since this is a book from Google, there is a lot of focus on Google Cloud, but they also talk about open source solutions like Kubeflow, for example. ( 53:37)

about the authors

The second part consists of over 200 knowledge questions, each noted with its level of difficulty -- interviews for more senior roles should expect harder questions -- that cover important concepts and common misconceptions in machine learning. Alexey: Okay. So we do this, and then you also mentioned A/B tests. We define a metric, and then we say how exactly we are going to measure this metric. What do we do next? ( 44:01)

Alexey: I think there is an article, or more like a mini-book, from Google, which is called The Rules of Machine Learning and I think there the first rule is, “You don't need machine learning.” Or something like that. ( 52:32) Alexey: I think this is a nice example to show the difference between the two. In one you need to design a system – you don't think about machine learning at all. Then on the second, you don't need to think about the scalability or load balancing, sharding – all that – you have a specific machine learning problem that you need to solve and then you go through the solution. Right? ( 23:07) Don’t forget to bring up advanced issues specific to these models. Eg. For logistic regression you could talk about regularization with lasso or adding interaction features to deal with non-linearity. If your model training uses an optimizer, talk about the loss functions you can use. Talk about hyper-parameters and design choices for each model. This is a chance to show off your depth and go beyond the typical shallow ideas people can grok from a data science tutorial! Online Evaluation We’ve trained our model with the hyperparameters that led to the best evaluation metrics in our holdout data. Should we just launch this to the entire user base? Unless you’re fresh out of school, you should know the answer is NO! Valerii: To some extent, it’s like cases for a consulting company. They train you to solve any case, even if you've never been working in their aircraft manufacturing company. But somehow, now you're an expert and you can suggest to the CEO of this company how to run his or her business. ( 39:13) The importance of defining a goal and ways of measuring it

DataTalks.Club

A high level of technical skill is required in the machine learning field, particularly for machine learning engineers. In a machine learning interview, you’ll be asked open-ended questions to test your ability to solve an ML system design problems, similar to system design interviews. Valerii: Yeah. Then for example – I remember that I was doing that for Facebook, and suddenly the guy asked me, “Okay, you said that a metric would be AUC. What is AUC? Why did you say that it's a ranking metric?” I said, “Well, that's because it does that and that.” And he said, “Okay. You know what you're talking about.” ( 25:13) Alexey: Yeah, so the question is, “With this profile, you're very good at doing data science stuff. How did you transition from data science to being good at system design?” ( 59:01) Valerii: Well, you can find me on LinkedIn. Just type in my name, you use a y instead of ii. With the new rules, it should be ii at the end. ( 1:00:12) Brainstorm user features, item features and sources of data signals at the company you’ll be interviewing at. It’s possible you’ll get asked something completely unrelated, but you’ll be thankful if these do come up.

Logistic regression. Try to implement logistic regression from scratch. Bonus point for vectorized version in numpy + completed in 20 minutes sample code from martinpella. Followup with MapReduce version. Alexey: Yeah, exactly. Okay. Maybe one last question. It seems like you have a very solid data science profile, from Grandmaster at Kaggle. That's pretty solid. ( 58:35)

Team

Valerii: Yeah, true. Good catch. Yes, level five is a Senior in terms of the level on Facebook, which means that, if you're on this level, it is an honorary thing to be on this level forever. So if you ended on level four, it was probably because of the ML system design interview. This interview tells the interviewer (Facebook or Google, or whatever company) your ability to have an overview of the system. In 45 minutes, you have to be able to tell a story – almost a monolog of yours – about how you will build the system and touch very different points. ( 11:23) Valerii: As soon as we have a probability, we can calculate the expected fraud, which already leads us to the first metric to assess the quality of the model, which is “expected calibration error,” or “weighted expert calibration error.” Okay, we've got that. We also know that the ideal solution would be a binary classification task – one and zero – the crystal ball, right? We know that this will never happen, however, we know that it's a binary expression and that the output has to be between zero and one and it has to be a probability. So that also tells us “What should be our loss function?” The loss function should be from a family of a proper scoring function. ( 13:58) Let A and B be events on the same sample space, with P (A) = 0.6 and P (B) = 0.7. Can these two events be disjoint? There is so much information one needs to know to be an effective machine learning engineer. It's hard to cut through the chaff to get the most relevant information, but Chip has done that admirably with this book. If you are serious about ML in production, and care about how to design and implement ML systems end to end, this book is essential." - Laurence Moroney, AI and ML Lead, Google