Answer by Justin Rising:

These are the books that I've found helpful. This is by no means a complete list–and in particular, I'm not trying to cover anything beyond the core topics–but it is a solid start. As always, my recommendations tell you as much about my biases and interests as they do anything else.

Applied Statistics and Statistical Computing

- Statistics (Freedman, Pisani & Purves) is hands down the best introductory book for statistical thinking. There's almost no math in here, but reading this and doing the exercises will force you to engage with the material.
- Probability and Statistics for Engineering and the Sciences (Devore) is a perfectly good introduction to basic applied statistics.
- Modern Applied Statistics with S (Venables & Ripley) is a more advanced course on applied statistics with an emphasis on computational methods. All of the methods it uses are available in R, so you can use it without S-Plus.
- Software for Data Analysis: Programming with R (Chambers) is an introduction to programming in R. You won't find a lot of statistics in here, but it's still material that you need to know.
- The Art of R Programming: A Tour of Statistical Software Design (Matloff) is another good book on programming in R.
- Linear Regression Analysis (Seber & Lee) is a nice introduction to linear regression for someone with a strong background in linear algebra.
- Bayesian Data Analysis (Gelman, Carlin, Dunson, Vehtari & Rubin) is the only reasonable choice for starting out with applied Bayesian methods.
- Data Analysis Using Regression and Multilevel/Hierarchical Models (Gelman & Hill) is a fantastic introduction to very important class of regression models. This one is accessible to a wide audience.
- The Elements of Statistical Learning: Data Mining, Inference, and Prediction (Hastie, Tibshirani & Friedman) is a classical text on machine learning methods. I'm not going to try to give a comprehensive list of books on ML, but I wouldn't feel right completely leaving it out, and this is still the standard by which other books are judged.

Mathematical Statistics and Statistical Theory

- I learned my basic theory from Mathematical Statistics with Applications (Wackerly, Mendenhall & Scheaffer). It's a good book, but there's a lot that it doesn't cover.
- Statistical Inference (Casella & Berger) is the standard book on mathematical statistics. It's still missing some stuff, but this is definitely essential.
- Mathematical Statistics: Basic Ideas and Selected Topics (Bickel & Doksum) is a more advanced book on inference that covers statistical decision theory. If you want to really understand statistical methods, you have to understand the basics of that framework.
- A First Course in Linear Model Theory (Ravishanker & Dey) is an introduction to the theory of linear models as opposed to their applications. Even if you want to do applied statistics you need to understand this stuff on a basic level.
- Generalized, Linear, and Mixed Models (McCulloch, Searle & Neuhaus) can be viewed as a second course in linear models that deals with some very useful special cases. This is pretty dry, but it's thorough and fairly clear.
- An Introduction to Multivariate Statistical Analysis (T. W. Anderson) is a reference book on classical multivariate methods. Basically, if you can assume multivariate normality, this book has something on your problem.
- A Course in Large Sample Theory (Ferguson) is the standard introduction to classical asymptotics.
- Asymptotic Statistics (van der Vaart) is a more modern and encyclopedic book on asymptotics.
- The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation (Robert) is a more mathematical introduction to Bayesian statistics. I could put this one in the applied statistics section, but it's not wrong to put it here either.

Probability and Stochastic Processes

- I learned undergraduate probability from Probability (Pitman), which is a fine introductory book that covers all the standard topics.
- I also like Probability: The Science of Uncertainty (Bean), which is another good introduction that has some topics that are a little less standard. If you want to prepare for the actuarial exam on probability, this is a very good book to read.
- I used Introduction to Probability Models (Ross) for my undergraduate stochastic process course. It's a very standard introductory book.
- For a slightly more advanced stochastic processes textbook, I recommend Stochastic Processes (Ross). For most work, this will have pretty much everything you need.
- But if it doesn't, you can probably find what you're looking for in either A First Course in Stochastic Processes (Karlin & Taylor) or A Second Course in Stochastic Processes (Karlin & Taylor).
- Applied Probability (Lange) is a nice book on probability and stochastic processes that covers some unusual topics. It's meant to be accessible to non-mathematicians, although you still have to be mathematically literate.
- If you want (or need) to get into actual probability theory, A First Look at Rigorous Probability Theory (Rosenthal) is a very good place to start. It's very clearly written with a good emphasis on understanding what's going on.
- Probability (Shiryaev) is a more rigorous book that's still clear with a lot of examples.
- Of course, you will have to engage with Probability: Theory and Examples (Durrett). This is the standard reference for probability theory, and almost everything that a non-probabilist could need is in here. It's also the standard textbook for a graduate-level probability theory course, but you will find it helpful to supplement it with something else.
- Stochastic Calculus and Financial Applications (Steele) is a gentle and intuitive introduction to stochastic calculus. It's more aimed at someone who wants to use stochastic calculus than the hardcore theoreticians.
- If you do want to be a hardcore theoretician, Brownian Motion and Stochastic Calculus (Karatzas & Shreve) is the place to start.
- Last but certainly not least, I'm going to throw in a recommendation for Counterexamples in Probability (Stoyanov). You can't really understand a subject until you've made a few wrong conjectures and learned why they're off, and this book probably talks about some things you'll believe by the time you get here.

What are some good books for learning probability and statistics?