I'm working on my master's in computer science, and this fall I took my first course: Machine Learning for Trading. I picked it partially out of interest and partially out of pragmatics: since I haven't been in school for many years and I've never worked full time while in school, I only considered courses that took 10 hours per week, rather than the 20-30 for the other data-related courses.

I was also curious about trading, and I'll admit I went in biased: were hedge funds as soulless and unethical as I'd heard? How does this stock market thing actually work? And what sorts of machine learning techniques work best on the ultimate random walk?

Here are some things I learned. This post will probably make more sense to you if you're a data scientist, but it should be skimmable even if you don't know your random forests from your black-red trees.

Reinforcement learning is kind of astounding

As it turns out, you can apply lessons learned from physical robots to stock trading. A technique called reinforcement learning (RL) works for both, and is remarkably flexible. I hadn't used RL in practice yet and understood it only at a high level, so it was really instructive to go through concrete examples. We started in a virtual world, on a 10x10 board with a lonely robot who moved through a harsh landscape in search of the goal. Thwarting her were long walls and a quicksand pit. Every move that she wasn't in the goal was agony, sapping her one point. Quicksand cost her -100 points. And getting to the goal got her only one measely positive point. Interestingly, that's plenty to motivate our little robot.

She starts her run, randomly bouncing around the board until she happens to stumble on the goal. It's been painful, but she's learned a lot about what's worked and what hasn't. She starts over and tries again, still randomly. Gradually we decrease the rate of randomness and she's allowed to use the knowledge she's learned about the board. The quicksand is such a hazard (and her wheels are a little hard to control, we've added randomness there too) so she'll be cautious and stop trying the side of the board with the quicksand entirely.

If you've ever watched a piano player go from an awkward Chopsticks to an elegant Moonlight Sonata, this will sound familiar. It's more similar to mammalian learning than you might think. When we start, our neural circuits are inefficient and our movements uncoordinated. As we practice, the right connections are reinforced and the wrong ones ignored, and our movements get more smooth. To be sure, there are major differences. Our robot doesn't understand concepts like "walls" and "quicksand." She only knows that when she's on square 44, she should go north because that has historically resulted in less pain. That's it. But she can do a surprising amount with just that. This is a really powerful, flexible machine learning technique.

We adjusted a few other parameters about our learner. Instead of just learning from the next step, we had them learn from the next possible steps from there as well. We used a learning coefficient to adjust the relative weight between step n and step n+1. This also allows the goal value to propagate back through the map, and my physics brain made me think of a vector field—it was almost like at each point in the map there was an arrow pointing the robot in the best direction. We also implemented a technique called dyna, which allows the robot to learn faster in situations where it is expensive to take steps in the real world. She "hallucinates" possible outcomes based on the experiences she's seen so far, and updates her transition tables according to that probability distribution.

In doing this, I learned some interesting things about random sampling performance—I coded up this beautiful algorithm to track the probability distributions and then sample from them, but it was too slow to pass the course's timeout tests. Much to my chagrin, it was much, much faster to just save every single experience tuple in memory and then randomly sample those. This is an unsustainable memory strategy in bigger systems.

Our next project applied reinforcment learning to the stock market. The first challenge is mapping the price and volume data to a "board" like we did for our robot. To do that, we had to discretize the prices into discrete states. We used indicators, which are scores we compute to quantify the current state of stocks. These indicators and states were designed in such a way that at any point in time, the stock fell into a well-defined state. In the robot's grid world, we described her state as "square 44" which was the 4th row and 4th column. For stocks, I created states like "012" which meant that the first indicator had a value of 0, the second had a value of 1 and the third had a value of 2. 0, 1, and 2 are essentially low, medium, and high. I expected more states to be more predictive, but using three levels per indicator (so 333 = 27 states) worked best. Perhaps they were overfitting less.

The second challenge is defining actions and resulting rewards. For stocks, this maps very neatly: the reward is the money you earn from trading, and our action options are to hold, buy, or sell. The resulting Q-learner trading bot was reasonably successful, earning hypothetical money even during the rocky years of 2007-2008.

I've started looking for places around the company where q-learning could be useful. The e-commerce team has some thoughts around optimizing our email marketing: I can see how the customer's place in the conversion funnel would map to a state, sending various emails would be our actions, and clicking or buying would be rewards.

I also got a puppy while taking this course and did a lot of thinking about how dog training relates to reinforcement learning, but that's a topic for another post!

Random Forests from scratch

I'd used random forests before in my day job, but had never coded one up from scratch just using Numpy. It felt a bit like I was back in the Recurse Center, which believes the best way to deeply understand how something works is to build one yourself.

We used a very "numpy" data structure to hold the data. everything in a dataframe: one row per decision node or leaf node. The root node was in the 0th row, and then the entire left tree was in the next N rows, then the entire right tree. Each subtree had the same structure as the main tree so we created the trees recursively. The columns held the split criteria, split row, and the index of which row to go to for right or left.

There are so many different data structures you could use for building trees. In my current course, Artificial Intelligence, we're building our random forests with a custom DecisionNode class that has pointers to a left and right tree. So it's essentially a linked list, but with a tree structure—a "linked tree"? I find this approach to be more intuitive and less bug-prone than the dataframe approach, but the dataframe is really, really fast and simpler to debug.

How to use numpy directly.

Numpy is a Python library for doing efficient computation on vectors of numbers. Pandas is a Python library that builds on numpy by offering convenience methods. For instance, a Python DataFrame is a dictionary of numpy arrays, plus extremely helpful class methods added. I mean, just look at its read_csv method! You can do fairly sophisticated data cleaning in just one line of code.

I use pandas all the time in my work but hadn't really looked under the hood to do much with numpy directly, and I was really interested to dig a layer deeper. We used numpy directly for our first few projects. I don't think this will change my day-to-day work much, but understanding numpy helps me understand why pandas is architected the way it is, and I am now much more confident applying numpy's many useful vectorizing functions. I also suspect it will come in handy if I need to optimize pandas code, as numpy can e.g. have less memory overhead. It's also useful for matrix operations where you just want to index the data by row and column.

That Pandas was really written specifically for trading

Pandas is a Python library for data manipulation and analysis, and I use it all the time for my work. Wes McKinney wrote it while working at AQR Capital Management. Before this course, I knew intellectually that it had been created by a trader, but after using it for trading I really viscerally see it. It had a bunch of the functions I needed built in and the timeseries handling was really thoughtful. Rather seeing it as a general-purpose tool that can be used for trading, I now see it as a trading tool we're using for other things.

How to fool yourself with sloppy machine learning

You can screw up your trading algorithm by committing classic machine learning errors, like overfitting or getting too excited about insample performance. There are also subtle problems specific to trading that were new to me. For instance, you can get in trouble backtesting your trading strategy with the S&P 500. Today's S&P 500. Your strategy will do great historically, but you've got survivorship bias: you're only including the companies that didn't die. By definition they're the top performers, so of course your strategy did great. Testing on historical S&P 500 companies brings your strategy back down to earth.

There are a few other interesting pitfalls, like accidentally training on future information, ignoring the market impact of your own trades, or proposing buying $10M in a $1M company. Also, some strategies can perform very differently depending on which particular day you happen to start trading! Wild. You can read about these in more detail here.

What those indicators in Fidelity mean!

I was checking my investment account the other day and for the first time really looked at these graphs:

Screen-Shot-2018-03-08-at-9.52.39-AM

I now know exactly what these indicators are, and I've coded up RSI, MACD, and Bollinger Bands from scratch! Pretty cool to see coursework in real life.

How hedge funds work: who can invest in them, their incentive structures, how they go about making money, some pitfalls of underregulation, and thoughts on ethics

Warren Buffet uses an approach called fundamental analysis, where his team learns as much as possible about the business and the sector before investing for the long term. Many (but not all) hedge funds try the opposite: they do technical analysis, looking only at the price and volume information, and then trade very very fast. They're essentially looking at timing and patterns in the stock market and trying to spot trends before anyone else does.

But consider for a minute how many eyes are on the stock market at all times. If there's a sudden change, many other people will see it too. Is it even possible to win? The Efficient Market Hypothesis says no. The market is efficient, and that means that any potential advantages will be acted on very quickly. So how do hedge fund managers get consistently rich?

They set up very clever incentive structures. They traditionally take a "two and twenty" cut: 2% of the assets under their management, and 20% of any profits made. So if the market goes up, they make tremendous money. If the market goes down, they still make 2% on a large asset pool anyway. Only investors feel the pain of a down market. So they're strongly incentivized to take aggressive risks, since there's structurally no downside for them.

But what about the investor. Don't they have a lot to lose? Yes, in exchange for potentially enormous profits, they take on significant risk. In order to become an investor you need to have a very high net worth, and hedge funds are mostly unregulated beyond that. For instance, they have no obligation to share information with investors the same way mutual funds do. I guess regulators figure that if you have a high enough net worth, you can afford to lose a ton of money in a risky investment so why not?

The ethics are murkier than I thought

I went in with a vague anti-finance stance. My college and social circles generally disapprove, and I'd absorbed that stance. But I didn't really knowing how finance works, and I wanted to have a more informed opinion. I expected to be able to come out of this course able to defend that stance with Facts and Figures and Knowledge. But it is less clear than I thought.

Two opinions didn't change: regulation is absolutely essential to not letting greed crash the economy (this course did take us on a brief, technical tour of the 2008 crash). And it's a field that is by definition removed from humans. But it's very hard for me to make the case that trading on its own is actually unethical.

At its core, it's using programming and math on publicly available tickers to try to beat other people to a good deal. And it makes money for pension funds, right? It can only make rich people richer, so if you believe we have a moral obligation to redistribute wealth you'd probably be opposed. But in practice, most funds lose money, so maybe you'd be in favor. The broader regulation environment is (unethically) set up to advantage financial institutions over individual persons, as we saw in the crash, but I find it really hard to argue that most individual funds are acting unethically. Regardless, this is not what I want to spend my limited time on earth working on. Sorry, recruiters!

(I'm very interested in discussing this further. The course did almost zero discussion of the ethics of trading, so I feel like I have a lot to learn about it)

How to be an efficient student.

I was always pretty good at school, in that I got good grades and went to a top college. But I was thorough and linear, struggled with procrastination, and worked too many hours. I bought and read through this book which is all about how to do well without working all the time. It's aimed at the undergraduate level, but it's made a big difference already. It helped me realize that I could skip literally all the reading across all three textbooks for this course, as the material was entirely covered in the lectures. I only wish I'd had it in hand during college!


Scott Young recently profiled Ben Franklin and Van Gogh as people who used copying to learn writing and painting, respectively.

He points out that by devaluing copying in creative achievement, we also throw out its value in learning fundamental techniques.

I've seen this in action watching so many people improve their programming skills at the Recurse Center. Copying existing paradigms and software is an speedy way to quickly and thoroughly understand new architectures and techniques.

For instance, building a to-do list app can be your first mobile app. Cloning Requests teaches you about structuring an API. When I attended, I'd heard that Python was bad at concurrency, so I built a deeply unreliable chat client to probe exactly how it breaks. It was ugly!

None of these are Original Ideas—chat apps have been around for decades—but all are incredible learning experiences. This was a great reminder to keep this learning tool in my back pocket.


Context

Today I ran an informal Lunch & Learn for my coworkers on the basics of getting a password manager set up. I intended for the session to be ten minutes of light lecture and the rest of the time to get set up—many people have always meant to set it up but never gotten around to it. People had a lot of questions, which was awesome, so I ended up talking for about a half hour.

In case you want to run one of these with your family or coworkers or book club or whatever, here is the outline of what I shared. Please note that I've read a bunch about security, so I'm neither an expert nor a n00b, and I'm assuming you're roughly the same. Some advice is non-obvious so make sure you have enough background knowledge to give correct info.

My general approach was to aggressively simplify my recommendations, provide time and support for installation, and leave room for lots of questions.

Setup

A goal is to have them walk out with the software installed and set up, so don't forget to ask people to bring their phone and computer to the session.

I started the session by asking people what they wanted to get out of it. There was a range: some didn't really know what password managers were but were curious. Others already were using them but wanted reassurance they were using them in the right way.

Providing Background

Since this session was voluntary, all attendees were already more motivated than average to attend. I included brief motivation anyway for two reasons: some might need an extra push if they hit a roadbump, and to give them tools convince others!

  1. The Stick: people are trying to hack you right now and this is one of the best things you can do for your security. I have 400+ logins and you probably do too! There's just no way I could do it on my own without reusing passwords. Plus, a little prevention up front is much easier than cleaning up identity theft later. People are also very motivated by stories of epic, tragic hackings like Mat Honan's. I know that strong passwords would not have helped him, but people find this story motivating—I think because it makes the threats tangible and real—so use it.

  2. The Carrot: you're going to love using it! I haven't typed out my credit card number or address in five years! The web is faster and easier to navigate! Never try to remember a password again!

  3. Key concept: (Pun intended.) The crucial idea to convey is that you have one really secure master password, and then it remembers all your other passwords. And this is very exciting because it means that you can make them totally random, all different, very long, and therefore very hard to crack! I showed them a sample password and the Password Generator feature here.

  4. Walkthrough: I then showed them what daily life looks like. I opened 1Password, showed them Logins and Notes (the only two sections I use regularly) and showed them what it looks like on the browser and how I'd use it to buy something. This section provoked lots of great questions!

People asked about my personal practices, and many questions gave nice lead-ins to discussing threat modeling. For instance, I use a much longer master password than I recommend for them because I am worried about online mobs and doxxing. I confessed that I literally used Diceware to choose my master password. Someone brought up that she'd seen Edward Snowden enter passwords with his coat over his head, so I got to talk about the threat model of nation states coming after you.

I like the frame of "what threats are you realistically at risk for" rather than "how paranoid are you feeling" for discussing security preferences. People routinely use the word "paranoid" to dismiss realistic security concerns, so I recommend steering clear of the word entirely.

Because people were interested, I also did a very brief detour into the math of password complexity: if you have a three-character password, each character can be A-Z, a-z, or 0-9, which is 62 characters. So there are 62 * 62 * 62 = 238,328 possibilities. That may sound like a lot, but your clunky old laptop computer can try all of those possibilities in literally seconds.

If you have a three-word password, that is essentially the same number of things to remember but is much harder to guess. Because for each word, you're choosing from a longer list: there are only 62 characters, but Diceware has a list of 76,000 words, and that gives us 438,976,000,000,000 possible three-word passwords. This would take about 2 million times longer to guess than our three-letter password, but is actually easier to remember.

Getting set up

There seem to be several main reasons why people never do this on their own: they don't know how to start, they are paralyzed by the options, and it 'seems like a lot of work.' Lets cut through all three in this section.

Day One: Most people think they need to move every password in right away, and to me that sounds super tiring.

Instead, have them install the software right now and then just go about their everyday life. Each time they log into a new site, the manager will offer to remember their existing passwords. Later they can go back and change weak ones. Some managers also allow you to bulk change weak or reused passwords, which is awesome!

Then try to get them to do these four things before they leave:

  1. Install a password manager. Cut down choices to jolt them out of decision paralysis. Give just two: 1Password is nicely designed, hasn't been hacked yet, and is great if you mostly use Macs, iPhone, and Android. LastPass is a little cheaper and runs on Mac and Windows, but I find the red design a little alarming. Much, much more detailed comparisons here. If they can't decide, just start with LastPass! I also went on a little rant to not be cheap about this—the security of your identity and personal information is worth much, much more than $36 or $12 a year.

  2. Install browser plugins as these are the tools that fill forms for you as you browse.

  3. Install the mobile app (this also lead to a discussion of border security, and sharing the surprising fact that in the US law enforcement can compel you to lock with your fingerprint but not with your passcode)

  4. Create a STRONG master password. It is CRITICAL that they know how important this is! It is the key to the rest of your accounts, which contain not just your identity and your money, but personal information about you and everyone you know and love. DO NOT FORGET TO EMPHASIZE THIS! You don't need to scare people but be crystal clear.

Creating a strong master password is so important that it's worth explaining in some depth. One thing to know is that combining whole words together can provide more security than combining characters together (see complexity sidenote above), if done properly.

If you must show that famous-but-misleading XKCD comic, please emphasize you can't just string together the first words you think of and make a secure password. This would be easy to guess because humans are actually extremely predictable! To give you a sense of what happens when people don't fully understand this concept, one attendee floated the password "my password is secure". Others thought about working in their birthdates or pet names, which is risky if they ever celebrate their birthday on Facebook or post their pets on Instagram. Longer is not stronger if the words are easy to guess!

Acknowledge that doing this right is a pain, but that's part of the tradeoff with password managers—we only need to grapple with one password, but it has to be a great one.

  • 1Password has a good guide to thinking of a strong, memorable master password.
  • Another strategy is to use the Password Generator in Word mode. It will then suggest passwords that are combinations of words that are truly randomly chosen, like "trustful dross sanctify memo", and then you can memorize that. As of 2017, people should use at least six words for their passwords. The faster computers get in the future, the longer your passwords should be.
  • You can literally recommend Diceware if you have the type of audience that would actually do it, but this is very rare. Use with extreme caution as it may discourage them from getting started or worse yet, make them feel like they're not the kind of person who would use a password manager.

I grabbed this book from the library because it's been so highly recommended, and because I've been enjoying the backlash to Busy Culture the last couple of years. I've been finding that my life feels much better when it's not packed to the gills and I can focus on only a couple of things that matter. Because I started that process years ago, many of the ideas here are ones I've seen elsewhere. I think if I'd read this five years ago it would have been a life-changer.

The ideas in this book are simple to articulate but not easy to implement: some tasks will have bigger results than others, so you should spend more time on them. Don't try to do everything at once, think of it as dominoes that fall one at a time—the first one should set you up for the second one. Multitasking is a lie. You should get enough sleep if you want your brain to work. Your purpose should inform the things you choose to work on. Focus means saying "no" (with the obligatory Steve Jobs reference).

The central question they pose is this: what's the ONE thing I can do such that by doing it, everything else will be easier or unnecessary? They suggest applying it to many different areas of your life: spiritual, physical, personal, relationships, job, business, and finances. So "For my physical health, what's the ONE Thing I can do to ensure that I exercise such that by doing it everything else will be easier or unnecessary?"

I wish they'd unpacked this question more, and included even a single example. It's not obvious in my work what is most important—there are many skill areas I could focus on. It also makes it seem like learning or skill development are the most important things, and Doing The Work is an afterthought, which is a little puzzling. The devil is in the details here and the book is very short on details. (This seems like a gap that Top Performer is meant to fill, since it helps you figure out what matters most for success in your field)

I did find it useful to get some reminders of these ideas, and leave these here as notes to myself:

  • Set your goal a little higher than you think you can get.
  • Start the day on uninterrupted time on your one thing, and then spend the rest of the day on everything else.
  • Clear a path to your timeblock—see if you are getting waylaid by distractions.
  • Other things will fall by the wayside and feel chaotic, and you will probably feel guilty about it. You might just have to be ok with this. (This is related to saying 'no' to things in order to do the one thing well). I struggle with this!
  • There is a difference between trying to do the best you can do and trying to do the best that can be done.

Bottom line: it's a quick read, so pick it up if your life feels full and cluttered, or if you don't currently have a regular planning session built into your week and month. It'll take you hours to read and years to implement the ideas.


We're doing a geospatial project at work, and Voronoi diagrams came up. I hadn't heard of them, so did a Google image search. To my surprise, it was beautiful!

I mean, would you look at that:

Google image search results for "voronoi"

Just gorgeous.

Sample Voronoi diagram
From Alex Beutel's blog, more on this in a minute.

So what are these? Each of the spots is a point of interest, like a store in a city. Every colored splotch on the map has the same closest store.

State capitals are important, right? What if the states were defined by the closest state capital?

U.S. map plot

There are a few interesting things that fall out of this simple rule. For instance, every point on each line segment the exact same distance from the two nearest capitals. As you move along it, you get closer to both capitals and then farther from both capitals, but it would be exactly the same to go to one or the other.

You also start seeing this pattern emerge in nature. Like when you put bubbles together, they settle down into a 3D Voronoi diagram, their membranes farthest from the "center" of each bubble:

Soap bubbles

You can also use all kinds of distance measurements. No need to stick to "crow flies" distances. Here's a Voronoi diagram using "Manhattan distance," which is basically how far you'd have to walk if you were a pedestrian in a city:

Want to see diagrams of points in motion? Check out the videos on Alex Beutel's website.

If you think this stuff is cool, check out all the examples and uses of it on Wikipedia!

UPDATE: Carlos Rey pointed me to this Voronoi Reese's cup!

from Reddit