Github Slots

Posted on  by admin
Github Slots

Fan run database for One Piece Treasure Cruise. Includes a character database with all unit captain abilities, descriptions, sailor abilities, and special abilities. Slot Machine in Python. GitHub Gist: instantly share code, notes, and snippets. GitHub is where people build software. More than 50 million people use GitHub to discover, fork, and contribute to over 100 million projects.

# Slots This slot is the element that triggers the dropdown. It can be any HTML element, the components surrounds it with a div which handles the events like hover/click etc. This slot is the content of the dropdown menu. Slots is on PyPI, so you can simply install with pip install slots. Currently, slots works with both Python 2.7+ and 3.4+ and the only dependency is Numpy. Slots is open source (BSD license) and I welcome outside contributions. My desire is to make slots easy-to-use, robust, and featureful.

Multi-armed banditry in Python with slots

Github Slots

Roy Keyes

22 Aug 2016 - This is a post on my blog.

I recently released slots, a Python library that implements multi-armed bandit strategies. If that sounds like something that won't put you to sleep, then please pip install slots and read on.

Multi-armed bandits

The multi-armed bandit (MAB) problem is a classic problem of trying to make the best choice, while having limited resources to gain information. The classic formulation is the gambler faced with a number of slot machines (a.k.a. one-armed bandits). How can the gambler maximize their payout while spending as little money as possible determining which are the 'hot' slot machines and which are the 'cold' ones? In more generic, idealized terms, you are faced with n choices, each with an associated payout probability p_i, which are unknown to you. Your goal is to run as few trials, or pulls, as possible to establish the the choice with the highest payout probability. There are many variations on this basic problem.

Getting the best bang for your buck

Most of us do not spend our time strategizing about real slot machines, but we do see similar real-world problems, such as A/B testing or resource allocation. Because of that, strategies to solve the multi-armed bandit problem are of both practical and intellectual interest to the data scientist-types out there.

There are several strategies for solving the MAB problem. All of them attempt to strike a balance between exploration, searching for the best choice, and exploitation, using the current best choice. Because of these competing goals, determining if you are making optimal choices is not trivial. Instead of simply looking at your average payout from your trials, a quantity called regret is calculated. Intuitively, regret is the payout value lost by making the sequence of choices you have made relative to the payout you would have received having known the best choice from the start. Regret can thus be used as a stopping criterion for making a 'final', best choice.

Example strategy: epsilon greedy

To understand how you might approach the multi-armed bandit problem, consider the simplest reasonable strategy, epsilon greedy:

  • You spend the fraction e of your trials randomly trying different choices, i.e. exploration.
  • For the rest of the time, 1 - e, you always choose the choice with the current highest reward rate, i.e. exploitation.

Too little time exploring the options might lead you to stay with a sub-optimal choice. Too much time spent exploring might lead you to spend unnecessary money on options that you already know are sub-optimal.

slots

I wrote slots with two goals in mind, investigating the performance of different MAB strategies for educational purposes and creating a usable implementation of those strategies for real world scenarios. For both of these goals, a simple API with reasonable default values was desirable.

So what does slots do? Right now, as of version 0.3.0, it has implementations of a few basic MAB strategies and allows you to run those on test scenarios and with real, live data. Currently, those strategies include epsilon greedy, softmax, upper confidence bound (UCB1), and Bayesian bandits implementations.

For 'real world' (online) usage, test results can be sequentially fed into an MAB object. After each result is fed into the algorithm the next recommended choice is returned, as well as whether your stopping criterion is met.

What slots looks like:

Using slots to determine the best of 3 variations on a live website.

Make the first choice randomly, record the response, and input reward (arm 2 was chosen here). Run online_trial (input most recent result) until the test criteria is met.

The response of mab.online_trial() is a dict of the form:

Where:

  • If the criterion is met, new_trial = False.
  • choice is the choice of the next arm to try.
  • best is the current best estimate of the highest payout arm.

For testing and understanding MAB strategies, you can also assign probabilities and payouts to the different arms and observe the results. For example, to compare the regret value and max arm payout probability as more trials are performed with various strategies (in this case epsilon greedy, softmax, upper credibility bound (UCB1), and Bayesian bandits):

The resulting regret evolution:

The estimated payout probability of the 'best' arm after each trial. In this case, the actual payout probability of the best arm is 0.85.

Github Slots

Making it happen

slots is on PyPI, so you can simply install with pip install slots. Currently, slots works with both Python 2.7+ and 3.4+ and the only dependency is Numpy.

The future

slots is open source (BSD license) and I welcome outside contributions. My desire is to make slots easy-to-use, robust, and featureful. If you are interested in slots or the multi-armed bandit problem in general, please check out the References and further reading section below.

This was a very brief overview of the rich subject of dealing with the MAB problem and the slots library. Please checkout slots and send me feedback!

Follow me on twitter!

References and further reading

  • Multi-armed bandit (Wikipedia)
  • Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems (Bubeck and Cesa-Bianchi)
  • Multi-Armed Bandit Algorithms and EmpiricalEvaluation (Vermorel and Mohri)
  • Upper confidence bound (Auer et al)

I have a handful of web projects that I maintain. Some of these are sites like the one for my films (Hello World Film and Don't Worry, I'm Fine) but some others are community projects like Atlanta Code Camp. I've battled with the best way to develop and deploy these projects for quite a while.

The problem has been that for a long time I've just been checking into a main branch in Git and publishing that on every check-in. That's caused me some down time over the years. Maybe with these small projects it doesn't matter, but I like to keep these up (especially this blog) so I wanted to make a change that would address it.

First thing I did was create a branch in Git for working on the project. I thought about doing Pull Requests but since I'm the only dev on most of these, it felt a little to over-complicated. So instead I have a branch I call 'next' that I do active development into:

I've set up an Action to automatically build and deploy the project on every build. If you're curious how to do this in GitHub Actions, see my blog post about it.

The difference here is that on every 'Next' deployment I'm building a 'next' container. If we look in Azure container registry, we can see that there are tags for latest (my released version) and next (my test version):

With that in place, let's look at slots. Azure slots are a way to have multiple versions of your site running at once. One reason for this is what we're using it for. You can have a separate versions for testing, development, etc. Another common reason is to support A/B testing. It allows you to swap slots instantly (to prevent downtime) for those different scenarios. Let's see what it looks like.

In the App Service for this blog, there is an item for 'Deployment Slots':

Opening this you may get the dreaded 'You're not paying enough for this project' notification:

Github Slate

But if you're on a Standard (or better), you'll be greeted by this page:

Note that your main site is already running as the first 'slot'. It's your 'production' slot and 100% of the traffic is going to it. You can add a new slot and this will make a copy of your project, but it's a completely separate (but related) App Service.

Note: you can use this whether you're using containers or not. Just in this example I'm using container deployment

Let's create a new slot. Click on the 'Add Slot' button:

Enter a name for the slot and Click Add (you can clone the settings from the original site if you are using settings). Each slot gets it's own App Service so separate settings and configuration.

The name of the slot becomes a new AzureWebSites.net address so you can view it directly using that domain. For my use, I use the 'sitename.azurewebsites.net' address as my testing address.

Once you have a slot configured, it'll look like this:

While, in my case, I'm always leaving the traffic to the main site at 100%, you could also use it to do A/B testing by having two versions of the site running and sharing the traffic to test how well the sites compare.

Now that I have the slot, I just configure it to use my testing (or next) container:

This image is doing a lot of heavy lifting. Note on the top-right that this is showing that it is an App Service Slot. So while it is an App Service it it's own right, it's related to the main App Service.

By looking at the container settings, I'm setting the container to use a 'next' tag so that this is the test version of this blog. I can view it and be sure that I didn't break anything. Now I'm ready to push this to the live version.

Github Slots Game

To do this, I go back into GitHub and create a Pull Request:

Github Sloth

Once I merge the changes from my 'next' branch to 'main', the GitHub actions take over and create a new 'latest' version of my container.

You can see my actions and source code for this blog if you're curious how it works:

Github Slots

Github Slatejs

Is this something like what you're already doing in real production environments?