Spirob

Short video summary:

AutoScout

Video summary

A match in the FIRST Robotics Competition is a 2.5 minute game where two alliances of three teams each compete using their robots to score points and complete objectives. Knowing the strength and strategy of other teams is a vital part in coordinating match strategy and drafting alliances for playoffs. AutoScout aims to be a tool to track the paths and game-elements scored of each team’s robot.

Each match is recorded to be livestreamed, and that livestream contains the score and play field from three different camera angles. My hypothesis was that this video feed was enough to accurately track the position of each FRC robot throughout a match.

I initially tried OpenCV’s built in trackers, like CSRT and KCF. Although it processed quickly and worked quite well and the robot was in the open, the tracks required initialization and they would lose the robot from minimal obstruction. So, I wanted to take a different approach that could take advantage of all three cameras. I theorized that you could do this by detecting robots in each frame using a YOLO model for each frame, then combine those detections into full paths with an algorithm.

To create a YOLO model, I needed to make a dataset of annotated images. I wrote a script that randomly downloads random match videos using the TBA API and YT-DLP, then I uploaded those frames to Roboflow, where I annotated ~200 images. After that, I trained a YOLO v8 model on a Google Colab notebook and downloaded the weights to my computer.

Next, I began to work on the tracking part. After detecting all the robots on screen, I would project those points to a top-down map of the field so that all the robot positions would be on the same coordinate space. I marked four points on each camera feed so that I could use a perspective transform to move the points from the camera perspective to the top-down field.

Then, I wrote a set of rules in order to decide on the actual positions of robots by looking for clusters of points. I wrote a set of rules that grouped pairs of detections from the side-view camera and full-field camera, and also accounted for edge cases like double detections. This outputted list of fairly-accurate robot detections.

In order to make detections into tracks, I ran looped through each frame of the video and assigned the new detections to be part of the tracks of detections from the past frame that were the closest.

However, there were some moments where the robots were completely obstructed from all cameras. So instead of trying to get the full tracks of each robot throughout the entire match, I just made small fragments of paths. If a tracklet didn’t detect a robot for several frames, it would be terminated. Afterwards, I went through again and stitched together the tracklets. Once a tracklet ended it would look for the next tracklet that popped up closely. This let me get tracks of all six robots throughout an entire match.

This worked quite well for some matches, but struggled with matches where there were frequent false positive detections or the robots were obstructed for long periods of time. In those matches, it would lose the robot and be unable to recover. I set out to make a few improvements:

First, I improved the reliability of tracking the robots between frames by implementing a Kalman filter with a Hungarian matching algorithm (instead of greedy closest-pair matching).

Also, I made improvements to how tracklets were being stitched together. In the current system, some tracks sometimes got lost chasing the small false positives and discarding the long, real tracks. To improve this, I made the objective to minimize a loss function, where big jumps in robot position or not using long tracklets would be penalized. Then, I ran a solver to determine the best assignment of tracklets to full-match tracks of robots. Finally, I added some smoothing over the missing frames and got results like this.

I wanted to make this more polished, so I made an interface with Tkinter that created a pipeline, so matches could be processed in a queue. I also experimented with reading bumper numbers with EasyOCR and determining when robots scored game elements by detecting score increments.

I challenged myself to build an AI system to play Clash Royale, a complex real-time strategy game. The objective of the game is to place troops to destroy your opponents towers. What made this difficult was that the matches had a long horizon, being ~3 minutes long, and there were over 100 different cards, each with their own unique interactions.

To build such a system, I modeled my approach off of AlphaStar, an AI by Google Deepmind that beat players at StarCraft. They first used supervised learning to learn how to imitate the human player from millions of human replays, then used reinforcement learning in a simulated environment to further refine their playstyles.

For the supervised learning phase, there were no official game replays I could access, so I looked online and found Stats Royale, a youtube channel that had automatically uploaded tens of thousands of match videos. By using computer vision, I theorized I could turn this into an effective dataset.

For the reinforcement learning phrase, I wanted to make an environment that could simulate the game thousands of times faster than normal speed. So, I attempted to recreate a 1:1 replica of the game interactions, utilizing a CSV of card stats that I found in the game files.

Because the game was quite complex, I needed a custom ML structure. I feed the game state and grid of troops into a LSTM, which also looks over the past several frames. Its output is fed into two heads that make the game decisions and a value network. The first determines which card is played, either 1, 2, 3, 4 or NO-OP when no card is played. The second decides on where the card is played. The value network evaluates which player is winning (and will assist in Actor Critic learning for RL).

The first challenge I tackled was reading the board state. The game had no official API to extract the exact board state, so I needed to do everything based on computer vision. In order to count each player’s elixir (currency), I masked the frame for a certain purple color and counted the length of the bar. To read the timer, I initially used EasyOCR, but this was slower than what I wanted. So, I first found the numbers by shooting a ray from the right side of the screen and flood-filling white pixels. Then, I classified the numbers matching those pixels against a mask of pixels for each number and seeing which mask matched the closest. This was over a hundred times faster.

Reading the cards in hand was another challenge. Cards could either be in color or black and which depending if the player could afford the card. I decided to only classify colored cards, because the greyed out version had a timer overlay that would be difficult and prone to error from pixel-based matching. After processing a match, I could analytically solve for what the grey cards must have been. To classify the colored cards, I split the card into 16 regions and in each region got the average color. I compared those averages against reference images for each card and matched it with whichever one was the closest. I found that matching in the LAB instead of RGB color space further improved accuracy. I tested this out on hundreds of random frames and was surprised in how consistent it was.

Getting the placement of cards would also be necessary for the dataset. I determined when cards were played based off of when the elixir count decreased. There, I would check for the empty slot, and that was the card played. To determine where the card was played, I used a YOLO v11 model which was trained to look at the field and locate where a stopwatch icon appeared, since that was always the exact tile where a troop was played.

I didn’t just want to determine what cards were in hard, but also the exact order of the cards in the deck (cards played always go to the bottom of a players deck). Doing so would also let me solve for the greyed out cards. I assigned each card an index (0-7) and kept track of the exact order of the deck and hand. I would keep track of which detected label was associated with each card index. I could use the information of the when cards were played to reconstruct the hand at any given timestamp. This worked nearly all of the time, but there were some rare edge cases where card placements were or missed and there were false positives. These were dangerous because they completely messed up the solve at the end. To fix this problem, I tried a bunch of things, but ended up with an algorithm that experimentally tested out removing placement instances and adding new placement instances in between two existing ones. If it increased the solve accuracy, I added that change. Usually 1-2 edits were made per match, and this did a great job at improving reliability.

Next, I tackled troop detection, and for that I wanted a YOLO model. I first looked at RoboFlow Universe to find datasets by other people, and while there were many out there, since there were 240+ different labels needed, their size of around one to two thousand images would not suffice. So, I decided on creating a synthetic dataset. I extracted the game assets from the .apkg, and placed the sprites on a blank arena, with annotations automatically added. This saved me from dozens of hours of annotating by hand.

ClashBot

Hackthon projects

PillPal

CanvAI

Kittenbot

Created a simulated and physical soft robotic tentacle capable of lifelike grasping in MuJoCo, with ongoing work toward transferring learned policies to real hardware via IMU feedback.

Created a simulated and physical soft robotic tentacle capable of lifelike grasping in MuJoCo

Created a simulated and physical soft robotic tentacle capable of lifelike grasping in MuJoCo, with ongoing work toward transferring learned policies to real hardware via IMU feedback.

Data science

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incidid

App Usage prediction

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incidid

Song Popularity analysis

Classmate Survey

Movie Clustering

Income & Air Quality

Copper Future prediction

Games

You didn’t come this far to stop

Pufferchase

Time Crystal

You didn’t come this far to stop

Survival school

Purwatt

You didn’t come this far to stop

Discord bots

You didn’t come this far to stop

Let's build something great together