So, let’s talk about timing of tasks in FRC, or anything really. In an FRC match you have 135 seconds in which to do actions to win the match. What these specific actions are is irrelevant for this discussion, however, we’ll be using a subset of 2013’s actions as examples.
First, a bit of book keeping. When we say win I mean that your score is higher than that of your opponents. It doesn’t matter if it is 1 to 0 or 157 to 156. We’re also going to define a limited set of actions so that we don’t go too far off point here.
Shoot 3 pointers
Defend
Ok, so we skipped a LOT of possible actions in there. That’s fine, this isn’t about showing that a certain strategy is best; We’re investigating a process you can use to choose actions and strategies.
We’re going to assume that we are a reasonable cycler robot. This means we take about 27 seconds per cycle and are capable of 4 cycles in a match. For each of these scenarios we’re going to say that we are 80% accurate in whatever goal we’re aiming for. That’s a HUGE assumption since the overwhelming majority of teams were nowhere near that accurate in the 3 point goal. 469 was, you weren’t, end of discussion.
Now, that doesn’t seem like a lot of points but that is ignoring auton and end game points. We’ll get to end game in a bit, for now how do we lower point output? Simple, we cut down a cycle.
contribution(runs=3) => 28
Well, that was effective. But what would the opponent have to do to remove those 10 points? Remember how we said each cycle takes 27 seconds? You’ve probably picked up that 27 seconds per cycle x 4 cycles < 120 seconds. In fact, it’s a lot less, and that fact plays to our advantage. We are wasting 12 seconds, this means that to remove a complete cycle our opponents would need to slow us down for that time period. Or, impede us over 3 seconds per cycle.
By this point you’re probably wondering why we’re boring you with the obvious “moar cycles = moar wins” schtick. Bear with us here… We’re now going to play against Team Plowie, they were capable of 6 cycles per match. Meaning their contribution is:
contribution(runs = 6) => 57
Clearly Team Plowie is a nasty opponent, how do we counter them with our measly 28 points? To answer that we have to see how they got to 6 cycles.
Simply put,
cycles(time) = 120/(time)
So we can derive why we got 4 cycles from
floor(cycles(27)) => 4
Solving backwards we find that time defined in terms of cycles is:
time => 120/cycles
So to get how fast they need to cycle we can plug in 6 and get:
time(6) => 20
But how long do we have to slow them down before they lose a cycle? Not long at all, in fact they are running so close to the limit that if we can harry them for 1 second per cycle they will most likely lose a cycle bringing their points to:
contribution(runs = 5) => 48
Hey, not bad we’ve cut them down by by 9 points, but what impact did we have on our own scoring now that we’ve added a second or two to our own cycle times? Let’s see…
contribution(floor(cycles(29))) => 38
Wait, what is this black magic? By slowing down and getting in Team Plowie’s way for 2 seconds per cycle we’ve knocked 9 points from their
score without impacting ours at all. Well, what if we slow them down more? We can add another three seconds to our cycle time and slow them down more right?
So Team Plowie now takes 25 seconds per cycle so they can contribute:
contribution(floor(cycles(25))) => 38
And what is your team’s contribution now that it takes 32 seconds per cycle?
contribution(floor(cycles(32))) => 28
And you’re still down 10 points. What happened? Well, playing defense has diminishing returns after a while. It’s the basic economic concept of opportunity cost. Those 20 seconds you’ve spent harrying Team Plowie to lower their score were seconds you COULD have been scoring. Yes, the numbers in this example were contrived and rounded slightly but they are here to illustrate a point. What we’re pointing out is Defense is Effective but you have to play it right.
We would like to point out that smart defense is very important…a bad defensive team will spend 10 seconds of driving to perhaps slow down the team they are defending by 2 seconds. A good defensive team could, and arguably should slow down an opponent by MORE time than the time spent defending.
Simply put, this is an overly simplified example showing you a proper way to play defense. The concept is sound but it needs some expanding to work in the 3v3 model. That, as well as expanding it to include your various other options and scoring modes, is left as an exercise to the user. Why? Because working out the parameters and options is good practice for when you have to do stuff like this next year.
*And, for the record, Cycles is one of those metrics like shots on goal in hockey. A cycle where you spray your shots wide the minute you cross the mid field line is akin to that lazy bounce off the boards that the goalie leisurely deflects to a player. It’s just padding numbers. *
We’re still alive! We got a little busy building our own rowboats, so we went missing for a while, and now we’re back!
Look Ma! It’s reasonably close to normally distributed!
We didn’t make predictions about Ultimate Ascent, in large part because we didn’t want to make fools of ourselves. We are now quite glad we did that, because we would have made fools of ourselves! After a full day of Ultimate Ascent all across the country, the average (mean) score was 41.4 points, and the median (50th percentile) was 37 points. That is higher than we were expecting, and we are quite pleased to be wrong!
Breaking it out across the 4 phases of game play (formatted as MEAN | MEDIAN)
If you compare this back to our Rebound Fumble analysis, at the end of the first day of Week 1, teams are scoring about twice as many points in autonomous, three times as many points in teleop, and twice as many points in the end game! While climbing isn’t really 1:1 with balancing, thanks to the point structure shooting pretty much is. Whether the higher scoring nature of shooting is from the game (the goals are way better) or teams (robots are better) is an open question at this point. Either way, we think higher scoring games are better for everyone.
Check out the plots below:
While zero is still the most common score, way more frisbees are going goals than basketballs did. Three Alliances have gotten a 48 point “ one less than perfect” score. Who will be the first to do perfect and then perfect plus? Congrats 191, 3838, 145, 862, 3667, 302, 910, 68 and 70! (Bin width is 6, and zero is its own bin)
Again, the teleop scoring distribution is much flatter and goes to much higher scores. Major props to “the outliers” so far, 610, 230, and 1922 at BAE! 610 also took part in the second highest teleop score as well with 229 and 4124.
There have been two 60 point climbing alliances, both of them at BAE. I believe 213 and 1277 got a double 30, and 213 also took part in the other 60 pointer that was a 30+20+10. (bin width 10, zero is its own bin)
For the most part a fairly foul-free game. Still, stay clear of interfering with a hang as there are a few 20 pointers on the board, one alliance wracked up over 100! :( (bin width 3, zero is its own bin)
Very few FRC teams come close to undefeated seasons. Even if you build exceptional robots, the random nature of qualifying matches will occasionally just stack the deck against you. For example, 2056 OP Robotics has won every single regional they have attended (for the bean counters in the audience, that’s 14 in a row!).* However, over the life of the team they’ve “only”** won about 80% of the matches they have participated in. A number of teams have come close to a perfect season, and today we will be discussing one of them.
In 2006 Team 25, Raider Robotix dominated their way through 2 regionals, and most of their way through Championship qualifying rounds before losing their second to last qualifying round (which would lead to Newton ‘06 being a case study in picking strategy). On their way up the ladder they would lose twice more before ultimately being eliminated in the finals.
You’ll notice something very interesting about their robot though. They shot from basically the floor! All you had to do to block them was stand in front of them. And yet it took almost all of the road to Einstein before anyone did this well enough for 25 to lose a match. They also exclusively human loaded, which CD was generally dismissive of until 25 began dominating.
But after you’ve watched some really great robots play two really great matches, there is a lesson here for defensive and offensive robots. If you’re playing offense, the defense might not be that smart. And if you’re playing defense, play smart defense! Great defense often does not involve ramming the other alliance really hard, even if that’s what 90% of defensive robots do.
*In a perfect world if OP Robotics had a 95% chance of winning any regional they entered, there is still only a 50% chance that they would have won all 14! (Binomial Distribution)
**This is also just insanely competitive. I would guess that maybe 10 teams have played at this level at a large number of events for the past 6 years. On the back of this napkin, it looks like 1114 has won around 85% of their matches over the same period, while 111 has won around 70%. That level of sustained quality of play is just mind-boggling.
P: First off, could you tell us a bit about your team?
FRC3313: We are Team 3313 Mechatronics from Alexandria, MN which is about 2 hours northwest of the Twin Cities. We have some engineering support but not as much as many other teams. Our head coach is a math teacher by day, robotics obsessed by night. We have a large group of incredibly dedicated, incredibly awesome students who work their tails off on a consistent basis.
P: How were you founded?
FRC3313: We were founded in 2010 (Breakaway) after another FRC team (unsure of who) came to our high school in 2009 and presented about FIRST Robotics. Our principal and an Alexandria Technical College professor got together to create and run the team. The team was run primarily through the Technical College. The following year, the administration wanted Team 3313 to be based at the high school so they needed a high school teacher to run it. They approached the naive new math teacher, Mr. Bydlon, and he became the head coach. We’ve had a HUUUUUGE learning curve since then.
P: How many students do you have?
FRC3313: We currently have approximately 20 students with 13 males and 7 females. The numbers fluctuate a little bit because we don’t charge an activity fee (only charge for apparel and trips). This leads to our team size being a little up in the air.
P: Primary Sponsors?
FRC3313: A national 3M grant is our primary sponsor for competition fees. A group of local manufacturing companies called the PMMC (Packaging Machine Manufacturers Consortium) is also a major sponsor of our team. Both of those groups are local to our area and we always consider ourselves lucky for all the support financially and with mentors.
P: If you don’t mind talking about it, what sort of strategy did you decide on this year?
FRC3313: We decided on a shooter robot this year.
P: How did you determine this?
FRC3313: We spent time thinking about both climbing and shooting ideas, but the ideas for climbing didn’t make it past our requirements of being able to build the robot. Also, because we know that autonomous is an extremely important aspect of the game, we agreed that a shooter is a safe bet for a robot. As a team, we knew how to deal with the shooting aspect because it would be our second year approaching this task.
P: How do you honestly evaluate your resources?
FRC3313: We don’t spend much time evaluating resources initially. Its mostly just throwing ideas at the wall and those ideas create sparks for the ideas which can become our main robot ideas. We have been focused heavily on fundraising so that our previous building limitations can be greatly reduced. For example, we are building a prototype robot this year and keeping it. Then simultaneously building our competition robot.
P: What trade offs did you have to make based on your resources?
FRC3313: Like I stated earlier, we originally wished to make a climbing robot but we lacked the ability to create what we were imagining. We experience difficulties gaining a consistent engineering mentor; this can be a huge limiting factor especially when trying to attack the pyramid.
P: What process did you use to reach a consensus?
FRC3313: We used a system of needs and wants which was introduced to us at the kick-off for 2012 Season from GoFIRST. The system takes about five needs and five wants of the robot. The needs are normally qualitative: Must be able to build it, must be legal, must be able to move, and then two team decided traits. These were important because they decided the basis for our robot. Our two additional needs were must be able to shoot and must be able to obtain frisbees. We then decided our team’s wants list; these are much more quantitative. Our five were we want to shoot accurately at the three point slot, we want to be able to get frisbees by the 42” loading station, we want to do a ten point hang, we want to be able to adjust our shooter angle, and we want to pick up frisbees. It is not unwise to wait for some prototyping to be done before deciding this list.
P: Prototyping wise, what sort of techniques do you use?
FRC3313: “FAIL FAST” This is a new one for this year. It means test the idea as soon as possible to judge if it works or not. If it does fail, then we can spend our time on ideas that do not fail.
“Don’t worry, it’s just a prototype” means don’t stress about how it looks or exactly where the bolt holes are. We stress that if it’s just a prototype, it’s alright for it to be cheap and for it to fail. While working towards more of a final product, we refine it to have a very similar functionality as the final robot. Turned into a running joke for anything and everything. “Your outfit doesn’t match!” “Don’t worry, it’s just a prototype.”
P: How do you evaluate a prototype’s performance?
FRC3313: As we have discussed in the topic above, we evaluate it by seeing if it meets our needs. We must be able to build it, the mechanism has to be able to pass inspection at the regional, and the robot needs to be able to drive with it on the frame.
P: At what point do you cut your losses?
FRC3313: Mostly when time constrains us. It’s a gut feeling as well as a robot weight, time left in build, and energy to finish feeling. Here’s a quote - “Truly successful decision-making relies on a balance between deliberate and instinctive thinking” - Malcolm Gladwell
P: You have quite an active social media component. Do you feel being active in the online community has helped you?
FRC3313: Absolutely! It provides such a boost for us to see teams doing similar designs or totally different designs. We LOVE talking to other teams and seeing how they think and then applying some of those principles to our team. That’s what makes FIRST Robotics such an awesome, amazing experience. It’s competition mixed with friendship. It’s being the best while also beating the best.
P: What resources do you think are most useful?
FRC3313: The rule book is the best resource. We feel that FIRST did a great job this year of making a rule book that is simpler to understand and use. There are also the Team Updates of the rule book that are important to see.
If you can get past the technical jargon (and slightly bragging nature) of Chief Delphi, it’s frankly the best resource out there for technical robot questions. You have to deal with a little bit of condescension every once in a while but, again, for a robot related question that is your main place.
The TwentyFour blog has totally changed the way our team thinks about game strategy. Conversation our team had about a toaster, “Where have you been all of our FIRST lives???”
P: What resources would you like to see more of?
FRC3313: We would like to see more resources about how to get in touch with other teams. The first year or two of our program we felt very isolated and did not know who to ask a question to or how. We just kind of figured it out on our own but we hope other teams can kind of skip over that struggle.
Especially for out state teams (teams not near a major metropolis), how do you do some creative fundraising and get your community more involved. Smaller towns make these situations harder.
P: Is being so open about your prototypes new to you?
FRC3313: Absolutely not. We have been uploading YouTube videos since Mr. Bydlon became our head coach in 2011. We did a similar process last year with our Rebound Rumble prototypes on YouTube. The main difference this year is the ability of Twitter, our blog, and Tumblr to have those videos spread quicker and create more discussion.
P: Do you feel it has/will help or hinder your competitiveness? .
FRC3313: We believe if our videos can help bring rookie teams and teams with limited resources to our level at the competition we can create a more active game year after year.
One of the main pillars of FIRST Robotics is coopertition. Sharing our videos and our struggles and ideas totally fits in with what we feel FIRST is all about.
Alliances are selected based on robot performance and team notoriety. Getting our name out there as a team with solid design principles and a willingness to help, we feel, helps us get picked if the Regional does not go the way we wanted it to.
P: What teams do you strive to emulate?
FRC3313: We feel we are a special case for an FRC team. We really only have one full time mentor and he’s a math teacher with up to 2 years ago, ZERO engineering experience but a lot of social media experience. That puts our team in a totally different frame of mind. We have no trade secrets. We have some great, creative ideas for robots and for connecting FRC teams but none of them, we feel, are ground breaking. We want all teams to achieve success (and we don’t just say that). So while we see it as sort of a cop out, we don’t really emulate any teams because no other team is quite like us. Hopefully, all teams get to that point. They get to the point where they can listen and take knowledge and ideas from other teams but still maintain their own team identity.
P: What local teams are inspirations to you?
FRC3313: Many of the larger Twin Cities teams like KingTec, Blue Twilight, and Green Machine were and still are inspirations to us. We are envious of the large programs (multiple FTC and tons of FLL teams), large budgets, team organization, and outreach efforts they all do. We are working with all of our might to create an environment like that on an out-state team. This leads us to do more creative activities and drive to trainings and meetings.
P: Do you have any problem with the fact that you were just interviewed by a toaster that has no discernable appendages with which to type this?
FRC3313: Not if it makes me toast; I want toast… give me jelly too please.
This is how the programmers feel: the nintoaster
In Rebound Rumble, the top, middle, and bottom goals were worth 6, 5, and 4 points respectively. Each robot started the match with up to 2 preloaded balls. There were also balls that started the match on the bridge, and top tier alliances would occasionally score those.
In Ultimate Ascent, the top, middle, and bottom goals are worth 6, 4, and 2 points respectively. Each robot can start the match with 2 frisbees if you want a close shot, and 3 if you are willing to shoot from farther away. There are also frisbees on the ground, and if history holds a small percentage of teams will also pick these up in autonomous.
It is very clear from the histogram below that teams were aiming predominantly for the high goal and that lots of balls were not being shot or were being missed. It would be very interesting to see some actual scouting data to see what percentage of the field was not shooting, versus the percentage that was shooting and missing.
But we already knew teams weren’t scoring a ton of points in autonomous mode. The important question is how do teams to better this year? We think the answer is to get right up to the goal and shoot for the insanely large target (the 2 pt goal). Maximizing points is all about maximizing expected value, that is the reward of a task times the probability of success. We would guess that most teams maximize that number by doing something easy, even if it is worth less points per completion. Consider the following:
Inside Auto Zone / Middle Goal (2 frisbees) x (80% success rate) x (4 pts) = 6.4 pts
Outside Auto Zone / High Goal (3 frisbees) x (30% success rate) x (6 pts) = 5.4 pts
Is that a contrived example we made up to fit our point? We’re going to shamelessly admit that it is. The math for your team will depend entirely on your scoring mechanism testing. However, in our years of FIRST we have seen much more aiming high and missing than aiming lower and scoring.
I wanna be the very best Like no one ever was - Ash Ketchum
Most of our predictions have been decidedly gloomy. History shows that the typical robot just doesn’t score many points. It might average out to one score in autonomous, one in teleop, and an end game contribution every few matches. We like to talk about the typical robot because our target audience is the team that usually doesn’t play in the elimination tournament. Our goal is to show them that really a very simple robot can be quite competitive. In other words, the bar is set quite low.
It is worth noting though that elite teams are significantly better than average teams. Because we don’t want to explain OPR and we think alliance scores do a good job of demonstrating this, we are going to use them. In Rebound Rumble, the 95th percentile alliance scored about 3x as many points as the median alliance (47 pts compared to 17). If you look back to other games you find that this is not atypical. We think the multiplier may be even larger this year as climbing the pyramid is quite hard. In Rebound Rumble the 75th percentile alliance is over 1.5x as good as the median (28 vs. 17). In absolute terms this may not be huge, but for many teams to double the number of points they score can actually be quite difficult.
We think that building an FRC robot is just like a good FRC game. It isn’t that hard to be average, it’s difficult to be good, and it’s really quite difficult to be great. Deciding which level your team wants to compete on is obviously up to your team, but if you have traditionally struggled to be average, then aiming to skip straight to good may be a bridge too far.
In the previous article, we found that during the typical Rebound Rumble qualifying event mean alliance scores would increase roughly 50% from the first to final match. While that is impressive, the increase in mean score doesn’t tell us a terribly large amount about which alliances are improving. It could be driven by the the small number of very good alliances getting significantly better, or it might be driven by the middle of the pack getting somewhat better. To see which teams were getting better, we tracked the quartiles (25th percentile, 50th percentile, 75th percentile) over the course of all events. The evolution of the median (50th percentile) is shown below.
We found in the last article that while alliances get significantly better at scoring points, they get only slightly better at avoiding penalties. We assumed that this meant the bottom tier of alliances that is more likely to get penalties didn’t improve over the course of an event. It turns out we were wrong!
The 25th percentile, 50th percentile (median), and 75th percentile alliance all improve dramatically over the course of the event. The 25th percentile alliance got 130% better over a typical event, the 50th percentile alliance improved 70%, and the 75th percentile alliance improved roughly 40%. In absolutes, the 25th percentile alliance scored 6 additional points by alliance selection, the 50th percentile alliance scored 8.5 additional points, and the 75th percentile alliance scored 9 additional points.
Perhaps the most interesting factoid to come out of that is that while the 50% percentile and 75th percentile alliance learned to score the same additional number of points over the course of the match, the 75th percentile alliance showed up to the event capable of scoring an additional 10 points. Interestingly enough if you repeat the exercise for the 80th, 90th, and 95th percentile those alliances also only get roughly 9 points better over the course of the event (but their initial scores increase). We are not sure what causes this – but had you asked us for our prediction we would have gotten this one wrong. Is there a certain level of robot ability that once you break all drivers can reasonably improve? Or are the really top teams in the 90th percentile alliance not improving but their lower scoring partners are? Also worth noting that R^2 starts to drop off above the 80th percentile.
What is R^2? In extremely rough terms it says how well the model fits the data. A number near 1.0 means that you (probably…) have a good fit, a number near 0 means you most likely have a bad one. We highly recommend you take a statistics class (or a couple of them!) to learn how frustratingly complicated the intricacies of this can get. Particularly Econometrics. It was probably the most interesting class I took in college, and I got the nerd cred of learning how to use Stata, which runs the FiveThirtyEight model.
Moral of the story is that if you want your friends and neighbors to come check out “that robot thing”, they will see the highest scoring (and hopefully most fun) matches on the final day of the tournament. And since scores in all quartiles improve as alliances get more practice, we are looking forward to the day that all teams get to play in district style events where they see more matches.
We also think this data shows that most teams benefit significantly from practice. So finish your robots early and practice how to drive ‘em like you stole 'em.
On a totally unrelated note, we would encourage teams that must yell something as they walk through the pits to say 'Rowboat’ instead of 'Robot.’ And maybe people would actually clear out of the way if you were dragging a rowboat on a trailer…
If there is a universally “good” weekend in the FIRST Robotics Competition it is the first weekend. You’ve come in hot off your brainstorming and strategy sessions, you might have a kit-bot up and running, and you’ve probably got a few mock ups in the shop that look REALLY promising. That’s awesome!
Lets say you read our Rebound Fumble: Aim Low post and decided to build a simple robot. You might have decided in your strategy selection that you were only going to be a 2 pt goal robot this year, or maybe decided you were only going to hang from the first level of the pyramid. But now you’ve built these plywood and polycord drill driven prototypes, and they look really promising. So you ask yourself, “Is the 30 point hang really that much harder? This looks so easy, if we don’t do it we’re probably toast…”
If history is any indication, YES! It is that much harder!
Ultimate Ascent and Rebound Rumble are not the same game. But they do have plenty of similarities that lets us compare them. Both games involve throwing something into a 1, 2, or 3 point goal that is worth more points in autonomous. There are safe zones if you want to shoot from further away, defense is allowed up close. Teams are limited to a relatively small number of game pieces that you must retrieve from a human or pick up from the floor. The similarities break down in the end game, but for our sake lets say they aren’t that different for the median robot. There are lots of differences we aren’t discussing, but we don’t think these change how many scoring actions teams will be able to accomplish in a typical match.
In Hybrid in 2012 the median alliance scored 6 points. That’s one score in the high goal per alliance.
In Teleop in 2012 the median alliance scored scored 4 points. That’s two scores in the middle goal, or 1 high/1 low per alliance.
On the Bridge in 2012, the median alliance scored 0 points. The mean scored about 5. In the last hanging game (2010) where a hang was worth 2/3 of the median match score, only 30% of qualifying matches ended with one or more robots hanging! And we feel hanging in 2010 was easier than climbing will be this year.
We don’t mean to be Debbie Downer, but we feel it’s important to frame your outlook on the competition in realistic terms. If you think you have to be able to average 30 points a match to be a 2nd round pick, then maybe that 30 point climber is the minimum competitive concept. But if that turns out to be the case, we will eat our power cord.
“How will we know it’s us without our past?” – John Steinbeck
And if you think we are pessimistic, you might be right. We think we’re being realistic, but we would love nothing more than for scores to blow our estimates out of the water. And if most of us built a simple effective robot, they probably would. :)
The simple answer is yes. The more complicated more useful follow-on question is “Which teams get better and how?”
If you take a moving average across all qualifying matches at qualifying events (we ignore MAR, MSC, & CMP), you find that during the first qualifying match teams average about 15 points. By the time alliance selection rolls around they are averaging around 22. That’s a nearly 50% increase over the course of the average event!
You can see this in the linear regression above. Plotted in black is a 30 match moving average. Plotted in red on top of that is a linear regression through the moving average points. Since different events are different lengths we normalized each event by number of matches and plotted them all from zero (first qualifying match) to 1 (last qualifying match before alliance selection).
Now that we know that the average team improves, there are a couple of follow on questions that we think are important. Thanks to the @FRCFMS twitter feed, the relatively easy one to answer is in which areas teams improve. Relative to the first match, teams improved roughly 50% in autonomous mode, 40% in teleop, 100% at balancing (for regular points), 80% (for coop points), and yet are only 10% better at avoiding penalties.
The harder question that we are not yet ready to answer is which teams improved. Robot goodness is not a normal distribution. By OPR in Logomotion, about 50% of robots were worth <5 points. Meanwhile there were a small handful of robots that would regularly post 40+ points. For the most entertaining matches, we would hope that the middle to lower end of the field is driving most of the improvement. However, if it happens that most of that improvement is being driven by teams that are already very good getting even better then blow-out qualifying matches will become even more lopsided. We’re not ready to issue a final verdict on that topic – we’ve got some more math to do.
