These lessons are derived from working towards the Amazon Picking Challenge
(APC) between September 2015 and June 2016. I had an opportunity to interact
with other teams during the final competition at RoboCup, Leipzig, Germany.
Our system consisted of modules for perception, grasping and motion planning
tied together using ROS and smash state controller. I think our strength was in
the simplicity of system.
- Should have sat down and written a list of software code style rules Have
atleast a basic code review system in place
- Not abusing ros - It was meant to be
a message passing framework. We used it everywhere - even when passing data as a
function parameter would have worked more wellthough out architecture, naming
conventions Lesser ros. More cpp.
- Code freeze 1 month before UBonn - Nimbro
team had nice software. They had some superb rqt plugins which would have made
testing a lot easier. Lot of prior robocup soccer experience
Better testing architecture. Like really solid. Break down into smaller
components. Decide interface. Test independently. integration would have been
In our case we had a perception, planning and grasping. Should have built
enough infra code for testing: trajectory
Introspection tool - to help us visualize grasps
Better state machine: A lot of time was last massaging SMACH to pass messages
around. It would have been more efficient to write our own.
Trajectory caching: Better way to cache and playback trajectories for
inspecting, better planning speed.
Focus on score before hand - read into competition rule. Better to get 8 than
all and fail.
Interestingly, no team used in-bin collision checking. Convex hull for detecting
collisions as it is faster.
Robots & Actuators
Early on, we made a decision to switch from the PR2 (which was already available
to us) to a UR5. The PR2 had a moving base and a telescopic spine - these would
have made planning more complex and slower. Universal Robots loaned us a UR5.
The UR5 was an amazing robot - it had tight ROS integration and was force
compliant. We were up and running in no time.
However, with the UR5, we had reachability issues - We could not easily plan
paths to the corners of all the shelf bins as our end-effector greatly limited
workspace. At the competition, we were the only team using the UR5. Many of the
other teams had a UR10 robot with a linearly actuated arm - resulting in a much
larger workspace, better reachability and easier planning. Next time, UR10 with
a linearly actuated arm would be the way to go!
Due to the task, items and grasping complexity, most APC teams prefer to use
suction grasping end-effectors. This obviates the need for accurate perception
and force-closure for finger grasping.
The MIT team’s end-effector was unique and large - it had grasping arms and
suction. The Delft teams’ was more elegant - suction and a clamp. These would
have enabled them to grasp even the 3lb dumbbell and black mesh pencil cup.
Should not have finalized and put the end-effector design on hold. We saw a lot
of teams having smaller suction cups. Better suction system.
We used pretty basic electronics - Arduino, AC relays, pressure sensors and a
- Controlled lighting: The top teams had a lighting setup mounted inside their workspace. This allows
them better control for using CNNs for item identification. It would also allow
them to transform the perception data to match lighting conditions of their
training data set. We didn’t have one.
We were probably the only team using the bulky Kinect 2. Most teams used a
smaller Creative Auto or Intel Realsense 3D sensors. A smaller sensor mounted on
our arm would have allowed us to mount multiple sensors, giving better
recongition. Would have also made in-bin planning much easier.
- Sensor selection: A huge drawback was that Kinect2 used time of flight technology compared to the
other Structured Light sensors. // put links. The Halogens lamps mounted on the
ceiling emitted IR radiation which interfered with our sensors and they had to
be removed. Most teams used creative. Halogen Lamps.
A smaller sensor with a shorter range would have helped increase the search
space and grasp items in tighter spaces
- Multiple images + pointcloud fusion - We should have taken multiple images of the items in the bin,
giving us better recognition.
** Shipping & Handling **
- Plan to ship the robot in advance due to Murphy’s law
- Have some backup systems and be mentally prepared for the worst
packing - send things before. more thought. Things which will not work
- Better planning for transportation - If anything goes wrong, it will cause
stress and in our hurry, we would overloop some other important factor. Me
forgetting about transformer is a good example.
Teams - this is the first time I’m working in a 5-person team for extended
periods of time.
worked with awesome team. Learnt a bit of mech and factors that go into
designing things. Prototyping. programming funda.
Engineering workflow - I realized that the MRSD project course was structured
very similar to the workflow in an actual company.
Amazon Picking Challenge is a competition for designing a robot system to
complement the Kiva automated warehouse shelving system. The workspace consists of a
12-bin shelf and a storage box. Each shelf bin can have multiple items placed in
any configure with partial occlusions. A task list is provided of the required
items. The goal is to pick the correct item from the bin and place it in the
As a part of MRSD course project, our 5-person team worked on the project from
September 2015 to May 2016. You can see our course project webpage at
During this period, we designed the system from the ground up.
We were mentored by Prof. Maxim Likhachev from the Search Based Planning Lab at CMU. We
intially developed the system on a Willow Garage PR2 robot but found its
reachability too limited for our task - in order to reach all the regions in our
shelf, we would have had to change the telescopic spine height and move the
robot, both of which would have added a layer of planning complexity and
increased execution time. Universal Robots was kind enough to lend us a UR5 arm.
Migrating our code to the UR5 was a breeze. We were super psyched when we were
shortlisted as finalists in February 2016.
During summer, 3 members of our team started their internships in California. We
would work on this project in the evenings after work and during weekends.
Day -1: June 27
We disassembled and packaged the robot, mounting frame, controller, items and
the two computers. It was 450lbs between the four of us. We had decided to take
the robot as a checked-in luggage instead of shipping it before as it would buy
us more development time.
Day 0: June 28
We were scheduled to fly out of Pittsburgh in the afternoon and reach Berlin on
June 29, just in time for the robot setup and testing day. As Murphy’s law would
have it, our connecting flight to Newark got cancelled twice due to ATC issues.
We spent the night camping at the airport as the next available flight was in
Day 1: June 29
Spent in airports and air. Our quickest route to Berlin was to fly backwards to
Chicago, go to Frankfurt and then to Berlin. We were super concerned about our
robot cargo not making it on time.
Day 2: June 30
We finally landed in Berlin! We were delayed by 28 hours. We rented a Peugot and
raced to Leipzig on the Autobahn. Rick and Bhatia had already got there before
Today was the practice run day where we get and item arrangement closest to the
competition - a mock competition.We saw a lot of teams tweaking their perception
system to adjust to the warehouse lighting.
We turn up at the conference venue and frantically unpacked and setup our robot.
Much to our chagrin, we found our computer cases smashed with dangling CPU
cooler, RAM and the power supply cables. Thankfully, we could patch it up
Then our linear actuator wasn’t being powered properly, our linear power supply
wasn’t working and our shop-vac suction system was super loud. Power supply
conked. Then we realized we were powering 110VAC 60hz equipment with 240VAC
50Hz. In my hurry, I mistook the power junction box to be a 110VAC-240VAC. We
had to run out and get a new Shop-vac. We then rigged a power source for the
linear actuator using a drill battery!
Kinect was giving very noisy pointclouds. Later we learnt it was due to the
Halogen lamps which emit IR. Our Kinect was 2 - using Time of Flight instead of
structured light which was more immune.
Day 3: July 1st - Stow task finals
We finally got our system up and running in the early hours of the final event.
Due to an error, we were giving the wrong JSON file. So we had to run twice.
First time, we stowed 7 items we got 88. Second item, we stored 8 items but
misidentified one, fetching us 77.
We were placed 6th in the stowage competiton out of the 16 finalists.
The Delft team aced it - they had a really tight integration and a fast
perception system. They completed the task with 6 minutes to spare. They were
far placed first and were far ahead of the second.
Day 4: July 2nd - Picking task finals
We made a decision to modify the grasping logic for certain items in fear of
dropping. We had a lot of time and were looping between 2 shelves. 7/12 items
4 items - 33 points
We were placed 7th in picking.
I skipped the evening reception banquet and instead took off to explore Liepzig.
Day 5: July 3rd - Demo and media day
Each team had a 15 minute presentation. And there were a lot of people asking
questions about sub-systems. There were a few TV crews interviewing.
There was this kid from UAE who must have been in 5th or 6th grade. He was
asking the MIT team member what language his robot was programmed in and was
noting it down in Arabic!
Day 4: July 4th - Return to pittsburgh
Got back just in time to catch the 4th of July fireworks!
Overall, it was an awesome experience and we were very happy with our
performance. We would like to thank John Dolan, Dimi and Maxim for their
support throughout this competition.
Team HARP had a successful run! We would be handing over the project to the next
year’s MRSD team.
I’m sitting inside some really fancy hall inside the New York Public Library, I realize I have visited NYC thrice and spent more than a week but haven’t visited most places.
This library is breathtaking. Beautiful architecture. Wonderful place to sit and write stuff. It has the perfect ambience.
A key advantage of living in Pittsburgh is the proximity to major cities in the East Coast and Midwest. Never would have I thought I’d live just 6 hours away from Big Apple. Thanks to bus connectivity, it sorta feels like travelling overnight from Chennai to Trichy!
I shifted to Hyderabad on
3rd July, 2014 and I'm really excited to be joining as an intern at IIIT.
My initial work is on TORCS - The Open Racing Car Simulator which allows
you to program your own 'robot' driver to navigate the car around the
race-track. I'll be porting over a few existing driving control algorithms
I'm just getting starting and I'm studying the tutorials at
be uploading some of the code to
I think it's time I start maintaining a dev-log.