Lessons Learned - Amazon Picking Challenge 2016

These lessons are derived from working towards the Amazon Picking Challenge (APC) between September 2015 and June 2016. I had an opportunity to interact with other teams during the final competition at RoboCup, Leipzig, Germany.

Our system consisted of modules for perception, grasping and motion planning tied together using ROS and smash state controller. I think our strength was in the simplicity of system.

Software Codebase

  • Should have sat down and written a list of software code style rules Have atleast a basic code review system in place
  • Not abusing ros - It was meant to be a message passing framework. We used it everywhere - even when passing data as a function parameter would have worked more wellthough out architecture, naming conventions Lesser ros. More cpp.
  • Code freeze 1 month before UBonn - Nimbro team had nice software. They had some superb rqt plugins which would have made testing a lot easier. Lot of prior robocup soccer experience


  • Better testing architecture. Like really solid. Break down into smaller components. Decide interface. Test independently. integration would have been awesome

  • In our case we had a perception, planning and grasping. Should have built enough infra code for testing: trajectory

  • Introspection tool - to help us visualize grasps

  • Better state machine: A lot of time was last massaging SMACH to pass messages around. It would have been more efficient to write our own.

Motion Planning

  • Trajectory caching: Better way to cache and playback trajectories for inspecting, better planning speed.

  • Focus on score before hand - read into competition rule. Better to get 8 than all and fail.

Interestingly, no team used in-bin collision checking. Convex hull for detecting collisions as it is faster.

Robots & Actuators

Early on, we made a decision to switch from the PR2 (which was already available to us) to a UR5. The PR2 had a moving base and a telescopic spine - these would have made planning more complex and slower. Universal Robots loaned us a UR5.

The UR5 was an amazing robot - it had tight ROS integration and was force compliant. We were up and running in no time.

However, with the UR5, we had reachability issues - We could not easily plan paths to the corners of all the shelf bins as our end-effector greatly limited workspace. At the competition, we were the only team using the UR5. Many of the other teams had a UR10 robot with a linearly actuated arm - resulting in a much larger workspace, better reachability and easier planning. Next time, UR10 with a linearly actuated arm would be the way to go!

End effector

Due to the task, items and grasping complexity, most APC teams prefer to use suction grasping end-effectors. This obviates the need for accurate perception and force-closure for finger grasping.

The MIT team’s end-effector was unique and large - it had grasping arms and suction. The Delft teams’ was more elegant - suction and a clamp. These would have enabled them to grasp even the 3lb dumbbell and black mesh pencil cup.

Should not have finalized and put the end-effector design on hold. We saw a lot of teams having smaller suction cups. Better suction system.

We used pretty basic electronics - Arduino, AC relays, pressure sensors and a linear actuator.


  • Controlled lighting: The top teams had a lighting setup mounted inside their workspace. This allows them better control for using CNNs for item identification. It would also allow them to transform the perception data to match lighting conditions of their training data set. We didn’t have one.

We were probably the only team using the bulky Kinect 2. Most teams used a smaller Creative Auto or Intel Realsense 3D sensors. A smaller sensor mounted on our arm would have allowed us to mount multiple sensors, giving better recongition. Would have also made in-bin planning much easier.

  • Sensor selection: A huge drawback was that Kinect2 used time of flight technology compared to the other Structured Light sensors. // put links. The Halogens lamps mounted on the ceiling emitted IR radiation which interfered with our sensors and they had to be removed. Most teams used creative. Halogen Lamps.

A smaller sensor with a shorter range would have helped increase the search space and grasp items in tighter spaces

  • Multiple images + pointcloud fusion - We should have taken multiple images of the items in the bin, giving us better recognition.

** Shipping & Handling **

  • Plan to ship the robot in advance due to Murphy’s law
  • Have some backup systems and be mentally prepared for the worst packing - send things before. more thought. Things which will not work
  • Better planning for transportation - If anything goes wrong, it will cause stress and in our hurry, we would overloop some other important factor. Me forgetting about transformer is a good example.

Other Lessons

Teams - this is the first time I’m working in a 5-person team for extended periods of time.

worked with awesome team. Learnt a bit of mech and factors that go into designing things. Prototyping. programming funda.

Engineering workflow - I realized that the MRSD project course was structured very similar to the workflow in an actual company.