Trouble Running Single Finger Test - TriFinger Robot

[Ross_Worobel]

Hello everyone!

I am building the TriFinger robot and the current status of my build is that I have a fully built single finger module. I have the correct CAN cards, everything is plugged in, connected, etc. I have all of the other components of the other two fingers, but first I wanted to test my first finger was working.

My goal is to run the “Single Finger Test” in order to test that this finger is working as intended.

Using the documentation provided here:
https://open-dynamic-robot-initiative.github.io/code_documentation/robot_fingers/docs/doxygen/html/index.html

I have been able to do the following:

1. Install Treep
2. Install Singularity
3. Download the trifinger_base.sif file

At this point, following the documentation, I attempted to run the following command, which caused the related issue seen on screen:

To alleviate this issue, I ran the following slightly altered command, which produces the next issue which I haven’t been able to solve:

The issue is the warning:
“Skipping package ‘robot_fingers’ because it has an unsupported package build type: ‘ament_cmake’”

I’ve tried the following things:

1. Attempting to build it outside of the Singularity container using ROS noetic, melodic, and foxy.
   All of these produced the same error, or worse errors.
2. General Googling about ‘ament’
   Nothing helped here
3. Replacing every instance of “ament_cmake” in the project with “cmake” or “catkin”
   No longer runs, produces a much worse error

I just don’t quite know what I’m supposed to do here.
I’m very exited to get my finger module working and I would love any input from the team!

Thank you,
Ross

[felixwidmaier]

Hello Ross,

I’m sorry, apparently the automated documentation build was not running anymore recently. I’ll try to fix that.

For the meantime: We moved to ROS 2 and thus are using colcon instead of catkin now for building. Download the updated Singularity image. With this follow the documentation but use colcon build instead of catbuild.

When the build is finished you can do a first test by running the following inside the container:

Singularity> source install/local_setup.bash
Singularity> ros2 run robot_fingers single_finger_test

This should hold the finger at the current position and display all sensor data.

[Ross_Worobel]

Awesome, thank you for the quick reply!
I’m not super familiar with ROS2 so the errors were not throwing off alarm bells to me about versioning.

I’ll be in the lab in a couple hours to test this out!!

[Ross_Worobel]

Hi Felix,

Unfortunately I’m still not able to get the package running. After downloading the new .sif file, opening it in Singularity, and attempting to “colcon build”, I get the following error about pybind11:

If I try to build this project outside of Singularity, it appears (??) to work – there’s no errors or anything, and I can go into the “build/” folder and find the “robot_fingers/” folder, but I can’t actually run it, so I don’t know what is happening there either:

Another thing I tried was to install pybind11 locally, then, within Singularity, set the the “pybind11_DIR” variable, which still errors with this:

I figured this was the wrong way to approach the problem so I stopped going down this path, though.

I would appreciate any input you have!!
Thanks again,
Ross

[felixwidmaier]

It is weird that “robot_fingers” is the first package that is starting when you call colcon build. It seems you only have this one package in your workspace? You can check with ls workspace/source. It should look like this:

blmc_drivers       real_time_tools           tags
googletest         robot_fingers             time_series
master-board       robot_interfaces          trifinger_cameras
mpi_cmake_modules  robot_properties_fingers  trifinger_object_tracking
package_template   serialization_utils       trifinger_simulation
pybind11           shared_memory             yaml_utils
pybind11_opencv    signal_handler

I have an idea what could be the cause. Is it possible that you cloned the project with treep --clone robot_fingers instead of treep --clone ROBOT_FINGERS? There is an important difference: lower case will only clone the “robot_fingers” package while upper case will clone the “robot_fingers” project (including all the other packages). You can simply re-run the clone command if this is indeed the case.

Regarding the attempt to build without Singularity: This is odd, colcon build should have some output in any case.
However, independent of that after building the workspace, you have to call source install/local_setup.bash so that packages of the workspace can be found. Note that this is also needed when using Singularity.

For a first test if the build is good you can use the following command which does not need any actual hardware to be connected:

ros2 run robot_fingers demo_fake_finger

If all is good, this should run and print increasing “Position” values until you stop it.

[Ross_Worobel]

Felix, you were correct.

It was because I needed to run the --clone ROBOT_FINGERS, then colcon build, then the sourcing etc.
I did get the demo_fake_finger to work also. The single_finger_test is currently working as well.

Thank you very much for your help!!
Ross

[felixwidmaier]

I’m glad I could help.
The updated documentation is now accessible here: http://people.tuebingen.mpg.de/mpi-is-software/robotfingers/docs/robot_fingers. It is still work in progress, though. I’ll try to improve it/add more content but if anything is unclear, please don’t hesitate to ask.

[Ross_Worobel]

Hi Felix,

I have another question. So I was able to get that software up and running, but I’m still working to get the finger recognizable by the computer. I did find the “initialize_CAN_bus” script, and ran that as well. I can now see the CAN ports as seen here:

I can start the software up and run it, but then I see the message “waiting for board and motors to be ready”. None of the LEDs are illuminated unless I plug the board into a computer via USB. I think the issue might be that that I haven’t put the correct software on the TI boards, but I’m not sure.

Following the TI documentation, I plugged the boards into a Windows computer, let Windows search and install the drivers. At this point, the board has a green and blue LED illuminated rather than only the green LED which was illuminated before.

The part that I’m a bit stuck on is seen on this page:

open-dynamic-robot-initiative/open_robot_actuator_hardware/blob/master/electronics/ti_electronics/README.md#texas-instruments-evaluation-board-electronics

Near the bottom under the “CAN Control” section:

I know how to change the switches – that part is fine.
But I’m a bit confused on the flashing aspect and what software I’m supposed to flash onto the boards. I have downloaded the various TI software (ControlSuite, Code Composer Studio, C2000, MotorWare) and looked on them, but there isn’t an obvious script or code that I should flash onto the boards, or an obvious way to even do the flashing.

So I guess my question is, first, how am I supposed to flash the Launchpads, and with what software? And secondly, do you think there is some other issue which is unrelated to this that is causing my computer to not be able to recognize that the boards are plugged in?

Thanks again for being so helpful!!
Ross

[felixwidmaier]

Hi Ross,

sorry for the late response. You indeed need to flash the boards with a specific firmware and it seems we didn’t have the full documentation for the CAN-version of the firmware public so far.

The firmware you need can be found here: mw_dual_motor_torque_ctrl
I just added build/flash instructions to the README, please let me know if anything is unclear.

[Ross_Worobel]

Hi Felix,

Absolutely no problem at all. I highly appreciate the responses regardless of how long they take – I would be stuck without them!!

I did figure this out after a while of troubleshooting today. I discovered that the issue was that I had already installed the booster packs and removed the jumpers – simply switching the third switch to “ON” was throwing cryptic memory errors. I’m not sure exactly what is required, but to flash the boards, I returned them to their out-of-the-box configuration: no booster packs, jumpers JP1,2,3,4,5,7 installed, JP6 not installed. Once I did this, I was able to follow the instructions and flash the boards.

Another thing of note is that I did this all using CCS 10.3.1 rather than 9.1.0 – it all worked exactly the same.

Tomorrow I will go to the lab and test to see if everything is working. I hope to have no issues and run the single_finger_test!!

[Ross_Worobel]

Hi Felix,

Unfortunately I was not able to get the single_finger_test running. I just want to provide a quick overview of what I have done to see if there are any obvious issues that you see. This is a long post so please feel free to take some time to respond!!

The current situation:
I have a fully built single finger:

(Obviously, there’s a bit of wire management that I need to do, but I’ve been moving everything around trying to get it all working)

The “upper leg” module is connected to the bottom microcontroller (the microcontroller with only 1 Booster Pack). The “hip fe” module is connected to Booster Pack A on the top microcontroller, and the “hip aa” module is connected to Booster Pack B on the top microcontroller. On both microcontrollers, only jumpers JP3, JP6, and JP7 are installed. Both microcontrollers are set to the “ON-ON-OFF” position. Both microcontrollers have been successfully flashed with the provided software using the instructions within the Github page. Both motors are connected to a nearby lab power supply supplying around 3.3V at around .3 A:

When turning the power supply on, the green D1 LEDs are illuminated on both boards:

Playing with the amperage supplied can also turn on the blue D10 LEDs as well:

If I set the voltage and/or amperage too high, the red D9 LEDs will illuminate:

I don’t know what any of these mean, but I generally assumed that the green was the best and left it in that configuration.

On all three Booster Packs, the red “nFault” LED, and the green “D3 3.3V” LED are illuminated:

I assumed the “nFault” LED was not supposed to be illuminated, but the official documentation says that “A fault may appear [after turning on your power supply] on the nFAULT LED. This is normal and should be cleared when the status registers are read or EN_GATE is taken HIGH.”, so I didn’t look too far into it.

Both microcontrollers are connected via CAN to a temporary computer that I built for this project:

The components on this computer are very bad and around 10+ years old (core i3-2100, 8gb ddr3, no gpu, etc.) but I assume this does not matter. On the SSD is Ubuntu 20.04. The CAN ports have been initialized by running the “initialize_can_bus.sh” script provided in another Discourse thread, which allows me to see that they are being recognized by the computer.

With your help, on the computer, I’ve been able to fully get the Singularity image working and the demo to run as seen here:

You can see the warning “Warning this thread is not going to be real time.” which I assume is a warning notifying me that I have no yet configured my Ubuntu install to be an RT_PREEMPT installation – I attempted doing this but had some issues which led me to stop trying for the time being. Seeing as the demo worked, I assumed this meant that it would be possible to run the single_finger_test, just without real-time results.

Next, I try to run the single_finger_test and see the following:

One thing to note here is that there was a code update on Mar 1 which set the value of “homing_with_index” to “false” which is not compatible with my codebase – it threw some error that I needed to use “homing_method” instead, which I set to “none” and it seemed to work.

After this, I get the message “waiting for board and motors to be ready” as seen in the screenshot. I’ve tried various things that I don’t really know if they matter such as: changing the CAN ports for each microcontroller, swapping the microcontrollers out for different ones (I have 6 fully prepared ones), connecting the different modules to different power sources (not sure if there is a required order for this, but I assume not), and various other software things (restarting my computer, etc.) and I just can’t seem to get it working…

I’m not sure what I’m doing wrong, and what should do next but I would love your input.

Once again, thank you very much for designing this awesome robot and helping me build it!!
Ross

[felixwidmaier]

Hi Ross,

some thoughts:

Real Time Linux

The RT_PREEMPT patch should indeed not be needed for a first test, so that should be fine.

Power Supply

I think the power supply should actually be set to 24 V (unless I’m missing some difference in our setups). @fgrimminger can you confirm?

Connection of Motors to the Boards

Regarding the connection of motors to the boards: They need to be connected in a specific order (not sure if this might be missing in the public documentation?)

Joint	Board/Connector
lower (= upper leg?)	Board 1 / Motor 2
middle (= hip fe?)	Board 1 / Motor 1
upper (= hip aa?)	Board 2 / Motor 1

I assume that “Motor 1” corresponds to Booster Pack A and “Motor 2” to Pack B.
@fgrimminger Can you double-check that I translated upper/middle/lower to the correct joints?

I don’t think that this is the reason for the current issues, though. The software should still be able to connect to the robot it would just have the joints in a wrong order.

Checking CAN Port Order

I’m using the following trick to find out which CAN port corresponds to which name (can0, can1, …):

1. Initialise the CAN bus but do not connect anything
2. In a terminal run watch netstat -i
3. Power on the robot and connect one CAN plug
4. Check the output in the terminal. For one of the ports some of the numbers should increase. That’s the port you are currently connected to.
5. Once identified, you can add labels to the ports. In my experience they don’t change as long as you don’t add/remove any CAN carts in your computer.

But again, if they are connected in the wrong order, it should still be able to connect and only result in a wrong joints order in the software.

[fgrimminger]

Hi Ross,

thanks for the pictures and your description.

The supply voltage is too low - I would increase it to 12V or 24V.
(make sure that the power jumpers are removed if you connect the cards via usb again)

I observe the same behavior here when I reduce the voltage below 5V.

Below is a screenshot with the motor assignment that we use - I will also document this on GitHub.

The naming of the degrees of freedom was changed for the fingers - below is a screenshot with the convention we use.
It’s as @felixwidmaier described in his reply.

Best. Felix

[Ross_Worobel]

Thank you, and @felixwidmaier, for your detailed answers!! I’m going to head into the lab and work on these modifications ASAP.

[Ross_Worobel]

Hi Felix,

I did get a chance to try this out in the lab. After moving the wires around to the correct positions for each motor, increasing the voltage on the power supply, and turning it on, I was able to actually run the single_finger_test.

What I saw was this screen appears for around 0.5 seconds:

Followed by this screen:

Everything looks good – it appears to all be plugged in correctly, I can “watch netstat -i” to see that data is coming in through the CAN ports, the boards all appear to be powered on, etc.

Have you ever seen this issue? Or do you have any other ideas of what could potentially be wrong?

Sorry for the delayed response, and thanks again,
Ross

[felixwidmaier]

Hi Ross,

that you reached the first screen, showing observation, action, etc. means that the software could successfully connect to and initialise the robot, so that’s good news!

The “Action did not end on time” error means that the robot backend detected a violation of the timing constraints. To overcome this, you most likely need to install the PREEMPT_RT kernel.
We are using this script for this: ubuntu_installation_scripts/install_rt_preempt_kernel.sh at master · machines-in-motion/ubuntu_installation_scripts · GitHub
I think we don’t yet have a good documentation on how to use it, though. So some hints:

• The VERSION_ variables at the top refer to the kernel version that will be installed. Usually you can leave the default values.

• DEFAULT_CONFIG needs to be modified to match the kernel version you are currently using.

• In the beginning (after downloading some things) you will be asked to manually adjust some configuration settings. Before entering the menu, the script prints the following instructions:

  Please apply the following configurations in the next step:
  
  General setup
    Local version - append to kernel release: [Enter] Add '-preempt-rt'
  
  Processor type and features ---> [Enter]
    Preemption Model (Voluntary Kernel Preemption (Desktop)) [Enter]
      Fully Preemptible Kernel (RT) [Enter] #Select
  

  However, I observed that depending on the kernel version the “Preemption Model” setting may be found in the “General setup” menu, not in “Processor type and features”.

• Once the kernel is installed, you should configure Grub to load it by default and add your user to the “realtime” group.

• When using the real-time kernel, the following command should be run after each reboot:

  sudo chmod 0666 /dev/cpu_dma_latency
  

  Alternatively you can set up the following udev rule to do this automatically:

  KERNEL=="cpu_dma_latency", NAME="cpu_dma_latency", MODE="0666"

[Ross_Worobel]

Hi Felix,

First of all, sorry for the late reply – things were hectic around the end of the semester here.

I did end up installing the real time Kernel using the script provided, so thank you for that. I have also ran the command you provided here successfully:

sudo chmod 0666 /dev/cpu_dma_latency

I’m able to confirm that the real time kernel is the one which I am using as seen here:

In addition, we can also see that it’s properly being recognized within Singularity as seen here:

Unfortunately, I’m still getting the same errors about timing. I see the error warning that the thread is not real time during both the “demo_fake_finger”:

And the “single_finger_test”:

I was wondering if you had any idea of what else this could potentially be, or any other ideas of how to proceed?

Sorry for the long wait between responses and thanks again for all your help,
Ross

[felixwidmaier]

Hi Ross,

No need to apologise!
You probably need to rebuild the workspace. There are some build flags that are set automatically based on the detected kernel, so to get a real-time capable build, the preempt-rt kernel needs to be running at build time (or alternatively you can set the flag explicitly: colcon build --cmake-args -DOS_VERSION=preempt-rt).

So I expect that when you do a clean rebuild (rm -rf build install; colcon build) while the preempt-rt kernel is active, it should solve the issue.

I’ll update the documentation to add this information.

[felixwidmaier]

I added a section regarding the real-time stuff to the documentation: Real Time Setup — robot_interfaces documentation

[Ross_Worobel]

Hi Felix,

Thanks again for the input here. I did rebuild the project with these flags and it worked as expected. When I run the program now, I do not see the “warning” any more.

I think at this point I’ve done everything possible – with the updated Singularity image, I’ve successfully built the package, am able to successfully run the “demo_fake_finger”, installed the RT_PREEMPT patch to my kernel, and everything else above (i.e. the motors are connected to the correct positions on the board, the power supply is supplying 24V, the CAN ports are initialized, everything is plugged into the correct CAN ports, I ran all of the commands on startup, etc. etc.) but the robot still doesn’t actually work. I also built a second finger using higher resolution 3D printing in the hopes that potentially the printing quality might make it work, but it did not.

The situation is that I begin the “single_finger_test”, the screen outputs “Waiting for board and motors to be ready”. The LEDs on the TI boards begin flashing as seen here:

I’m not sure what this flashing means but it never stops, the status message of “Waiting for board and motors to be ready” never changes, the robot never moves, and that’s basically where I’m stuck at.

I’d love any insight you have as to where to look next! I don’t mind putting in some effort to get this working, I’m just not sure exactly where I should focus that effort…

Thanks again!
Ross