Dpu syntax to send od_90 / od_135 commands

Happy New Year everyone!

We’re testing the communication between the dpu, Raspberry Pi and microcontrollers and have been stumped by what seems to be a syntax problem when we try to send commands from the dpu for od_90 and od_135 commands using server_test.py. Hopefully this is a straightforward question: what is the correct dpu syntax to send commands from the dpu for od_90 and od_135? For example, this is the syntax for the od_90 command we are trying without luck:

data = {‘param’: ‘od_90’, ‘value’: 1000, recurring: False, ‘immediate’: True}
print(data)
evolver_ns.emit(‘command’, data, namespace = ‘/dpu-evolver’)

Is there a field needed to specify which vials to take the photodiode measurements from?

For context, we have been able to load the RS485STIR, RS485TEMP, RS485_OD and RS485_PD code onto an arduino and communicate with it from the dpu to send commands for stir, temp and od_led so far by modifying the server_test.py code. We added an od_led command with this syntax:

data = {‘param’: ‘od_led’, ‘value’: [4095] * 16, ‘immediate’: True}
print(data)
evolver_ns.emit(‘command’, data, namespace = ‘/dpu-evolver’)

We are able to see the stir, temp and od_led commands show up on the arduino Serial monitor and be executed.

However, this syntax for the od_90 command doesn’t seem to work:

data = {‘param’: ‘od_90’, ‘value’: 1000, recurring: False, ‘immediate’: True}
print(data)
evolver_ns.emit(‘command’, data, namespace = ‘/dpu-evolver’)

Specifically, we don’t see a corresponding string like od_90i,1000,! or od_135i,1000,! show up on the arduino Serial monitor. And on the Raspberry Pi Electron GUI the output we see for these commands is:

Response: Nothing Changed, no vials selected

whereas for the other commands we see what looks like an appropriate response.

We can see however that the field values (e.g. value = 1000, recurring = False, etc) are updated when we change them using this syntax (e.g. value = 500, recurring = True, etc). Also, viewing the log file in real-time while server_test.py is running using:

sudo supervisorctl

tail -f evolver

we can see that, for example, the od_90 data is received by the Raspberry Pi via debug print statements inside of evolver_server.py. So for example, we can see that data like this is received:
data = {‘param’: ‘od_90’, ‘value’: 1000, recurring: False, ‘immediate’: True}

However, no corresponding string command like
od_90i,1000,_!

ever seem to appear in the log file even though we can see string commands for stir, temp and od_led like
od_ledi,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,_!

We would be happy to try any debug tests that anyone might suggest!

Thanks for reading and hope it is a simple syntax fix or something like that we’re missing!

Sincerely,
James

Hi James,

For the first question about syntax of od_90, what are you trying to do? That value is for how many data points to average on the SAMD21. The way you have it formatted looks correct. When you execute that code in your python script, do you see the Received COMMAND log pop up in the server log?

The raspberry pi will take data measurements from all parameters that are connected and configured in the conf.yml file. So if you have 2 photodiodes (od_90 and od_135), as long as they are both in conf.yml file and you have the two SAMD21 boards properly connected you will get data back from both every time the rpi cycles through everything.

How are you sending these commands to do the rpi? If you are using the electron GUI, by default the GUI is not configured to be able to send commands to od_90 or od_135 unless Brandon helped set you up for it or you modified it yourself. It looks like you’re running things in a python script, but you mentioned the GUI so I’m just making sure I understand how you are interacting with the rpi.

Zack

Hi Zak,

Thanks for your explanation, I really appreciate it and it’s helpful to understand that the syntax we have looks correct. That is also useful to know that the GUI is not able to send od_90 or od_135 commands and that measurements are taken for a given parameter as long as it appears in the conf.yml file.

In answer to your first question, we are just trying to debug the photodiode portion of the RS485_OD and RS485_PD code on the SAM21 controllers with the server_test.py code because we never see od_90 or od_135 commands like

od_90i,1000, !

or

od_135i,1000, !

appear in the Serial monitor inputString so they don’t get executed by the photodiode logic. For example, for the RS485_OD code, we are printing out the inputString to the serial monitor and can see the other commands for stir, temp and od_led but for some reason we never see a corresponding command for either od_90 or od_135 when we use the syntax that I had described in the previous post:

data = {‘param’: ‘od_90’, ‘value’: 1000, recurring: False, ‘immediate’: True}
print(data)
evolver_ns.emit(‘command’, data, namespace = ‘/dpu-evolver’)

Using this syntax, the RS485_OD code never seems to find the od_90 address when line 85 is called (see below)

line 85 in.analyzeAndCheck(inputString);

and so line 92 (see below) always evaluates to False and the photodiode logic and measurement never gets executed:
line 91 // Photodiode logic
line 92 if (in.addressFound) {

For reference, this is the RS485_OD code we are flashing:

When we include an immediate od_led command with an immediate od_90 command, we are able to see the od_led command show up in the inputString in the serial monitor output as well as verify that the od_led address is found by the analyzeAndCheck call in line 115:

line 115 if (led.addressFound) {

The attached screenshots of the serial monitor show that the od_led command gets executed once inside the if (led.addressFound) loop.

We do see the Received COMMAND message show up in the server log (please see the attached screenshots) and also can see that the conf.yml file is updated with new values that we set via the commands we program in server_test.py. For example, if we program the command below

data = {‘param’: ‘od_90’, ‘value’: 500, recurring: False, ‘immediate’: True}
print(data)

evolver_ns.emit(‘command’, data, namespace = ‘/dpu-evolver’)

we can see the ‘value’ field for the ‘od_90’ param changed to 500. Interestingly, we never see the ‘immediate’ field added to the conf.yml file but assume this isn’t necessary as that information can be encoded in the ‘recurring’ field.

On the SAM21 controlller side though we are still not ever seeing a command like od_90i,1000, ! or even the ‘od_90’ param itself appear as part of any of the inputStrings that are being read in despite the fact that we see the Received COMMAND message on the Raspberry Pi server side. This is certainly the confusing part at the moment.

In case it is helpful, we have previously been able to see the od_90 and od_135 commands in the inputString and get the arduino to recognize the od_90 and od_135 address without sending explicit od_90 or od_135 commands. This at least makes sense based on what you described about the photodiode measurements being taken as long as the od_90 or od_135 parameters appear in the conf.yml file. For example, when we were first testing we flashed the RS485_PD code and ran the default server_test.py and were able to see that the photodiode logic was executed on the serial monitor. Given this, we were wondering if there is a way to reset the system to this state for testing as we haven’t been able to revert it so far?

Since I am not able to attach text files here, I have uploaded screenshots of the server log, SAM21 serial output and server_test.py output for the two cases we have just tried today: sending a recurring od_90 command alone and sending an immediate od_90 command and od_led command. I have also uploaded a screenshot of the conf.yml file showing the updates it is receiving. It has just let me know that I can only upload one image at a time so I will reply to this post with separate screenshots.

In case it is easier to read them by email I can also send the log files and debug output to you that way. And we’re always happy to try out anything that you think would be useful for debugging.

Thanks very much again for the help and hope we’re slowly making progress!

Sincerely,
James

Hi Zak,

Here are the rest of the screenshots, one at a time unfortunately given the forum restrictions.

Thanks again!

Sincerely,
James

Couple quick suggestions. First, in conf.yml change the serial_timeout field to something like 1 or 1.5. Also change in evolver_server.py the two lines that have time.sleep(.05) (line 279 and line 309) to be 0.2. After you make these changes, either restart the rpi by unplugging it and plugging it back in or typing sudo reboot now in the terminal.

I also see in your server log some weird flush = False param = od_90 lines - I have never seen that before. Did you modify the server code at all? If so, can you share what you modified?

Do you see the rpi cycle through all the params every 20s if you just watch the logs without sending commands or anything? If it’s stalled, you can try just restarting it. I have a way to automatically detect this if it’s causing problems for you.

Lastly, can you share your script you are using to send commands to the rpi?

Also, what frequency are you sending commands in the python script? Are you waiting at all or just sending them one right after the other? It shouldn’t really matter, but it looks like multiple commands are coming in with nothing happening in the server logs.

Hi Zack,

Thanks for the suggestions and follow up questions, much appreciated!

I’ll give the suggestions a try later today and report back as well as send detailed answers to the questions you raised.

Thanks again!
James

Hi Zack,

We ended up first trying to get the system back into its previous state where recurring od_90 and od_135 commands show up on the SAMD21 controllers by flashing an image on the Raspberry Pi as you describe in this thread:

Just by flashing the image in the above thread, that seems to have fixed at least the issue of not seeing od_90 and od_135 commands on the SAMD21 controllers! Now, for example, when we use the default dpu server_test.py script with only the time_to_wait value changed so that stir and temp commands are sent together every 10 seconds, we can see the recurring od_135 command show up in the serial monitor for a SAMD21 controller that has been flashed with RS485_PD. Additionally, the evolver_si instance seems to process the recurring od_135 command correctly and initiate the photodiode logic based on recognizing the ‘od_135’ address.

There seems to now be a different issue however that appears related to the handshake between the Raspberry Pi and SAMD21 controller. Specifically, the logic in the RS485_PD code hangs at the line within the photodiode logic that waits for the OK from the Raspberry Pi to execute.In the serial monitor, we see the message

Waiting for OK to execute …

but not the message

PD Command Executed!

If we manually change the value of in.input_array[0] to the “a” character we then are able to see the rest of the photodiode logic executed and the “PD Command Executed!” message appear in the serial monitor. We then checked this behavior for the rest of the params (e.g. stir, temp, od_led and od_90) and they also have the same behavior of hanging on the Waiting for OK to execute … line in their respective SAMD21 scripts.

So it seems like the error is somewhere in the SAM21 controller message being sent back to the Raspberry Pi either not being received or not being correct?

In case it is helpful, here a representative snippet of the log file output for evolver_server.py when we run tail -f evolver showing the data that is being broadcast and a series of error output messages of having to do with EvolverSerialError:

Broadcasting data{‘data’: {}, ‘config’: {‘lxml’: {‘fields_expected_incoming’: 17, ‘fields_expected_outgoing’: 17, ‘recurring’: False, ‘value’: [‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’]}, ‘od_135’: {‘fields_expected_incoming’: 17, ‘fields_expected_outgoing’: 2, ‘recurring’: True, ‘value’: ‘1000’}, ‘od_90’: {‘fields_expected_incoming’: 17, ‘fields_expected_outgoing’: 2, ‘recurring’: True, ‘value’: ‘1000’}, ‘od_led’: {‘fields_expected_incoming’: 17, ‘fields_expected_outgoing’: 17, ‘recurring’: True, ‘value’: [‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’, ‘4095’]}, ‘pump’: {‘fields_expected_incoming’: 49, ‘fields_expected_outgoing’: 49, ‘recurring’: False, ‘value’: [‘2’, ‘2’, ‘2’, ‘2’, ‘2’, ‘2’, ‘2’, ‘2’, ‘2’, ‘2’, ‘2’, ‘2’, ‘2’, ‘2’, ‘2’, ‘2’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’, ‘–’]}, ‘stir’: {‘fields_expected_incoming’: 17, ‘fields_expected_outgoing’: 17, ‘recurring’: True, ‘value’: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}, ‘temp’: {‘fields_expected_incoming’: 17, ‘fields_expected_outgoing’: 17, ‘recurring’: True, ‘value’: [2047, 2047, 2047, 2047, 2047, 2047, 2047, 2047, 2047, 2047, 2047, 2047, 2047, 2047, 2047, 2047]}}, ‘ip’: ‘192.168.1.90’, ‘timestamp’: 1641425713.1169145}
Received COMMAND
Received COMMAND
tempi,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,!
Traceback (most recent call last): File “/home/pi/evolver/evolver/evolver_server.py”, line 242, in run_commands
returned_data = serial_communication(command[‘param’], command[‘value’], command[‘type’])
File “/home/pi/evolver/evolver/evolver_server.py”, line 285, in serial_communication raise EvolverSerialError('Error: Response has incorrect address.\n\tExpected: ’ + param + ‘\n\tFound:’ + address)evolver_server.EvolverSerialError: Error: Response has incorrect address.
Expected: temp
Found:
stiri,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,
!
Traceback (most recent call last): File “/home/pi/evolver/evolver/evolver_server.py”, line 242, in run_commands
returned_data = serial_communication(command[‘param’], command[‘value’], command[‘type’]) File “/home/pi/evolver/evolver/evolver_server.py”, line 285, in serial_communication raise EvolverSerialError('Error: Response has incorrect address.\n\tExpected: ’ + param + ‘\n\tFound:’ + address)evolver_server.EvolverSerialError: Error: Response has incorrect address.
Expected: stir
Found:
Received COMMAND
Received COMMAND
tempi,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,!
Traceback (most recent call last): File “/home/pi/evolver/evolver/evolver_server.py”, line 242, in run_commands returned_data = serial_communication(command[‘param’], command[‘value’], command[‘type’]) File “/home/pi/evolver/evolver/evolver_server.py”, line 285, in serial_communication raise EvolverSerialError('Error: Response has incorrect address.\n\tExpected: ’ + param + ‘\n\tFound:’ + address)evolver_server.EvolverSerialError: Error: Response has incorrect address.
Expected: temp
Found:
stiri,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
!

In particular, it looks like the culprit has to do with line 242 and line 285 of evolver_server.py as it is telling us that there is an EvolverSerialError and that the Response has an incorrect address.

Is there a way that you might suggest to debug the Response to check the address that is being sent back from the SAMD21 controllers to the Raspberry Pi?

Also in case it is helpful, below is also the server_test.py code we are using:

import socket
from socketIO_client import SocketIO, BaseNamespace
import asyncio
from threading import Thread
import random
import time
STIR_VALUES = [[0] * 16, [8] * 16] # flip between on and off
TEMP_VALUES = [4095] * 16 # always off
EVOLVER_IP = ‘169.254.73.121’
EVOLVER_PORT = 8081
evolver_ns = NonesocketIO = None
class EvolverNamespace(BaseNamespace):
def on_connect(self, *args):
print(“Connected to eVOLVER as client”)
def on_disconnect(self, *args):
print(“Discconected from eVOLVER as client”)
def on_reconnect(self, *args):
print(“Reconnected to eVOLVER as client”)
def on_broadcast(self, data):
print(data)
def run_test(time_to_wait, selection):
print('time_to_wait = ', time_to_wait);
print('selection = ', selection);
time.sleep(time_to_wait)
print(‘Sending data…’)
# Send temp
data = {‘param’: ‘temp’, ‘value’: [2047] * 16, ‘immediate’: True}
print(data)
evolver_ns.emit(‘command’, data, namespace = ‘/dpu-evolver’)
# Send stir
data = {‘param’: ‘stir’, ‘value’: STIR_VALUES[selection], ‘immediate’: True}
print(data)
evolver_ns.emit(‘command’, data, namespace = ‘/dpu-evolver’)
# Set things for the next one
selection = 1 - selection
time_to_wait = 10
# time_to_wait = random.randint(1, 31)
print('Seconds to wait: ’ + str(time_to_wait))
run_test(time_to_wait, selection)
def start_background_loop(loop):
asyncio.set_event_loop(loop)
loop.run_forever()
def run_client():
global evolver_ns, socketIO
socketIO = SocketIO(EVOLVER_IP, EVOLVER_PORT)
evolver_ns = socketIO.define(EvolverNamespace, ‘/dpu-evolver’)
socketIO.wait()
if name == ‘main’:
try:
new_loop = asyncio.new_event_loop()
t = Thread(target = start_background_loop, args = (new_loop,))
t.daemon = True
t.start()
new_loop.call_soon_threadsafe(run_client)
time.sleep(5)
run_test(0, 0)
except KeyboardInterrupt:
socketIO.disconnect()

The only relevant change we have made to the code is altering time_to_wait to 10 seconds. Please just let me know if it would be helpful to include any other information.

Thanks again for your help and eager to test out any debugging suggestions that come to mind!

Sincerely,
James

Hi James,

Thanks for the information! Going forward, please put code within a preformatted text block when posting - it’s the little button that looks like </> in the editor window. Python is especially difficult to read without whitespace formatting :slight_smile:

Did you modify the timing aspects that I mentioned in my last post? If you did then and it’s still not working, I have another theory that I need to test out on my end. In the meantime, can you try putting a delay of a second or two between the two commands in the server_test.py script?

Hi Zack,

Thanks for your continued help, it is much appreciated! And sorry for the difficult to read code pasting!! I will definitely put code in the preformatted blocks from now on for sanity :slight_smile:

Here’s hopefully a sufficiently detailed summary in answer to your suggestions and questions:

We made the change to conf.yml so that the serial_timeout field has a value of 1.5 and the changes to evolver_server.py so that the time.sleep value is 0.2 for both occurrences (at line 279 and line 309). There are no other debugging changes we made (e.g. no more debug print statements that we have added) to evolver_server.py.

Also as you suggested we changed the dpu server_test.py script so that now it is only sending a single stir command once every 10 seconds (i.e. we command out the temp command block).

In answer to your question, if don’t run server_test.py and just watch the server log using tail -f evolver, we do see the Raspberry PI cycling every 20 seconds through all the params. Specfically, we see the Broadcasting data message every 20 seconds and for each param that is cycled through what look like the previous EvolverSerialError messages thrown at line 242 and line 285 that we were seeing. For example, when it cycles to the Stir param we see:

EvolverSerialError : Error: Response has incorrect address

Expected Stir

Found: tempr, 3030, 3030, 3030, … ,_!

Would this mean that it should be seeing a string like?:

stirr, 8, 8, 8, … , _!

instead of the

tempr, 3030, 3030, 3030, … ,_!

It looks like the params being sent back are somehow out of phase so that they don’t match up as expected. We get the EvolverSerialError message for each param (stir, temp, od_led, od_90 and od_135) that is cycled through.

On the SAMD21 controller side, we can see on the serial monitor that it is able to receive the broadcast data and parse it as before but still hangs at the line Waiting for OK to execute.

When we do run server_test.py, we see the RECEIVED COMMAND message in the server log every ten seconds as expected. For example,

RECEIVED COMMAND

stiri, 8, 8, 8, …, _!

but we also still see the same EvolverSerialError messages thrown at line 242 and line 285 that we see when not running server_test.py and just letting the data broadcast cycle every 20 second in the background.

We also tried increasing the time.sleep value to 0.3 without a change in behavior in either the tailed server log or the SAMD21 serial monitor ; the incoming message is still parsed successfully but then hangs at Waiting for OK to execute. And if we change the in.input_array[0] to “a” we can then get the Command Executed message as we did previously. Does it make sense continuing to increase the serial.timeout and or time.sleep value even higher?

In summary, it looks like we get cycling commands from the Raspberry Pi to the SAMD21 controller that can be parsed successfully but are either not sending the acknowledge command back successfully from the SAMD21 or having it be received successfully by the Raspberry Pi.

Given this behavior, is there a barebones script that we can write that would let us just test sending the acknowledgement command from the SAMD21 controller to the Raspberry Pi?

One practical question also came up: is there a way to send the tail -f evolver output to a file? e.g. the equivalent of the pipe command for supervisor like:

tail -f evolver > output_file

We’re more than happy to try out any other tests that you think would be helpful and would definitely be curious about any other theories!

Thanks again for the continuing help!

Sincerely,
James

I would try adding more debug print statements in the serial comms functions on the RPi - you confirmed that the message is being sent from the samd21, try checking if you see that come in on the rpi now.

At this point it might be helpful if we schedule a zoom call so I can see in real-time whats going on and try poking around on the system. Let me know if you’d be open to that.

Zack

Hi Zack,

Thanks for your suggestions! We will definitely give that a try now.

Just to make sure that we are adding debug print statements in the correct place, would you mind pointing us to which serial comms functions on the RPI you are referring to?

Also just another point of clarification: do you mean that we know we have confirmed that the message is being sent from the SAMD21 because we see

RECEIVED COMMAND
stiri 8, 8, 8, …, _!

? We’re a bit confused about what the format of the message is that we are checking for to see if it comes in on the RPI. Can you give an example of what the correct format of the message should be? For example, should it be

stiri 8, 8, 8, …, _!

or some other format? Maybe a better way to ask is, when we see the EvolverSerialError thrown, would the correct behavior be something like below for example?:

Expected: stir
Found: stiri 8, 8, 8, …, _!

And that sounds great about scheduling a zoom call, I really appreciate it. Would it be best to schedule over email?

Thanks very much again for all the help!

Sincerely,
James

Hi All,

Just wanted to write up a summary after getting incredible help from Zack. Thanks Zack!!

We initially were experiencing some odd behavior with no signals on the Rx line of communication when cycling commands were being sent from the RPI despite being able to observe Tx signals just fine. After a suggestion from Zack we tried removing the default resistor soldered on our RS485 hat. That resolved our Rx line problem!

Thanks again for all the help Zack!

Sincerely,
James

3 Likes