Currently running 3 eVOLVERs, one is dropping fluidic commands periodically. Chemostat commands to turn on pumps and turn off pumps. 5/16 vials have dropped at least 1 of the 24 commands that have been sent so far in this experiment. DPU is recording event properly, and so running restart_chemo by clicking “start/measure now” seems to fix, but I can’t be here for all pump events. Due to the experiment design, it’s receiving 4 fluidic commands at the same time, tried to space out manually to see if that helps.
If this is a bug, should be fixed, maybe bring back the queue? If this is an expected failure mode, then this should be more explicitly mentioned so people can make DPU workarounds. Particularly worried about efflux pump events separated from influx.
The other two haven’t dropped events, but have received way fewer commands so not a fair comparison.
Is there a way to tell which commands have not been picked up other than standing next to the eVOLVER?
I’m continuing my experiment with a quick change that logs the commands that were sent, can use to see if its a DPU error or a server error. If it’s a server error, I don’t know of a way currently other than manually checking.
I’ve been trying some things out with Chris in the lab and we can’t reproduce the issue. My guess is that something is going wrong between the rpi server and the arduinos where a command for whatever reason gets dropped (probably a broadcast request for data in between commands being executed?) Couple things we should consider doing:
-
On the arduino side, make it such that if a duplicate command comes in that is already being run, it essentially ignores the command. This will allow the dpu/server to repeatedly send chemostat commands that won’t interfere if it’s already running.
-
Also on the arduino, let the pumps be queried for their current settings, or respond after being sent a command like the temp/od arduinos do. This would let the server determine if it needs to update the arduinos if the dpu sends a command or if a command failed to go through and should be resent.
-
On the server, put code in place to check if an error occurs during a command and retry the communication with the arduino.
On the Arduino Fluidics Code, try moving the inputString clear command (line 186) only if the address is found. This is especially important with longer commands like the chemostat command because it might clear the string prematurely. This way, the input serial queue is filled up new incoming data and overwritten, instead of prematurely deleted:
See below:
void loop() {
// Read Serial Inputs and checks if it is the right addresses
serialEvent();
if (stringComplete) {
SerialUSB.println(inputString);
in.analyzeAndCheck(inputString);
if (in.addressFound) {
SerialUSB.println("Input Address Found");
fluid_mode = in.input_array[0]; // Obtains Fluid Mode ("p" or "o")
if (fluid_mode == "p") {
//Event for correct address
}
inputString = "";
}
// clear the string
stringComplete = false;
in.addressFound = false;
}
This might help it but not solve the issue completely. Note: I haven’t tested this out yet but adding more checks for the Arduino code, and also current status of the Arduino is also useful.
Still happening, here’s what we know:
-dpu is logging command correctly, and sending to server correctly, problem exists between server and arduino
-may not depend on the number of fluidic commands that get sent (most recent occurrence had only one command being sent)
-adding time.sleep() to dpu (in update_chemo) or to server code (check with Zack where) does not fix
-may be device specific (can’t reproduce on second eVOLVER)
Seems like there’s a port that doesn’t open correctly. This is a log of one of the failed commands:
This was supposed to shut off chemostat for vial 5, but vial 5 continued to pump. The subsequent commands executed correctly.
Since this doesn’t seem to depend on the type of fluidic command, I bet we could find other instances of this in turbidostat experiments where many fluidic events are requested. Ought to also try to reproduce on another device by manually sending a bunch of commands to see if its a hardware fluke.
@heinsz do you know where the NoneType error is originating from? Is it from the broadcast?
The none type comes from the previous broadcast od/temp data serial comms getting interrupted. Per our conversation, getting rid of the whole closing serial comms is the way to go since the new OD Arduino code is much faster now! I’ll update the server tomorrow and our Arduinos with the new OD code you wrote. @cmancuso we can do some testing on one of the unused evolvers tomorrow - if it looks good I suggest we update your currently running experiments. I don’t usually like updating things live but if it’s potentially going to make your data unpublishable we should just try it.
No rush, not going to change code now that I’m 80% of the way through the experiment. Will just be down a few vials. This is why replicates are important 
Not eVOLVER specific, since it happened on a second eVOLVER now. It seems like the first in group of commands is the most likely to get skipped. Just FYI
1 Like
