SER module and flow control on UL865

11 thoughts on “SER module and flow control on UL865

  1. UL865-EUR and firmware 12.00.606 

    Using Python  I’m implementing a kind of data bridge between the USIF0 serial port and an UDP socket.
    The bridge is not intended to be fully transparent, i.e. it’s NOT an endless byte data stream. Instead I need to send and receive predefined and variable length data packages via UDP, which I receive or send to the serial port on the other side. Packages must not be split or merged, because our custom protocol (and a server application) relies on these well defined UDP packages. Package sizes my vary between a few bytes and 530 bytes, but there is only 10 to 20 unique package types.

    I decided to use Python (2.7) on the UL865 with
    a) the _socket module (_socket.socket(_socket.AF_INET, _socket.SOCK_DGRAM)) to do the UDP traffic and
    b) the thread module to listen to incoming UDP data (sock.recv)  and
    c) the SER module to handle the serial communication.
     

    This works quite well in my first tests where I avoid to go to the limits, i.e. a buffer overflow cannot occur. But since for the real application the data volume transferred via this bridge is rather high, I definitely need a solid flow control. I don’t want to use hardware handshake, but implement my own software handshake.

    I have read that the SER module does not provide an automatic hardware flow control anyway, I would have to call SER.setCTS and SER.getRTS to set/check the physical lines on my own. But since my serial devices currently does not use these lines, I want to implement a software flow control. (kind of XON, XOFF, but more than a single character, and must operate on package level, not character level).
    Also SER.getRTS always returns ‘1’ in all my tests, regardless of what level is applied to the RTS pin. The pin itself is connected correctly, e.g. a level change can wake up the module from SYSHALT).  Other commands like SER.getDTR or SER.setCTS work as expected, but as long as I can avoid having to use it, this does not matter.

    And that’s where I’m stuck now. For sending data to the serial interface with SER.send I can use SER.sendavail to check for an approaching buffer overflow (limit 32kb). But for reading I did not find any construct to check the available/remaining size of the receive buffer.  I want to signal my serial device (via a SER.send control package) to stop transmission, if the receive buffer e.g. has less than xyz bytes free. This will especially be the case if the UDP upload is stuck (to slow, or connection lost) and in turn I do not consume and read from the SER.

    Is there any chance to obtain information about the USIF0 input buffer status from within Python ?

    The only workaround which comes to my mind is to always read ALL bytes from the input buffer with high priority (in an additional thread), and thus maintain a secondary input buffer on my own. However I do not know how much memory this would cost/require, and I doubt that the priority of my Python script / thread is high enough to avoid a buffer overflow of the “real” input buffer.

    1. Hardware flow control in SER module is supported since 12.00xx4 and 13.00.xx5 firmware versions, and I cannot see any way to find the fill status of the input buffer unfortunately. I would go with your solution with application own buffer of 4K, if this tends to fill for sure the SER buffer would be in the same state, and proceed with your software flow control.

      1. I will try to do test whether my workaround is feasible.
        Concerning the the hardware flow control, which I still might have to use: I also have read the Easy script in Python 2.7 r5 documenation, which says

        If flow control is not enabled (default) the SER built-in module will also allow to control physical lines used as GPIO. If flow control is enabled, only available from version 12.00.xx4, version 13.00.xx5 and version 18.11.004, it will not be possible to control physical lines used as GPIO.

        (At least on the UL865 the CTS and RTS are not multiplexed with the GPIO pins, so I do not understand how to interpret the 2nd sentence)

        So while my UL865 should support it, I completely fail to successfully use any SER hardware flow control.
        Either I do not understand the concept, or the hardware flow control does not work as documented. There are multiple problems.

        Since not otherwise documented I assume that enabling or disabling hardware flow control (also for SER) ist done with AT&K3 and AT&K0. I would expect “automatic” bidirectional hardware flow control, if &K3 is set, i.e. the UL865 interprets the RTS and sets the DTS lines as required.  I always verify with AT&V what the output tells about the current status.  I also tested to call it either from my Python script, or to set it via USIF0 before the script is started, which makes no difference.

        FLOW CONTROL OPTIONS           : &K3=HW bidirect.

        First, by default the hardware flow control (K3 setting) is always “HW bidirect”.  So either the documentation is wrong, or this is not the right command to enable or disable the hardware flow control for SER. Or I have to issue additional commands ? I know the AT documentation also says “hardware flow control is not active in command mode”, but since I decided to use the socket module for my UDP communication, I assume and hope this restriction does not apply here.

        Next, I never succeed to get something meaningful with the  SER.getRTS()  method. It will always return ‘1’, regardless of which logic level is applied to the RTS (pin 5 of the UL865). I measure the level with a multimeter, so its really connected, and as I said before, the UL865 also detects level changes on this pin for other functions like “wake up” from power save modes.  And to be sure I did each of my tests with and without the AT&K hardware flow control setting. 
        Note that SER.getDTR never has such problems,- it returns a change on the input pin correctly, and SER.setDTS also sets the output pin correctly.

        Further, trying to let the module handle the RTS and DTS lines automatically (I assume AT&K3) will also not behave as I expected.
        I have a small test loop, which sends a 512 byte testdata string.

        for i in range(0,100):
           av = SER.sendavail()
           rts = SER.getRTS()
           SER.send(“rn” + str(i) + “, RTS=” + str(rts)+”, SendAvail:”+str(av)+’rn’)
           SER.send(str(testdata)), 200)
           time.sleep(0.01)

        Regardless of what I apply to the RTS pin, the complete > 52 kb of data will be shifted out on the serial interface, i.e. any change on RTS is ignored. And also the SER.getRTS always returns ‘1’, which implicitly may indicate the cause for the problem. 

        If I do not call time.sleep in the loop, I can see ‘sendavail’ go down to zero and then transmission will stop after exactly 32 kb of data (the buffer size), although all of the data has already been transmitted. I.e. the transmit buffer has been transmitted, but according to SER.sendavail its full. I guess this is a different  problem of the internal handling and priority of the Python script.

        Finally, the other direction: when I send lots of data from my external serial device to the UL865 USIF0, running a script, but (intentionally) NOT reading the data from SER, then I cannot see DTS to change its level when the 4k buffer is full. If I do not use SER.setCTS manually, then CTS is always ‘1’. As a result the 4k buffer likely overflows. If I set the CTS manually, then my serial device will properly stop sending, when CTS is off.

        So nothing behaves as expected, and it seems as if hardware flow control is not working or not really enabled.
        How to get this hardware control working ? 

  2. My fault, I did not pay enough attention to the documentation of SER.set_speed. Using “hw” for hardware flow control results in SER.send to evaluate the signal at the RTS pin and stop sending data depending on the status. Thanks for the hint.

    The good thing is that this proves that the RTS pin is correctly connected, unfortunately SER.getRTS will still always return ‘1’as mentioned before.

      rts = SER.getRTS()
      USB0.send(“RTS:”+str(rts)+”rn”)

    regardless of the voltage level on the input. I tested for both  “none” and “hw”

       r = SER.set_speed(‘115200′,’8N1’, “none”)

    From the documentation of SER (Easy Script in Python 2.7 r5) :

    If flow control is not enabled (default) the SER built-in module will also
    allow to control physical lines used as GPIO. If flow control is enabled, only available from
    version 12.00.xx4, version 13.00.xx5 and version 18.11.004, it will not be possible to control
    physical lines used as GPIO.

    I would expect  SER.getRTS() to reflect the status of the applied RTS signal

    Critical baud rate problem with SER_set_speed

    Having read the the documentation for SER.set_speed more carefully, I now see just an even more serious problem for my ‘data bridge’ approach.  The documentation mentions that baud rates 300 … 115.200 baud are accepted.  Up to now I did not pay enough attention to this statement, because I assumed that the documentation simply cannot reflect all the different Telit modules. I thought that the SER implementation of our UL865 would be supported with higher dates, like they are available in AT mode / online mode

    at+ipr=?

    +IPR: (300,1200,2400,4800,9600,19200,38400,57600,115200,230400,460800,921600)

    But a quick test in Python with 230.400 baud

    SER.set_speed(‘230400′,’8N1’, “none”)

    returns -1 and it will not accept this baud rate.  In the early evaluation for our project I verified that the UL865 supports baud rates of at least 230.400 and above, but I tested for AT mode only. 

    This unexpected difference in the specification depending on whether I use AT mode or Python is critical for our project. With my current knowledge it would mean that we cannot use Python at all, which however was one of the reasons to choose the 3G module UL865

    Since the hardware has no problem with the higher baud-rates, the question is: 

    Is this a real performance limitation or is there any chance that only the “software interface” for SER is too general and restrictive here ? I.e. that it might be changed in an upcoming firmware update ?

    1. Hardware flow control for serial port ( ASC0 ) not working ?

      Since I cannot continue with my original Python approach, I decided to use the time waiting for an answer from Telit, to check an alternate solution without Python. I reverted from Python to the data / transparent mode. I use an UDP connection like this
      AT#SKIPESC
      AT#SD=1,1,5018,”74.125.136.99″, 0, 9050, 0
      and the transfer works fine in both directions. However while this allows me to use baudrates > 115200 baud, in this case I also need to rely on the hardware flow control (especially CTS is even more important than RTS here for our application).
      Unfortunately the hardware seems not to work;
      As e.g. described in the Telit Moduls Software User Guide
      “The CTS (output) of the DCE is not asserted when the data in its receiver buffer is greater than 75% of its capacity, the DTE transmission is stopped. The CTS is asserted when data in the receiver buffer of the module is lower than 25% of its capacity”

      However in my tests with both my UL865 custom PCB and a HE910 EVK board (also 12.00.006) the CTS is not asserted. The result is as follows (same for both modules)

      a) with a short Pyhton script and SER.setCTS, I first checked that the CTS signal is wired correctly between DCE and DTE and also that my software sending (TX) data correctly reflects the current CTS line state.

      b) I verify that the hardware handshake is active (though its active by default)
      DTE SPEED : 115200
      DTE FORMAT : 8N1
      GSM DATA MODE : Not Transparent
      AUTOBAUD : +IPRxxx00=NO
      COMMAND ECHO : E1=YES
      RESULT MESSAGES : Q0=YES
      VERBOSE MESSAGES : V1=YES
      EXTENDED MESSAGES : X1=YES
      LINE SPEED : F0=autodetect
      CONSTANT DTE SPEED : YES
      FLOW CONTROL OPTIONS : &K3=HW bidirect.
      ERROR CORRECTION MODE : RLP
      CTS (C106) OPTIONS : &B2=OFF while disc.
      DSR (C107) OPTIONS : &S3=PHONE ready->ON
      DTR (C108) OPTIONS : &D0=ignored
      DCD (C109) OPTIONS : &C1=follows carrier
      RI (C125) OPTIONS : R1=OFF dur. off-hk
      C108/1 OPERATION : &D0=NO
      POWER SAVING ON DTR : +CFUN:1=NO
      DEFAULT PROFILE : &Y0=user profile 1

      c)
      I used different software to send data in the online mode via the UDP connection. At the other end, the receiving software will show whether all data is received or if data was dropped. One test to send data is a simple C# application loop (sending 100 x 516 byte)
      for (i = 0; i < 100; ++i)
      {
      Console.WriteLine("CTSHolding {0}, {1}", i, serialPort.CtsHolding);
      tdata[3] = (byte) i; // byte 0..2 contain some header information
      serialPort.Write(tdata, 0, 512+4);
      System.Threading.Thread.Sleep(10); // wait 10 ms
      }

      d)
      For my testing I want to stress the Telit module and urge it to change the CTS signal. I use (slower) GSM (EDGE) connection only and I send lots of data through the serial port. Each package has 516 byte, with a small delay of 10 ms, so that the Telit module will send the data as UDP package due to the timeout (256 = 10 ms). Total amount is 51,6 KB in approx. 6 seconds
      at#scfg=1,1,650,60,600,256
      In all my tests the result is the same, (unless I use some rather low effective transfer rate, far away from the physical serial port baud rate)

      1) I regularily loose data during the transfer
      2) the CTS line is never raised (level never changes).

      To be sure, I did not only check the CTS status in the software, I also check the CTS line with a logic analyzer. I see that all the data is sent to the Telit module, but the CTS voltage level never changes (always ‘low’ (0)). On the receiving side I can see that and which data was dropped, because I put a sequence number in each package. I know that UDP packages may be get lost on their way, but this does really does not look like an UDP package loss, it looks more like a buffer overflow on the Telit module.
      What I’m doing wrong ? Which other setting might be required for CTS (hardware flow control) to work as documented ?

      1. Well I think this is because indeed the UDP protocol does not have any reliability built in, once connection is open the data is simply thrown out at maximum speed. To really check the flow control you must put in place a TCP connection & server, pause it and observe at the client side.

        1. Well I would never say never, but as I mentioned I know that UDP packages may get lost, and I already excluded this from the list of possibilites. It’s not a problem for our application, if single UDP packes get lost, this is and must be handled in any case. Also the starting and number of lost packages is comparibly reproducible, which is not typical for UDP loosing packages. 

          My observation is that the Telit module apparently has internal problems (not sending all my data packages out as UDP packages) due to a buffer overflow, which I cannot avoid without a working hardware flow control.  So I need to have a reliable CTS signal, and the question is why it does not become active in my setup when it should.  I send data faster via UART than the UL865 can theoretically send it out via GSM/EDGE (236.8 kbits max)

          I could imagine that
          a) there might be a problem with the Telit module firmware when used as in my scenario
          b) I simply make a mistake and need to make some additional setting for hardware flow control (especially CTS) to work.
              RTS is low (voltage level = 0) in all tests.

          I mentioned  in my last post that in my test I check which packages arrive at the server, which ones were dropped.
          To give you more significant numbers I repeated the test today with 921.600 baud, increased the timeout in my 100 x 516 byte loop to 15 ms.

          Then I made a test with the following setting, using a timeout of 10 ms

             at#scfg=1,1,650,60,600,256        i.e. data sending timeout is 10 ms

          and another test with

             at#scfg=1,1,516,60,600,260        i.e. data sending timeout is 50 ms, but due to packet size 516, all my packets are sent directly

          The second one results in an even higher effective data rate, since in my sending application I also change the delay of 15 ms to a marginal value of 1 ms.

          In both cases the problem starts after a data volume near below 32 kb.
          In the first case, after 62 packages ( 62 x 516 byte = 31.992 bytes) the problem starts.  Packages 62 … 65 are lost.  Then packages 66..95 are received again. 96..100 are lost.

          In the second test with faster effective transfer rate, I receive only packages 1 .. 60, i.e. 30.960 bytes,- all the rest (61..100) are lost. I would expect such effects if buffer overflows occur.

          The question is what can I do to get a valid CTS signal here ?
          Do you need additional information about any module settings ?

          1. Here is some additional information for R&D.

            In the last hours I did two additional tests, always using the same simple C# client attached to USIF0:

            1) I tested with a GE864-QUAD mounted in EVK2 (old GE864-QUAD, 115.200 baud only, Firmware 07.02.008)
            same UDP example -> this is will correctly set the CTS signal on and off during the (slow) transfer

            2) I tested with UL865 again in a special setup using TCP,  230.400 baud,  sending 200 x 516 byte, scfg block size set to 516, small delay of 1 ms between 516 byte packages. Network AcT (at+creg?) is GSM in all tests,  +CREG: 2,1,”02AA”,”B75E”,0)

              ->  this also will correctly set the CTS signal 

            3) I repeated the test with UL865 for UDP, with otherwise same settings as for TCP, testing with 230.400 baud and also with 921.600 baud. And I also increased the loop to send 1000 x 516 bytes
              This final stress test sends  516.000 bytes  in 7 seconds via the serial line, i.e. 73.714 bytes per second.  This should not be possible to send with 236.8 kbit (GSM/EDGE)

               -> in no case using UDP the CTS signal is raised.  The actually received data differs from test to test, but there is always a huge data loss.

          2. The following may be helpful for reproducing the problem using the UDP socket.
            To enforce an overrun the USIF0 input buffer, I already did all the tests with AT+WS46=12   ( select GSM digital cellular, avoid 3G)

            Today I have additionaly set AT#MSCLASS=1     (multislot class restrict to 1 up and 1 down channel)

            Default previously was 33.  (up to 236.8 kbit)
            This restricts the transfer rate to 59.2 kbit and even with a serial baud rate of 115.200 I then can easily reproduce the problem.
            Having already finished transferring all 200 packages of 516 byte over the serial line to the UL865, the module will not have sent more than 10 or 20 packages at this point over the air.  More buffered packages are transferred with a rate of only 1 – 2 packages per  second.
            Again the first problems started near below 32k ( after 57 x 516 byte packages = 29.412 byte). Total number of transferred blocks is below 50 %

            CTS does never change its status.