FTP overhead?

10 thoughts on “FTP overhead?

  1. We’re using a GC864 that ftp’s temperature data. Management is complaining about the cost of the service provider (who charges per byte).  We’ve gotten traces from the provider and it appears that there is quite a bit of overhead added to the ftp transactions. We estimate that our data is less than 10% of the total bytes.

     

    Is this normal? Can anyone corroborate this? 

     

    I’ve used the #GDATAVOL command, but this only verifies our data size.

     

    Thanks.

     

     Tom

    1. What is that overhead? Is strictly attached to your FTP traffic (packets) or are other traffic? How is your traffic pattern, big/small files, how frequent?

       

    2. FTP overhead is approx 1,5KB on control channel (port 21) that is connected with opening connection, authorization, closing connection and 108 bytes + data packet (by default 300 bytes) on data channel (port 20).

      Remember, that FTP is TCP-based – every data packet has 54 bytes TCP overhead + 54 bytes TCP acknowledge. 

       

      FTP is useful for sending if large data volumes that are logically united, like picture file.

       

      You can use FTP for off-line temperature monitoring, but if data are sending rarely and file is at least 1-2KB or more to have better data to overhead ratio. You can also increase data packet size to 1500 bytes.

       

      For on-line temperature monitoring is better to use TCP or USP data transfer to dedicated server.

       

      Personally, i am using FTP only for Python scripts update and for off-line bird tracking,

       

      For car tracking, i am using UDP transfer to simple Python based UDP-server. 

      1. FTP overhead is approx 1,5KB on control channel (port 21) that is connected with opening connection, authorization, closing connection and 108 bytes + data packet (by default 300 bytes) on data channel (port 20).

        Remember, that FTP is TCP-based – every data packet has 54 bytes TCP overhead + 54 bytes TCP acknowledge. 

         

        FTP is useful for sending if large data volumes that are logically united, like picture file.

         

        You can use FTP for off-line temperature monitoring, but if data are sending rarely and file is at least 1-2KB or more to have better data to overhead ratio. You can also increase data packet size to 1500 bytes.

         

        For on-line temperature monitoring is better to use TCP or USP data transfer to dedicated server.

         

        Personally, i am using FTP only for Python scripts update and for off-line bird tracking,

         

        For car tracking, i am using UDP transfer to simple Python based UDP-server. 

        Thanks Nikolay, that’s exactly the information I was looking for!

         

        To clarify, we gather data every 5 minutes, then connect every hour to send it up. The data collected each time is between 20 and 50 bytes or so, not very much. I’ll take a look at using UDP instead.

        1.  To clarify, we gather data every 5 minutes, then connect every hour to
          send it up. The data collected each time is between 20 and 50 bytes or
          so, not very much. I’ll take a look at using UDP instead.

           

          Even 50 bytes ever 5 minutes gives 600 bytes per hour. Too small amount of data for http://FTP... considering also time for connection establishment…

           

          TCP will be better  – 54 bytes overhead for data packet and 54 bytes for TCP stack acknowledge.

           

          UDP is the best – 42 bytes overhead for data packet and possibly 42 bytes overhead for user protocol acknowledge. I prefere to put acknowledge in my protocol to prevent any data loss. Usually no any data loss, but for any case…

           

          But most important issue making UDP better than TCP is that UDP is connectionless protocol. GSM/GPRS provider’s software do not know about your live UDP connection in contrast with TCP connection. Probably the problem is local, but our providers from time to time break TCP connection. Everuthing is for money – every connection establishment is charged with fixed traffic volume (1 KB or so). Traffic also is charged per KB, so try to send packets slightly smaller than 1 KB.

           

          On client side that is sending temperature reports, no need of socket closing. Just before next data transfer check the socket status. Usually it remains open. Only port number (possibly and IP-address are different), so some device identification is needed.

           

          One additional important issue about the UDP Server side. I am using Python and Windows based server machine. By default in Windows OS, UDP receive buffer is 8 KB.

          So if you consider to use UDP, increase buffer size. I am using 20MB buffer size and during stress tests this buffer was sufficient to receive raw GPS data (GPGGA + GPRMC sentences that are approx 150 bytes together) every second from 200 clients. After buffer filling to the top – packets are lost till next buffer reading to free some space.  

           

           

          1. Hi all

             

            Interesting issue and good arguments. But I have a feeling that not all GSM operators allow UDP traffic at all. Anyone having experience doing UDP over GPRS?

             

            Br, Tom

          2. Hi all

             

            Interesting issue and good arguments. But I have a feeling that not all GSM operators allow UDP traffic at all. Anyone having experience doing UDP over GPRS?

             

            Br, Tom

            Tom, many important application are based on UDP. DNS for instance. 

             

            In fact, GPRS in the internal network of GSM operators is based on UDP..

             

  2. Dear Nikolay

     

    Might well be as you say that GPRS is built on UDP. But I was rather referring to UDP over GPRS as an end user service. That might not always be available. Pleas have a look at the older discussion in thread 663. 

     

    Quotation from Cosmin´s say there:

     

    "Returning UDP with no public IP cannot work behind network operator’s routers/firewall because of the way these packets traverse them, being connectionless, and treated completely different from TCP packets for which traverse is tracked forth and back. So there is no way other than obtaining public IP from net operator (and talk to them about correct handling of returning UDP packets)."

     

    But still, one should try UDP, I think. It depends on your operator.

     

    Best regards,

    Tom 

     

    1. Dear Tom,

       

      From client point of view (GSM module), UDP transfer to UDP server running on PC with real and statuc IP address or url is without any pronlem.

      I have 3 different applications that are working now using UDP and networks of 3 different Bulgarian operators.

       

      The problem is when UDP server have to send packet back to device.

      It must be done within short timeout after receiving packet, because after elapsing of this timeout last IP address and port (visible from server side) are changed by Operator.

       

      If device sends packet regularly – no IP or port change.