FarmView [EN]

Proiect realizat de:

The idea

I have started this project a few months ago. My parents own an agricultural land and they asked me what can they cultivate on that land without involving much work on their side. My idea was to cultivate hazel trees. This idea was well received except for one thing: the plants should be monitored, something my parents can't do because they have full jobs.

Tacking into account that I had finished an engineering university, I had to find a solution to this problem using my knowledges. So that was the moment when I come up with this idea: To create a system that monitors the growing process of the hazel trees.

The sketch

Ok, a monitoring system, that is what I'm about to develop. My first step in the developing process was to identify the components needed for this system. So I started with the environment: this is an isolated area - no it is not in the desert, I mean there is no electricity source from where I can power up my system. The field is four kilometers away from the city. It is easily accessible by car which is a good thing, the field is like an open area, so that means there is plenty of sun, also the river is really close which can be useful at some point in time. To sum up the environment capabilities and how I can use this in my advantage, I have the following:

  • easily access - I can carry any tools that I need during the instalation
  • open field - I can use the solar power to power my system
  • river - I can use this as an alternative power source

I decided to use solar panels to solve the power problem. This involves three parts: a battery, a charger controller and a solar panel. The best candidate for battery is 12V acid car battery, it is cheap and it is compatible with a various number of chargers that are available on market. The controller is PWM type and not a MPPT. Yes you guessed, the main reason is the price which is to high for my budget, also I can update the controller to MPPT later, when the system is mature enough, so the PWM is fine for the first version. The solar panel should be something that is 50W or higher, but this will be chosen after I define all parts in my system and I have a better view of the power consumption.

Another problem is what device type I will use to monitor the area and what should be the purpose of this monitoring? Starting with the purpose of monitoring, the main reason is to prevent theft. The hazel tree is really small in the early growing phase, also it is expensive - around 2 euro per plant, then anybody can go around and pick it up and put it in his garden. Another reason is to get any useful data like the temperature or the humidity of the soil, or any parameter that can be used as an input data for growing phase. Based on this I have decided to use IP cameras and a combination of analog sensors. The IP camera is great because it can provide live stream without using any additional part like a DVR. Also on the market I found some cheap Chinese cameras that are working great. Yes, I had to concern the security, it is a known thing that some of Chinese products have security lacks, but in spite of this I still think that cheap IP cameras fit perfectly in my project. In any case, these can be easily changed to better products after the system is fully developed.

The last problem to deal with is the internet access. The only feasible option was to use a USB modem based on SIM card. The signal there is really great, the 4G is available in the area. The link between the IP camera and the USB modem is done using a "one board computer" device. Of coarse, I chose a Raspbery PI 3 for this job because it has great support and it is very affordable.

Finally I managed to put all the things togheter and I was able to have a first sketch of my system:

The "Power Management and data controller" from my sketch is an abstraction of the electronics that will be used to create a proper power interface for the devices like Raspberry Pi and IP cameras, or other 5V and 12V devices. Also this will be used to pass useful information to the higher layer, for example the battery voltage or the commands that will disable or enable the 5V and 12V outputs.

The bill

After I had put all the things together, the next step was to estimate the entire cost of this system. I tried to get a good price - performance rapport.

Component Price Reference
Raspberry Pi 3 Model B+ 40$ optimusdigital
ZTE mf823 4G 28$ aliexpress
2x HD IP Camera 70$ emag
12V 45Ah battery 47$ dedeman
Monocrystalline 100W solar panel 82$ netdeal
Solar panel PWM charger 13$ netdeal
Power Managment & Sensors module 50$
Total 330$

Actually the entire cost is a bit higher than I expected initially, but compared with other similar systems I think it is really affordable. Just one single IP Camera which is using solar power costs around 300$, but compared with my system, which supports multiple cameras and a lot of sensors, I can say it's worth it

The Implementation

The implementation has been taking around 3 months - yes I'm really lazy - and it is still ongoing at this moment. Next I'll discuss about each part of this project, results and what I can do in the feature to improve the system.

Raspberry Pi

I started with raspbian stretch. I am using the raspberry board to make the connection between IP Cameras and USB modem, in this way I assure the internet connection for the cameras. The cameras supports P2P. This was a must when I bought them because I want to access the video stream from any device really quick, especially from my smart phone.

The ZTE mf823 modem is using NAT, hence the port forwarding is not an option, so any scenario of using the DDNS to access the camera stream is not working - yes this is a big minus for my project, if the third part who manages the P2P just decide to shut down their server, then my system will be impacted really bad, so the access to the video stream should be improved.

How is possible to share the internet connection between IP cameras and ZTE modem? the answer is simple - google it :). This was what I did so I found the following topic . This is about a Wifi router based on ZTE modem. This is not what I really needed for my project, but it changed it to fit my requirements. At least I need a DHCP server to get a local area connection between my cameras and raspberry pi. Then the remaining thing is to share the internet connection between the ZTE dongle and raspberry's LAN. So lets go through this process.

DHCP server configuration

The first step to transform the raspberry pi in a DHCP server is to install the udhcpd software. I used the following command:

 sudo apt-get install -y udhcpd hostapd 

Then I turned off the udhcpd since it is not configured yet:

 sudo systemctl stop udhcpd 

The DHCP configuration can be added in udhcpd.conf file using the following command:

 sudo nano /etc/udhcpd.conf 
Then I replaced the entire content with:
start           192.168.1.10
end             192.168.1.254

interface mlan0

remaining yes opt dns 8.8.8.8 8.8.4.4 option subnet 255.255.255.0 opt router 192.168.1.1 opt lease 864000 # 10 days of seconds

I used the Google's DNS servers and the connected devices will take an IP within 192.168.1.10 - 192.168.1.254 range. Of course this can be changed to fit any range, but this was fine for me. The next thing was to enable the DHCP server, so I added the following line in /etc/default/udhcpd file:

 DHCPD_ENABLED="yes" 

I had to configure the mlan0 interface since I'd configured the DHCP server on the network 192.168.1.0/24 with 192.168.1.1 as default gateway. In this case I had to assign a static IP address to mlan0 interface. This can be done editing the /etc/network/intarfaces file and replace the mlan0 configuration with:

auto mlan0
allow-hotplug mlan0
iface mlan0 inet static
address 192.168.1.1
netmask 255.255.255.0 

To finish the DHCP configuration, the remaining thing was to restart the networking interface and then to enable and start the udhcpd service:

sudo /etc/init.d/networking restart
sudo update-rc.d udhcpd enable
sudo service udhcpd start

Install and configure the ZTE modem

This part is really straight forward. As soon as you plug the ZTE modem in usb port, it brings up the usb0 interface that gives internet access to the raspberry. The USB modem acts like a DHCP server, so the raspberry pi will get an IP that is in the following range: 192.168.0.2 - 192.168.0.254. Also I had to check that the DHCP server is not using the same subnet as mlan0, this can lead to subnet conflict error and the things will stop working

I had to set up the APN. This can be done using the ZTE's web server interface, which is available at the following IP address: 192.168.0.1. I navigated to the Settings->Device Settings->APN, I created a new configuration profile using the given interface and then I fulfilled the fields using my operator APN settings.

Another important thing is to set the PIN for the SIM card, so I navigated to the Settings->Device Settings->PIN Management and I put my SIM card PIN into the Current Pin field. Also I checked the Remember pin box. This is a very useful options because the modem will unblock the SIM card automatically each time the system is rebooting.

This is all, now the raspberry pi has internet access. I tested the connection and it was working well.

Share the internet connection

The ifconfig command shows the following interfaces: mlan0 and usb0. What I had to do was to route the traffic between these two interfaces. This can be done using linux firewall iptables.

I enabled the NAT with this command:

 sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward" 

I enabled the IP forwarding on my raspberry so I added the following line in /etc/sysctl.conf file:

 net.ipv4.ip_forward=1 

Then I added few firewall rules:

sudo iptables \
-t nat -A POSTROUTING -o usb0 -j MASQUERADE
sudo iptables \
-A FORWARD -i usb0 -o mlan0 -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables \
-A FORWARD -i mlan0 -o usb0 -j ACCEPT

I saved the firewall rules with the command:

 sudo sh -c "iptables-save > /etc/iptables.ipv4.nat" 

The rules can be reloaded automatically after each reboot adding the following line in /etc/network/interfaces file:

 up iptables-restore < /etc/iptables.ipv4.nat 
Finally I rebooted the board and checked the internet access on mlan0 interface. The easiest way to do that was to plug my laptop in the Ethernet port and then I checked if it receives some traffic and it did. I plugged my cameras to the Ethernet port and I was able to access my stream from my phone using P2P.

GSM interface

One of the features that is supported by ZTE is the GSM messages. I thought it will be cool to use this in my project to create a user interface. I can easily send messages or read the received messages using the ZTE's web interface, but I wanted to do this from a script so I had to figure out how does it work. Once again I searched on google and I found out that this can be done using goform APIs that are supported by ZTE modem.

Receive and send messages

This can be done just typing in the browser the following address:

 http://192.168.0.1/goform/goform_set_cmd_process?\
isTest=false&goformId=SEND_SMS&notCallback=true&Number=$1&sms_time=$current_time&MessageBody=$encoded_text&ID=-1&encode_type=GSM7_default 

There are some parameters that should be set like number or the SMS timestamp. These can be easily obtained, for example I figured out the timestamp format and then I wrote this line in shell:

 current_time=$(date +"%y;%m;%d;%H;%M;%S;+2") 
The biggest problem was that the format of the text must be GSM7 encoded, so I had to implement ASCI to GSM7 conversion. I had a look in my ZTE web server implementation and I found the GSM7 encoding implementation.

This function was wrote in Javascript. In this case I had two options:

  1. Use Node to call this JS code and get the output in a shell script
  2. Make my own Python function based on the current JS code

    Of course the first option was really quick, the only thing that I had to do was to add this to my shell script:

encoded_text=$(node ./4g_modem/encode_gsm7.js "$2")

Where the encode_gsm7.js is the file that contains the above JS function. This was working great but it was really slow. Calling Node from a shell script or a python script apparently is slow, it takes around one second... Yes one second for each SMS that I wanted to send or read. The reason behind this is the configuration that is used by Node and it can be changed to speed up the things. At that moment I decided to switch to second option and I implemented my own function in python:

import re
GSM7_Table = ["000A","000C","000D","0020","0021","0022","0023","0024","0025","0026","0027","0028","0029","002A","002B","002C","002D","002E","002F","0030","0031","0032","0033","0034","0035","0036","0037","0038","0039","003A","003A","003B","003C","003D","003E","003F","0040","0041","0042","0043","0044","0045","0046","0047","0048","0049","004A","004B","004C","004D","004E","004F","0050","0051","0052","0053","0054","0055","0056","0057","0058","0059","005A","005B","005C","005D","005E","005F","0061","0062","0063","0064","0065","0066","0067","0068","0069","006A","006B","006C","006D","006E","006F","0070","0071","0072","0073","0074","0075","0076","0077","0078","0079","007A","007B","007C","007D","007E","00A0","00A1","00A3","00A4","00A5","00A7","00BF","00C4","00C5","00C6","00C7","00C9","00D1","00D6","00D8","00DC","00DF","00E0","00E4","00E5","00E6","00E8","00E9","00EC","00F1","00F2","00F6","00F8","00F9","00FC","0393","0394","0398","039B","039E","03A0","03A3","03A6","03A8","03A9","20AC"]
GSM7_Table_Extend = ["007B","007D","005B","005D","007E","005C","005E","20AC","007C"]
specialChars = ["000D", "000A", "0009", "0000"]
specialCharsIgnoreWrap = ["0009", "0000"]

def dec2hex(b): return format(b, '04X')

def encodeMessage(b): global GSM7_Table global GSM7_Table_Extend global specialChars global specialCharsIgnoreWrap

a = 0
c = ""

for _b in b:
    f = ord(_b)
    if a != 0:
        if 56320 <= f and 57343 >= f:
            c = c + dec2hex( 65536 + (a - (55296 << 10)) + (f - 56320) )
            a = 0
            continue
        else:
            a = 0

    if 55296 <= f and 56319 >= f:
        a = f
    else:
        c = c + dec2hex(f)

return c

Then I used encodeMessage function to convert my ASCII data to GSM7 format. I made a simple shell script that does a curl on the given web server interface of ZTE modem. This script takes the following arguments: $1 the number to which you send the SMS, $2 the GSM7 encoded text. I set the message timestamp and I check the string length before sending, I prefer to keep it simple and I allow only strings that can fit one message, otherwise the string should be divided between two messages. The propose of this implementation is to have a simple GSM interface between user and system, so the information that is transferred should easily fit one GSM message, I don't see the need for commands or information that are two messages long.


#!/bin/sh

current_time=$(date +"%y;%m;%d;%H;%M;%S;+2")
encoded_text=$2
textlen=${#2}
if test $textlen -ge 1080; then 
    echo [-1] Sms limit is 1080
    return
fi
curl -s --header "Referer: http://192.168.0.1/index.html" \
"http://192.168.0.1/goform/goform_set_cmd_process?\
isTest=false&goformId=SEND_SMS&notCallback=true&Number=$1&sms_time=$current_time&MessageBody=$encoded_text&ID=-1&encode_type=GSM7_default"

And finally I put all together and I was able to send messages from my python script:


def sendSMS(Number, Message):
    output = ''
    global scriptsPath
    try:
        _enc_message = gsm7.encodeMessage(Message)
        output = util.executeShellScript('send_sms.sh',[Number, _enc_message])
        return output
    except:
        e = sys.exc_info()
        print("sendSMS: error during script execution")
        print(e)
        return ''

sendSMS("+40700000000", "Hello, it's me, Mario!")

The receiving part is quite similar with what I did above, the difference is that the messages which are received must be decoded, so instead of using a ASCI - GSM7 conversion, I had to do a GSM7 - ASCII conversion. I created a python function for decoding that can be found here.

What I needed to do in addition was to add other useful scripts that are used to delete the received messages or to clear some flags (when the modem receives a message, it sets a flag and returns the number of received messages, these flags must be cleared by hand). These scripts can be found here, so I'll skip the explanation since it is not rocket since.

1

Python application

For the main application I used Python language because it has good support for high level programing (Object Oriented), which will speed up the implementation, also it could be used to acceess the Rpi hardware peripheral, in my case the UART module. The first step was to define some functionality:

  • The application shall process GSM data (SMS commands)
  • The applicaiton shall process serial data to get useful information from hardware (e.g. battery level)
  • The application shall manage the battery level and it shall power off the system when battery is low
  • The applicaiton shall reboot the system when the system fails (undervoltage, network outage, etc)
  • The application shall notify the user using SMS about the internal state or other events (reboots, shutdown, lowbattery)
  • The application shall log all messages to the system messages
  • The application shall post data to the webserver

Based on that I made a representation of the modules:

1

Articol scris de : Andrei Mestereaga