MLX90640 thermal camera demo with the Blue Chip Technology Beta HMI & Android platform

Author: Dave Robinson – Lead embedded engineer at Blue Chip Technology

Many prototypes that I come across use sensors that implement a GROVE connector, and I am often asked, “Can I attach GROVE sensors to your Beta HMI?”, “Can you help support module X with your Beta platform?”, and “Can you recommend how we implement our software to work with module X?”.

The answer to the first two questions is usually yes though the ‘How’ usually comes with the typical engineering caveat of, “It depends!”. Sometimes our engineers will not know the answer straight away but will work with customers to come up with an optimised solution.

To demonstrate the ease of using the Beta HMI platform, and for a bit of fun, I thought I would go through the process of integrating a Melexis MLX90640 thermal camera. This particular sensor is well suited to the Beta as the platform supports actively driven TFT displays in various sizes for displaying the captured data. The MLX90640 has high compute requirements, requiring many floating point calculations to be performed on every frame of thermal data. Many microcontrollers struggle with this kind of compute requirement either through lack of memory or CPU performance. The Beta platform featuring ARM Cortex A9, ARM Cortex A53, or X86 cores has plenty of power to easily process the thermal data and several storage options for results archiving.

Attaching the MLX90640 thermal camera to the Beta platform

The Beta platform does not implement GROVE connectors however the 50 pin expansion header does feature I2C, UARTs, and GPIO which are the required interfaces for three of the four GROVE connector types. The MLX90640 sensor requires an I2C interface and the easiest way of attaching the sensor to the Beta platform is via the 50pin screw terminal adapter board. Using a standard GROVE cable I attached the cable to Beta as follows:

GROVE Signal	Wire Colour	Beta 50 Pin Signal
VCC	Red	B – VCC_3V
GND	Black	C – GND
SCL	Yellow	D – SCL
SDA	White	C – SDA

See page 21 – http://dl.bluechiptechnology.com/dl/hb6/Documentation/HB6%20-%20Technical%20User%20Guide%20v1_10.pdf

Software configuration

The Beta platform supports a variety of our SBC’s which between them offer embedded Linux, embedded Android, and Windows 10 support as well as BSP’s and development environments for each of the options. In this case I chose the TM1 COM module with a HB6, paired with a 4.3” LCD and PCAP touch panel.

There are numerous ways of implementing software solutions for any operating system and this complexity is something that we regularly advise and support customers with decision making progress.

Taking Android as an example two solutions immediately came to mind.

Linux Kernel V4L camera driver -> Android HAL -> Native Android camera API in a JAVA app.
JAVA app communicating with the camera directly using the BCT I2C API.

For the demo I wanted to capture absolute temperature data from the sensor, and then display it on screen as a colour image with the temperature measured at the centre of the frame printed on screen. From experience I know that writing Linux V4L drivers and Android HALs is a time consuming business, so I opted for talking to the sensor directly using the BCT’s Android I2C API.

A read through the MLX90640 datasheet gave a basic understanding of the communication protocol. I quickly reached section 11 where it became apparent how many equations must be performed on captured data before absolute temperature values can be realised.

https://mel-prd-cdn.azureedge.net/-/media/files/documents/datasheets/mlx90640-datasheet-melexis.pdf

The complexity and number of equations initially made the task of supporting the camera seem like a large one until I identified a library implemented by Melexis. The library contains all the functionality required for reading thermal data from the sensor, with the exception of a hardware specific I2C bus implementation, or in my case an operating system specific implementation.

https://github.com/melexis/mlx90640-library

Using the library seemed like the only sane choice as it would save time in validating lots of equation implementations, however the library is written in C and I was targeting a JAVA application. Given this I opted for a third type of implementation. It is possible for JAVA and C code to coexist in the same solution by using JAVA native interface (JNI) as the mechanism for calling C functions from JAVA code. To use the library I added a Linux specific I2C implementation, and added two additional functions to allow JAVA JNI calls into the library.

Java_bct_thermalcamera_ThermalCameraMLX90640_InitialiseMLX90640

Java_bct_thermalcamera_ThermalCameraMLX90640_GetMLX90640Frame

My final implementation was a JAVA Android app using SurfaceView classes to display thermal images on screen, with JNI function calls to receive thermal data from the MLX90640 sensor.

To finish the solution I enabled KIOSK mode and loaded in a custom splash screen which allows the system to boot straight into the app and a user is none the wiser that the system is Android underneath.

App source code is available at https://github.com/bluechiptechnology/ThermalCamera

Thermal accuracy

I was keen to quickly test the accuracy of the thermal sensor and the obvious option was to compare the thermal camera with our in-house thermal camera which we use for product validation. Unfortunately the camera was not available as it had been sent out for calibration. Instead I opted to use the MLX90640 sensor to measure the CPU temperature and compare the measurement against the core processor temperature reported by the operating system.

The processing module used for this demo was a TM1 featuring an i.MX6 processor. This processor has an integrated thermal sensor that reports the core temperature. In Linux (and Android) this can be read using the sysfs filesystem from a serial console.

tm1hb5:/ # cat /sys/class/thermal/thermal_zone0/temp

60220

The output 60220 above indicates that the CPU core temperature was read at 60.22 degrees. The thermal camera can only report the temperature measured on the surface of the processor so to convert a core temperature to a surface temperature we need to know the temperature delta. Referring to the i.mx6 datasheet the delta required is labelled as “Junction to Case” and defined as 9 °C/W.

Given a core temperature of 60.22 degrees and a Junction to Case delta of 9 degrees I expected the case temperature to be 51.22 degrees and I was impressed to find that the MLX90640 sensor reported the case temperature as 51.85 degrees, well within the 1°, precision noted on the Melexis website.

MLX90640 frame rate

As part of my experimentation with the MLX90640 sensor I decided to see how the Beta platform and camera performed at different frame rates. The Beta platform easily coped with all frame rates however I found that increasing the frame rate introduced noise to the captured thermal data. One benefit in capturing thermal data at high frame rates would be to monitor objects moving reasonably quickly however my initial conclusion is that the noise introduced with the higher frame rates would make it difficult. The MLX90640 is a low cost thermal sensor after all, and therefore not suitable for all use cases.

The thermal images captured in the frame rate comparison are of me and I was initially surprised to see that the thermal camera was measuring my temperature at between 32 and 33 degrees. Knowing that the human body temperature is typically 37 degrees I presumed that the camera was not working or not configured correctly, however research suggests that human skin temperature is typically between 32 and 34 degrees, and it is the body’s core temperature that is typically 37 degrees. This raises the question, How do the body thermometers that we have all become accustom to using during the pandemic determine core temperature as surely they can only read surface temperature? I assume they must apply a ‘core to skin’ temperature coefficient similar to the one employed for the CPU temperature measurement above.

TM1 CPU performance Vs ESP32

Knowing that the MLX90640 is compute intensive I was interested to find out how the TM1 processing module would handle the calculations and how the performance would compare to an ESP32 microcontroller. The library function that performs the calculations on every frame is MLX90640_CalculateTo and I decided to use this function to benchmark performance using a static set of test data (No I2C communication). I found there to be plenty of resource available in the Beta platform for processing the thermal data and that larger resolution thermal cameras could easily be accommodated.

	ESP32	TM1 – i.MX6
Core Frequency	240Mhz	996Mhz
MLX90640_CalculateTo Processing Time	14.9ms	0.38ms
Max MLX90640_CalculateTo FPS	67	2631

Whats Next?

Determining what the thermal camera is focussed on can be tricky, especially when there is only a small contrast of colour across a thermal frame. In a future blog I plan to expand the thermal camera demonstration by combining an optical camera as an overlay to the thermal image. This should make it easier to identify what the thermal camera is focused on. Who knows I may even add a ToF range sensor too!

About Blue Chip Technology

At Blue Chip Technology (BCT) we specialise in design and manufacture of embedded electronics systems, Hardware, Firmware and Applications. Our typical customer will usually have a proof of concept and will work with Blue Chip to engineer what is often a complex collection of circuit boards into a commercialised solution.

The Beta HMI platform offers an opportunity to develop those prototypes with a validated, field proven integrated SBC, LCD and touch platform, saving several design and development iterations. BCT’s in house manufacturing and commitment to full life cycle support offers a unique proposition to dramatically reduce time to market, approvals and start-up costs.