Bengaluru-based Cocoslabs Innovations will receive a loan from the Technology Development Board, under Science and Technology Department, to develop a “low-cost solution to identify persons with abnormal body temperature in a crowd and, at the same time, provide an alert system to notify about identified persons to authorities on their phones and laptops”. The product, according to TDB, includes features such as detecting and tracking a person with and without a face mask, predicting age, gender, race, temperature readings, and facial recognition in a single product that can track multiple people in a real-time environment.

“We made a pitch to the Technology Development Board, outlining the sectors we make products for including their use cases. We have multiple products. We are a software OEM,” Prathvi Palekar, founder of the AI video analytics company, Cocoslabs Innovations told MediaNama. His company is among the six start-ups approved by the TDB to build technological solutions to tackle the COVID-19 pandemic. The company, based in Central Bengaluru, brands its products under the Pixuate name. It was registered in 2012, and was in “R&D mode” for the initial 3 years, before finally starting operations in 2016, said Akshata Kari, the company’s co-founder and head of business development. Hindustan Unilever, Godrej, and Bigbasket are among its clients.

Kari told us that the solution which they pitched to the TDB is worth a million dollars. The TDB, though, will only cover around half of that. Kari said that they are not allowed to disclose the exact amount of financial assistance from the TDB, but explained that the Board will provide them a soft loan, at a lower rate of interest compared to the market. The company will be exempt from repaying the loan in the first year, following which it can pay it back in five years. Palekar said that their solution can be deployed at public places within seven to ten days, and the only reason that lead times could increase is because the company is currently ordering cameras on a case-by-case basis, but soon, is looking to place bulk orders. As of now, the company has access to thermal cameras developed in Germany and Sweden, and low-cost one developed in Taiwan.

System can record body temperature of 30 people at once

The solution, which the company has ready at the moment, involves connecting a thermal camera to a workstation which has Pixuate’s video analytics software installed. However, the company is soon looking to combine the two, and is envisioning to carry out analytics on a single device which can act as a camera, as well as a computational device. “Imagine a CCTV which can take temperatures as well as carry out mask identification,” Palekar said.

Tuned to detect temperature of people wearing spectacles and masks: “We have a detection mechanism that triggers as soon as a person comes near the setup,” Palekar said, and added they are using “high configuration” thermal cameras. Currently, their system can accurately detect 30 different individuals at once, Palekar said. For people who aren’t wearing spectacles, the camera detects temperature around the eye ducts, and for people with a spectacle, it notes temperature of the forehead. As of now, their system generates an alert if a person with a detected temperature of higher than 37.5℃ walks past it, but that setting can be adjusted for other temperatures as well, he said.

System generates an alert when it detects high body temperature: In case a person with a high body temperature walks in, the system will generate an alert and flash a thermal image as well as an RGB image of that particular action on the workstation. This detection and decision-making process currently happens on the device locally, and the company isn’t pushing the data to servers because the analysis could take a longer time. However, according to a promotional video shared by Cocoslabs with MediaNama, the company has ideas to carry out these decision making processes on its servers. Having a server room for enterprise clients makes more sense, compared to deployments for the government, Palekar said. He also said that they are in touch with chip designer Nvidia to procure their GPU (graphics processing unit) for carrying out analytics on the servers.

System trained on a dataset of 3 million images by a team of 20 people

AI has been trained on a facial dataset of 3 million images, but their software does much more than just detecting body temperature. Once a face is detected, the software also assesses the gender and age of a person, and whether they are wearing a face mask, Palekar said, claiming that Pixuate’s AI engine has been trained with a facial database of over 3 million images, which includes Indian faces, in order to identify a human’s gender and age. Cocoslabs did not clarify what percentage of the training dataset comprises of faces of colour, despite repeated questions. Palekar claimed that the accuracy rate for that is close to 90% on the ground, and “even more in the lab”. When we asked why their system needs to ascertain a person’s gender and age in the first place, he said that these statistics can help in carrying out analysis on the footfall of individuals in a given space.

20 people work for Cocoslabs in the backend to train the company’s face recognition AI. This team, which is based in Karnataka’s coastal town Kumta, manually annotates facial data collected by the company to classify it according to gender and age.

How the company gathered the dataset: When we asked about the dataset of facial images that the company has access to, Palekar claimed that the “collection process happens without anyone’s privacy being breached”. He claimed that they have facial data collected as part of their work with a “few early partners”, and they also gather facial data from open source platforms on Google. He denied collecting images from social media websites, because those pictures are “compressed” and “not very useful to train their AI”.

Apart from this, the company also gathers facial data from some of its enterprise clients to make customised biometric solutions for them. Since 2014, the company has also relied on data given to them by a particular client so that they can tune their algorithm accordingly. Now, we can obtain data related to safety gear detection in factories, for instance, Palekar explained. It is worth mentioning that currently the company’s face recognition algorithm isn’t compliant with either the US’ NIST, or India’s STQC, but it is planning to obtain an NIST certification. Both NIST and STQC provide standards for biometric authentication systems including facial recognition.

Soon, the system will detect faces even with a mask, and people’s ethnicity

The company doesn’t just aim to stop at face mask detection. “Right now, we’re also working on enabling facial recognition even for people wearing a mask, and will be releasing a beta solution in the next 6-8 months,” Palekar said. Face masks have emerged as an unlikely threat to most facial recognition systems today simply because they hide enough points on a person’s face to confuse any facial recognition system. Even when a company comes out with a workaround, like Cocoslabs claims, its accuracy will still potentially be a question. But in 2-3 months, the company’s software will be able to detect the ethnicity of people, he told us.

Data can be retained for a very long time

The company’s software allows for data retention for a very long period of time, but it did not specify the data retention period despite asking twice. It stores what Palekar called “reports” for “auditory purposes”, and older reports can be archived. The system also stores quite an exhaustive amount of data such as a person’s temperature, their thermal and RGB images, and walk in photos. All of this, is stored on the workstation itself, and not on a server, and is “masked” so that no one can find it in case of a hack, Kari claimed. The data stored on the system is encrypted, and this is “storage level encryption,” and not “end-to-end encryption”, Palekar clarified.

Illustrating the surveillance tool

What the front end looks like to the operator: Palekar went on to explain how the data capturing and retention works in real life — as soon as the software receives video feed from the camera, it is output to a dashboard which can be accessed using an HTTPS protocol. The display on the system itself shows “general analytics” of a week, along with two outputs from the camera. On the right side of the screen, a person can see the video from the camera in real time, and to the left, the software shows temperature of the people passing through the system and if they are wearing masks or not. The output on this particular screen can be accessed remotely as well, using a particular IP address, similar to the addresses used to connect to a WiFi router.

PDFs containing facial data of people can be downloaded: The software also has an option called “export logs” using which, it is possible for a person operating the device to download PDF files of data related to particular dates. This PDF will contain all the data the system would have collected, including images, temperature, gender, and age. “We recommend clients protect the PDF with a password,” Palekar said.

COVID-19 will change biometric authentication forever, much like how 9/11 changed security checks at airports, Kari said. Palekar said, “temperature checks will become common at public places, and people more people will be wearing masks”, and the “algorithm of post-COVID systems will have to be very efficient”. Kari also revealed that the company is in advanced talks with the Karnataka government for deploying their system at government institutions.